[volunteers] Why you are careful about /dev/null operations as the root user

[Sarah Newman]

On 12/16/2015 08:57 PM, Tim Utschig wrote:
> On Wed, Dec 16, 2015 at 05:29:42PM -0800, Rick Moen wrote:
>> Holy mother of god.  Someone/something overwrote /dev/null.
> [...snip...]
>> lists:/dev# cat null
>> 1287
>> 2756
>> 2903
>> 3025
>> 3033
>> 3034
>> 3332
>> 3333
>> 3457
>> lists:/dev# 
> 
> 
> Looks like PIDs. In /var/old-svlug-rfs/etc/crontab there is copious use
> of pgrep. E.g:
> 
>   * * * * * root pgrep apache &>/dev/null || [...snip...]
> 
> Here's my theory: There was at some point a 'bind' mount of /dev onto
> /var/old-svlug-rfs/dev, but nothing in place to replace that mount at
> boot time.
> 
> Maintenance happened, and the system rebooted.  /dev inside the chroot
> is now an empty directory.  Scripts running as root, such as the above
> mentioned crontab entries are now creating/overwriting /dev/null every
> time they run.
> 
> Around 4:00 PM today I logged into lists.svlug.org to figure out why I
> was seeing messages like this on my server:
> 
> ... connect from lists.svlug.org[71.19.144.13]
> ... SSL_accept error from lists.svlug.org[71.19.144.13]: lost connection
> ... lost connection after STARTTLS from lists.svlug.org[71.19.144.13]
> ... disconnect from lists.svlug.org[71.19.144.13]
> 
> A little research indicated that this was due to a problem accessing
> /dev/{u,}random.  I had a look and found the chroot's /dev contained
> only 'null' and nothing else.
> 
> I looked around and found /etc/init.d/start-chroot.  I stopped the
> services listed there, bind mounted /dev on /var/old-svlug-rfs/dev, and
> started them up again.
> 
> I added the bind mount to /etc/fstab next to the other two bind mounts.
> 
> However it occurred to me that I was unsure whether bind mounting from
> /dev would work during startup, or conflict with udev/systemd/whatever.
> Rather than try to figure that out now, I moved the bind mount to
> /etc/init.d/start-chroot, which I'm fairly certain should work.
> 
> 
>> Well, this cost me much of my first day back from vacation.
> 
> 
> I'm sorry it was so much trouble :-(
> 
> 

I saw lists.svlug.org was up as a web page and didn't take the time to test the email, sorry.