[volunteers] Why you are careful about /dev/null operations as the root user

[Tim Utschig]

On Wed, Dec 16, 2015 at 05:29:42PM -0800, Rick Moen wrote:
> Holy mother of god.  Someone/something overwrote /dev/null.
[...snip...]
> lists:/dev# cat null
> 1287
> 2756
> 2903
> 3025
> 3033
> 3034
> 3332
> 3333
> 3457
> lists:/dev# 


Looks like PIDs. In /var/old-svlug-rfs/etc/crontab there is copious use
of pgrep. E.g:

  * * * * * root pgrep apache &>/dev/null || [...snip...]

Here's my theory: There was at some point a 'bind' mount of /dev onto
/var/old-svlug-rfs/dev, but nothing in place to replace that mount at
boot time.

Maintenance happened, and the system rebooted.  /dev inside the chroot
is now an empty directory.  Scripts running as root, such as the above
mentioned crontab entries are now creating/overwriting /dev/null every
time they run.

Around 4:00 PM today I logged into lists.svlug.org to figure out why I
was seeing messages like this on my server:

... connect from lists.svlug.org[71.19.144.13]
... SSL_accept error from lists.svlug.org[71.19.144.13]: lost connection
... lost connection after STARTTLS from lists.svlug.org[71.19.144.13]
... disconnect from lists.svlug.org[71.19.144.13]

A little research indicated that this was due to a problem accessing
/dev/{u,}random.  I had a look and found the chroot's /dev contained
only 'null' and nothing else.

I looked around and found /etc/init.d/start-chroot.  I stopped the
services listed there, bind mounted /dev on /var/old-svlug-rfs/dev, and
started them up again.

I added the bind mount to /etc/fstab next to the other two bind mounts.

However it occurred to me that I was unsure whether bind mounting from
/dev would work during startup, or conflict with udev/systemd/whatever.
Rather than try to figure that out now, I moved the bind mount to
/etc/init.d/start-chroot, which I'm fairly certain should work.


> Well, this cost me much of my first day back from vacation.


I'm sorry it was so much trouble :-(


-- 
Tim Utschig <tim at tetro.net>
408-644-3861 (mobile)