[svlug] Inits (was: On the process of picking up systemd)

Rick Moen rick at svlug.org
Wed Jan 7 13:21:57 PST 2015


Steve Litt wrote:

> Also, there are other dependency models that can work with the kernel's
> increasingly event driven nature.

I'm certainly open to the possibility.

> There are many, many ways to handle stuff like this. A cost-benefit
> analysis must be applied. In my view, making init responsible for the
> likes of hot-pluggable devices is a huge cost. A proof of concept
> alternative I came up with after 20 minutes' consideration is:
> 
> inotifywait -m -e CREATE,DELETE /dev/usb
> 
> A little more plumbing and the preceding can mount plugged in thumb
> drives. Or one can use the C language inotify library for a more
> polished and professional look. Does this have costs? You bet it does.
> But so does an event driven init.

Hold that thought, as we'll get back to it.  (I've delayed a couple of days
getting back to you, in part because this really isn't at all my area.  I've
tried to carefully avoid hotplug-anything on Linux/BSD.  Also, ordinarily,
when has anyone previously cared very much about Unix inits?  That whole
area normally is remote from anyone's concerns, even those of us who've used
many of them.)

> I'm pretty sure s6 and nosh are event driven also, or at least have the
> API for you to use events in triggering service startup.

I'm pretty sure not, but first, let's for clarity's sake review what is
meant in this context by event-driven, and what is the reason for claiming,
as the Debian SysVInit package maintainers did, that it's a necessary trait
in next-generation inits for general use.  

Modern kernels detect changes to available hardware on various interfaces -
network interfaces, disks, USB devices of all kinds, PCI devices, CompactPCI
devices, PCMCIA CardBus devices, SCSI, IEEE 1394/ Firewire / iLink devices,
UPS power supplies, various input devices, laptop docking stations, hardware
interrupts, or when hardware undergoes a significant state change.  The
kernel notes these changes by emitting 'uevent' data that is available to
userspace through the netlink socket.  In addition, the kernel exports
another variant of that information to a root-owned userspace process it
spawns, one instance per event, with environment information about the
change.  By default, that will be an instance of /sbin/hotplug, but the
binary to be invoked can be changed by editing a new filespec into
/proc/sys/kernel/hotplug.

Running userspace daemons can be left monitoring the netlink socket.  Thus,
any such monitoring daemon has to be running in a loop that keeps constantly
checking.  (udevd and its various substitutes do such checking, for
example.)

The kernel also exports the current state of the hardware to synthetic
filesystem /sys, of type sysfs.  Userspace applications (or the admin) can
inject uevents into the hotplug subsystem by writing update to /sys. 

Keeping up with the changing picture of hardware is tricky.  For a system to
do so, it must start registering hotplug events immediately at boot time,
and then catch up with the dynamically built initial contents of /sys to
populate /dev with the right special files.  There is significant risk of
race conditions and of leftover /dev/ entries built based on /sbin/hotplug
information but no longer valid.  Careful use of the netlink interface is
said to help avoid that risk.  

The hotplug mechanism also takes care of timely loading of firmware blobs
under /lib/firmware, at times required to register the hardware in question
and load drivers.  One consequence of this need is that the kernel actually
runs /sbin/hotplug once before running the init process, if that binary is
present and executable in the initramfs, because doing so permits loading
firmware blobs needed to usefully run any statically linked, i.e.,
compiled-in, device drivers.

One write-up:
http://www.a-ticle.com/operating-systems/how-does-linux-kernel-detect-and-plug-in-your-hardware-kernel-sysfs-udev-and-dbus-collaboration/54/T

The exported hotplug and netlink information, in any event, must be used in
the right order at the right times to do things like load drivers to
initialise hardware.  Doing that task was formerly complex but do-able using
old-school inits, by careful attention to pre-declared dependency
information.  Thus, the nice procedural code that you, Marc Merlin, and I
like sufficed, provided the dependencies information was in good order.

However, in recent years, this has gotten crazy-difficult, for a couple of
reasons.  One, it's no longer possible to state in advance the order in
which hardware-related events such as initiailising network interfaces and
other components will occur, or to programmatically plan for when they will
complete.  Two, because the order of kernel operations can no longer be
guaranteed, I am _told_ that one runs the risk that, say, your eth0 and eth1
interfaces will be reversed during some boots.  

Serious Linux users who really ought to know what they're talking about have
told me that repeatedly, and, even though I keep saying 'Really?  I've never
had that happen in twenty years', they say it's so.  I cannot tell you the
circumstances that make it likely.  Maybe this is one of those things where
my habits in hardware selection and system management make it unlikely for
_me_.  But I believe the people who say it's a real problem.

Three, I hear that with the elimination of the kernel's much-disliked big
spinlock, hardware events are an order of magnitude more indeterminate in
their order and time to completion than they used to be, and a great deal
more parallel processing occurs in the kernel than before.

My understanding is that the term 'event-driven' in this context means code
that keeps an event loop running and is prepared and able to spawn a variety
of sequences in response to hardware (and other kernel) events.  This isn't
just a matter of dependencies, of knowing that relevant ethernets must be
initialised before NFS daemons get launched, and that NFS daemons must be
shut down before the interface is shut down.  It's also being able to notice
at any time during runtime that the set of interfaces registered has
changed, and trigger appropriate responses.  Speaking of network interfaces,
the time to initialise such an interface and reliably have it be usable is
also extremely indeterminate compared to how it used to be.  Therefore, old
heuristics of just doing a sleep() for a while before assuming the interface
functional just aren't good enough any more.

I downloaded the source code to nosh, and am looking at it right now.  I am
not seeing a general-purpose event loop to keep watching for data from
netlink or /sbin/hotplug.  Therefore (unless I'm missing something, which is
possible) , it is not an event-driven init in the sense discussed.

I like nosh; it's a nice piece of work, even with all of its DJB-school
oddities (and even though I think socket-activated services are in general
undesirable).  For your usage scenario, for mine, for Marc's, and for
similar admins who don't have a lot of hotplug activity and try to avoid it
being relevant, it's probably good enough for now.  I like the ability to
use such things.  But the problem that the Debian SysVInit maintainers
described isn't meaningfully addressed by them.

It's also a nice touch that nosh can intelligently import systemd service
unit files.

I did not (once again) download s6 source code, but a quick look through the
docs, such as they are, again suggests no event-loop code in the sense
intended.

You think inotify code and 'a little more plumbing' is a functional
equivalent.  I really don't think so.  That doesn't even begin to cover the
problem space.

Does hotplug code have a 'huge cost'?  Undoubtedly, which is one of the
reasons why I try to avoid it.  However, you're changing the subject in
stressing that fact.  You don't need to convince _me_ that I'd rather do
without hotplugging, or probably Marc and other ornery old sysadmins, but
that leaves 95% of all Linux users.

Debian and other distros that have leaned towards event-driven inits have
done so because they'd like a broad cross-section of usage models to all
have functional and deterministic startup, shutdown, and addition/removal of
hardware, and made an honest and reasonable judgement that they needed an
event-driven init to make that work.

nosh, s6, and several other elegant and simple inits are missing that trait.
They have very compelling other traits, but not that one.

BTW, I omitted one obviously qualifying init from the list:  uselessd.




More information about the svlug mailing list