[volunteers] More daemon fun

[Rick Moen]

Quoting Darlene Wallach (freepalestin at dslextreme.com):

> Since you could not telnet in you are going to try another method.

Aha!  No, please let me clarify.

The "telnet" tool (/usr/bin/telnet) can be used for _far_ more than just
the telnet remote shell service.  It connects to a remote server's
network telnet service (which is on TCP port 23) _only_ as a default
setting, when you say "telnet [hostspec]" without a qualifier.

In this case, I said "telnet lists.svlug.org 25".  The "25" is an
alternate port number.  It happens to be the port number assigned by
IANA (Internet Assigned Numbers Authority) standards to SMTP, the
protocol used for delivering mail between Internet mail servers.

On *ix machines, the file /etc/services maps TCP/IP service names like
"telnet" and "smtp" with TCP and UDP port (service) numbers.  Here's the
first part of that file:

  # Note that it is presently the policy of IANA to assign a single well-known
  # port number for both TCP and UDP; hence, officially ports have two entries
  # even if the protocol doesn't support UDP operations.
  #
  # Updated from http://www.iana.org/assignments/port-numbers and other
  # sources like http://www.freebsd.org/cgi/cvsweb.cgi/src/etc/services .
  # New ports will be added on request if they have been officially assigned
  # by IANA and used in the real-world or are needed by a debian package.
  # If you need a huge list of used numbers please install the nmap
  # package.

  tcpmux          1/tcp                           # TCP port service multiplexer
  echo            7/tcp
  echo            7/udp
  discard         9/tcp           sink null
  discard         9/udp           sink null
  systat          11/tcp          users
  daytime         13/tcp
  daytime         13/udp
  netstat         15/tcp
  qotd            17/tcp          quote
  msp             18/tcp                          # message send protocol
  msp             18/udp
  chargen         19/tcp          ttytst source
  chargen         19/udp          ttytst source
  ftp-data        20/tcp
  ftp             21/tcp
  fsp             21/udp          fspd
  ssh             22/tcp                          # SSH Remote Login Protocol
  ssh             22/udp
  telnet          23/tcp
  smtp            25/tcp          mail

So, the word "smtp" is associated with TCP port (service) number 25 by
that file.  Therefore, these two commands are equivalent:

   telnet lists.svlug.org 25
and
   telnet lists.svlug.org smtp

Either one opens a connection to the specified remote host's SMTP mail
service (if there is one, and it's running).  Attempting that and
getting "Connection refused" generally means, as it does here, that 
there's no such service running, hence it cannot be reached.

On occasions when you _do_ reach a running SMTP server, you suddenly
find yourself able, if you wish, to carry on an SMTP conversation
manually, as if you were a delivering MTA process talking to the remote
MTA and dropping off mail.  I believe further down in my post (from
earlier today), I illustrated such a manual SMTP session.


> ssh worked to login

Yes.  If it didn't, that would be when we'd either need to ask Via.net
to reboot the machine or for our volunteers to visit its console.

> After successfully logging in you checked if the MTA exim was running,
> it wasn't

Quite so -- and part of my point is that _only_ starting the MTA, which is
something I did the other evening (because I was short of time) risks
leaving unfixed problems with related network daemons that might have
gotten wedged by whatever the original underlying problem was.


> I had to "man ps" as I use the UNIX options vs the BSD options.
> I had not remembered "w" - wide.

Yeah, "ps" is a bit of a mess (in part) because you can use either the
BSD or System V-style options for it.  I happen to generally just start
with "-auxw" because that ensures that absolutely everything is
included, including full command lines (because of "w") -- and I then
feed it through grep to extract just the part I really care about.

It's a bit wasteful of machine resources, but not much, and eliminates
the need to think about "ps" options entirely.

> You are shutting down services in a certain order.

You bet -- though that probably doesn't really matter.  If exim4 had
been running, I'd have shut it down first, because it's the
outward-facing process that accepts incoming mail (among other things it
does), so shutting it down stops the inbound flow.

What's really rather more important is to put exim4 last when _starting_
the daemons.  Like, for example, you wouldn't want to start accepting
all incoming inbound mail before spamd (Spam Assassin in daemon mode) is
running, because the entire point of having spamd there is to measure
spamicity of incoming mail.  

> You stopped the services in this order:
> spamassassin
> mailman

Which doesn't actually matter.  What would have mattered is if exim4 had
been running.  Then, the desired shutdown order would have been:

exim4, then
spamd and qrunner (mailman) in any order. 

> You checked that spamassassin had stopped by grepping spamd
> You checked that mailman had stopped by grepping qrunner

Yep.

> You restarted processes in a certain order:
> mailman
> spamassassin
> exim4

And, again, all that really matters is that exim4 was after the others.

> You verified the processes had started.

Ja.

> Why are there so many "spamd child" processes? How many should be
> expected to start?

/etc/default/spamassassin controls how many separate spamd processes get
launched.  In this machine's case, it includes:

# See man spamd for possible options. The -d option is automatically added.
OPTIONS="--username=nobody --max-children 32 --helper-home-dir=/var/spool/spamassassin/ --nouser-config"


So, it's actually an interesting and debatable question whether 32 is
too high or not.  Setting it too high can, _itself_, cause eventual
overcommit of memory, causing the system to run out of memory (and
swap), and have problems.  What tends to happen is that you're operating
with some lower number of processes, and, one day, you notice this sort
of thing in the system logfiles:

 spamd[10234]: prefork: server reached --max-children setting, consider raising it

So, you think, OK, let's try that, increase the --max-children setting,
restart spamd, and notice there's a bunch more running and no more such
warnings.

One thing to know about spamd is that it's really slow at processing
mail, because it's pretty complex and because it's written in Perl.
Theoretically, the gain you get from having a whole bunch of them
running in parallel is that they can keep up with the much-faster MTA's 
shoveling of incoming mail at them.

> You successfully telneted in and sucessfully sent a message.

To very slightly correct the above:  I used the telnet client to open a
remote terminal connection to the remote telnet service, and then "spoke
SMTP" (which, as you'll notice, occurs entirely in English-like text in
ASCII) to the remote server, as if I were a delivering MTA (Mail
Transfer Agent = SMTP daemon) process.

> I will update the files after the August meeting.

Cool!  Thanks for your help.