[web-team] (forw) Re: Smaug mailing list settings

Rick Moen rick at linuxmafia.com
Wed Jan 6 15:35:58 PST 2010

For collective knowledge.

----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----

Date: Wed, 6 Jan 2010 14:58:17 -0800
From: Rick Moen <rick at linuxmafia.com>
To: Peter Belew <peterbe at sonic.net>
Subject: Re: Smaug mailing list settings
Organization: Dis-

Quoting Peter Belew (peterbe at sonic.net):

> Hi Rick -
> Ok - duly educated! :)
> I'll just do better filtering of what I receive from the list.

I hope you know I meant no personal criticism.  I was just taken by
surprise.  Also -- and there's no reason to think you should have known
this -- the lists.svlug.org machine had been having some substantially
worse performance, and the several hundred mostly semi-random e-mail
addresses I cited were a major part of the problem.[1]

Mailman is a big, slow Python script, you see.  Thus, like SpamAssassin,
which is a big, slow Perl script, you want to avoid having big filtering
rulesets in it.  The right place to the lion's share of a system's
spam-blocking, it turns out, is in the MTA (the SMTP server package),
right at the time of the inbound SMTP session where the remote system
attempts to drop off the spam.

Relevant to that:  In an ideal world, lists.svlug.org would have been a
maintainable Debian box that could be kept updated incrementally,
basically forever, with little effort.  Unfortunately, Marc Merlin
hand-built much of the software in 2003(?) from upstream (non-packaged)
betaware source code, so it's stuck in a scarily unmaintainable state,
and it's going to be necessary to cut over to a replacement box -- which
hasn't happened yet.  

Until then, lists.svlug.org's spam-blocking is not bad -- better than
most ISPs', for example -- but the machine itself has a problem.
Occasionally, the ancient handbuilt SpamAssassin daemon goes into
runaway RAM consumption and triggers the ancient 2.4.x kernel's
out-of-memory killer, which then somewhat randomly kills processes to
prevent the machine from falling over.  As a bandaid, Marc has installed
a cronjob that checks the process table every minute (IIRC) to ensure
that spamd, exim4, Mailman qrunner, Apache httpd, and sshd are still
running and relaunch them if they're not.

The biggest flaw in that scheme is:  What happens if crond dies?  And
that's what happened this past Sunday or Monday, as part of a worst-case
cascade failure that caused a temporary spam flood.  Near as I can tell,
it went like this:

1.  spamd (SpamAssassin daemon) goes haywire, starts grabbing RAM.
2.  kernel OOM killer zaps cron.
3.  kernel OOM killer zaps all spamd processes.
4.  System, which now is running a full-blown Internet MTA with almost
    no spam-blocking, continues to run for 2-3 days, accepting massive
    amounts of spam that would normally be rejected.
5.  Late Monday night, attempting to figure out why the monthly 
    svlug-announce message wasn't going out, I noticed the foregoing
    situation.  Fixing it required, among other things, force-delivering
    Exim4's outbound SMTP queue overnight -- and most of that would
    inevitably have been redelivered spam.

_None_ of that spam hit the mailing lists themselves, because all
mailing lists were and are set for subscriber-only posting.  However,
hundreds of held spams landed in Mailman admin queues -- all of which
I cleaned out manually on Tuesday morning.

The redelivered spam that _did_ go out was of several types:

1.  Mail to system aliases such as "root at svlug.org" and
"postmaster at svlug.org".  You aren't on those aliases, so you wouldn't
have gotten those.

2.  Spam addressed to list-administrative addresses.  You would 
have gotten any such mail addressed to "smaug-owner at lists.svlug.org",
for example.  Unlike spam addressed to the mailing address _itself_, 
spam to the admin addresses isn't required to be from subscribers.

3.  Notices sent out by Mailman to listadmins about spam added to the 
admin queues.  You would have gotten those, of course, for spam held in
the Smaug queue.

During the aforementioned 24-48 hour period of full MTA service with no
spamd, I estimate that an order of magnitude more spam than normal got
past Exim4 and hit Mailman's scripts.  I.e., 10x normal or greater.

So, part of my point is that, yes, as a Mailman listadmin you _are_
exposed to a rain of spam you would not see as a mere subscriber, if
only because you get incoming smaug-owner at lists.svlug.org mail.  There
are only two possible ways to reduce what you get:

1.  Improve spam-rejection at lists.svlug.org.  Will happen when/if
    the machine gets migrated to a replacement host, not before.

2.  Improve spam-rejection at _your_ MTA.  Because you are reliant
    on an ISP's MTA and don't do your own mail, you're sort of screwed
    in this area, because you cannot administer Sonic.net's SMTP
    practices.  By contrast, my own MTA's spam-rejection is really,
    really good -- with the result that my system refuses not only
    almost all forwarded mail from lists.svlug.org but also most
    advisories of held spam.  Actually, this is a mild nuisance, 
    because SVLUG's Mailman increases my subscription's "bounce score" 
    every time my MTA refuses a notice about held spam sent to an
    admin address, and tells me about every other week that it's 
    disabled delivery on account of "excessive bounces" (but I 
    fix that each time by visiting the indicated URL to reenable

There _is_ a Mailman posting we can tweak, to reduce the amount of
nuisance mail Mailman sends you (and me, though my system refuses 
almost all of it) when items are added to the admin queue.  It's on 
the http://lists.svlug.org/lists/admin/smaug  page:

   Should the list moderators get immediate notice of new requests, as
   well as daily notices about collected ones?    [ ] No  [x] Yes

Help screen elaborates:

   List moderators (and list administrators) are sent daily reminders of
   requests pending approval, like subscriptions to a moderated list, or
   postings that are being held for one reason or another. Setting this
   option causes notices to be sent immediately on the arrival of new
   requests as well.

We can change that from "yes" to "no", with the result that you (and I)
will receive only _one_ notice, daily, telling us about held postings
and any other administrative items awaiting review.

The drawbacks are as follows:

1.  If there's legit mail held for review, on account of being sent 
from a non-subscribed address, or implicit-addressed (Bcc'd to the
list), or too large message (usually an attachment), or too many
recipients (usually a big CC list), then you won't see it for on average
half a day.

2.  The daily summary will have only information about the queue that
so non-detailed (something like "There are 15 items awaiting review")
that you cannot tell whether any are non-spam.  You'll thus need to 
visit the admin Web pages to see if any are _not_ junk.  

I find reason #2 damning.  Under the present system, any non-spam held
mail is immediately obvious from the held-mail notice -- and I just
immediately delete all other notices, it being super-obvious that they
concern held junkmail.  Thus, I almost never need to visit the mailing
list's admin-queue Web pages -- except once in a blue moon when there's
a legit mail in it.

Changing the described toggle would eliminate that advantage, and I'd
need to visit the Web pages almost daily, instead of almost never.  So,
I personally consider the current setting a lot less work.

But I'll leave it up to you, whether you want to try that toggle.

[1] Bogging down Mailman performance is one reason why it's a bad 
idea to add forged spam-sender addresses like "fpasgbg at bramley.com"
to that roster.  An additional reason is that doing so is utterly
pointless anyway, because those senders are semi-randomly generated and 
are unlikely to ever be used again.

----- End forwarded message -----

More information about the web-team mailing list