[volunteers] So, this happened

[Rick Moen]

Quoting Sarah Newman (newmans at sonic.net):

> Is there anything particularly special about any of the software
> versions in use? Or just nobody has had the time to rebuild?

There's IMO nothing _desirable_ about the software versions in use, but
IMO there is something (bad) special about them.

Long ago, the system architect used beta, non-packaged versions of
Exim4, Mailman, sa-exim[1], and SpamAssasin, if memory serves.  (Above is
from memory, and I'm not taking the time to verify from the specific
contents of the chroot.)  He also did serious but somewhat eccentric
work on the Exim and SA antispam configurations, which thus likewise
ends up being a one-off.  The result was (in my view) a
special-snowflake machine that was difficult to safely upgrade, and some
tentative measures to do so seemed to portend fearful immediate results
if they proceeded, leading to a permanent state of deferred maintenance.

(Gory details are deep in this mailing list's archives, and would take a
while to find.)

The widely held perception, which I think is correct, is that the only
way forward is a from-scratch rebuild of everything that's currently in
the chroot.  And then, preferably, the new system would rely maximally
on distro-packaged software rather than... stuff from upstream tarballs,
preferably automatically buildable and everything under configuration
management.

One detail I'll elaborate on (below) is the mailman-owner mailing list
archive, which very likely was the lion's share of that 11GB I cleared
away last night.  (Doing system recovery on an urgent site-down basis
when you're already very tired, it's past midnight, and you have had two
glasses of wine is not recommmended, but I had little alternative after
finding that the reason my meeting announcement wasn't approvable via
the Mailman admin WebUI was that the root filesystem was full.)

Again, long ago, the system architect made the... odd decision to make 
Mailman's internal administrative mailing list mailman-owner the
listadmin/owner of all of the substantive (i.e., other) Mailman lists.
One result of that is that listadmins receive held-message and
administration-needed notices only if they subscribe to mailman-owner,
as I and Tim Utschig (last I checked) do.  Another consequence is that
the mailman-owner cumulative archive accumulates a metric fsckton of
notices recording the arrival and admin-queuing of spam that makes it
past Exim4's MTA-level spam defences.

The latter concern causes the mailman-owner archive to grow like
Topsy (to use the antique 1852 literary reference), with the full
contents of every spam making it past Exim4 getting stored (in 'this is
being held for approval') form) three times:  once in the cumulative
mailman-owner.mbox file, once in the HTML-format archive, and once in
the ASCII-format archive.  Last night, I saw that there were 16 years of
such accumulation, i.e., the archive started with a 2002 held-spam
notice.  Which means, very likely, I whacked off the 1998-2002
accumulation back in 2002, and just hadn't done so in a very long time.

No normal site uses mailman-owner in this fashion, and I've never seen
the same thing done anywhere else.  It should be understood, however,
that this odd decision was doubtless not problematic at the time it
occurred, probably in 1998, because the spam war was not yet in full
swing.

Whacking off the mailman-owner archive entails:

$ su -
# su - mailman
$ cd /var/local/mailman
$ bin/arch --wipe mailman-owner
$ exit
# exit
$

If I had not been very tired, and it past midnight, and my having had
two glasses of wine, I'd have kept more detailed records.

[1] I should be clear that the system architect is a valued friend, 
very talented and generous sysadmin, and author of sa-exim, so I'm sure
it made a great deal of sense to him to use his latest and best
development code, and I'm sure he had his reasons for the other
architecture decisions that made sense at the time.