[volunteers] Repaired mailing list mbox backups

[Rick Moen]

When Sarah did her heroic job of moving lists.svlug.org to a chroot
environment on a prgmr.com Xen virthost, just barely ahead of Via.net
ejecting the old machine from their colo, she kindly left me know that
she'd _not_ carried forward the ssh key I put in place to back up
Mailman's archives, correctly anticipating that the chroot directory
offset would temporarily break my backup script, and I'd want to fix the
script before running it again.

I thanked Sarah, and said I'd have to get to that later, as too much was
on my plate.  From then until today, backups have been kaput.

Today, I got around to fixing the script.  This message explains how it
works.

Mailman keeps two sets of records that matter.  One is the membership
rosters.  These get e-mailed out every Sunday.  That cron script
continues to work fine.

The other is the cumulative mbox files.  On each of the seven mailing
lists, Mailman appends each new arriving message to a gradually growing
mbox file holding all traffic the mailing list has ever received.  The
one for the mail svlug mailing list now is large and holds thousands of
messages, going back to 1997.  Each mailing list's archive is in a
separate mbox.  Mailman generates the ASCII and HTML versions of the
back-message archives using the mbox, which thus is the primary record.

Backing that up requires (absent some major screwing around) use of root
authority, which when used across an ssh channel can be a major security
risk to the host at the calling end, the host at the called end, or
both.  My compromise was to create on linuxmafia.com, using ssh-keygen
run as the root user, a RSA-type ssh public keypair.  The public half
got listed in /root/.ssh/authorized_keys on lists.svlug.org, and also put
as a file in /root/.ssh.  The private half remained as file
/root/.ssh/svlugbackup on linuxmafia.com, where the script runs daily.

The script is this line in linuxmafia.com's /etc/crontab:

3 15    * * *   root    rsync -e "/usr/bin/ssh -i /root/.ssh/svlugbackup" -ax "lists.svlug.org:/var/old-svlug-rfs/var/local/mailman/archives/private/{jobs.mbox,officers.mbox,smaug.mbox,speakers.mbox,svlug-announce.mbox,svlug.mbox,volunteers.mbox,web-team.mbox}/" /usr/local/src/rickstuff/svlug/mboxes


I just now added the '/var/old-svlug-rfs' prefix to the path, that being
the new chroot offset.


When Sarah moved the host to the chroot facility, the ssh host key
changed, which is what most immediately broke my script, as ssh balked
at the changed host key every day when the script ran.  To cure this
problem, first, one has to find and remove the _old_ key hash from
linuxmafia.com's /root/.ssh/known_hosts file.  The daily error said that
the 'offending' cached key was on known_hosts line 7, so I removed line
7.

Now, at this point, lists.svlug.org's (new) host key became _unknown_ to 
linuxmafia.com's ssh.  I ran the following as root on linuxmafia.com, to
grab and cache the host key (in /root/.ssh/known_hosts, again):

# /usr/bin/ssh -i /root/.ssh/svlugbackup lists.svlug.org

/root/.ssh/known_hosts thereby got a new line, and linuxmafia.com's
script would no longer balk on an unknown (or changed) host key.

Over on lists.svlug.org, the public half of the key used for backup was
still in /var/old-svlug-rfs/root/.ssh (the old /root/.ssh directory) as
file 'backup', so:

cp /var/old-svlug-rfs/root/.ssh/backup /root/.ssh

The public half had been registered in authorized_keys, so that likewise
was archived in the old tree, and could be copied in:

cp /var/old-svlug-rfs/root/.ssh/authorized_keys /root/.ssh

And the last step was to re-run the (now updated) /etc/crontab entry on
linuxmafia.com, to verify that backup now runs, which it does.