[volunteers] [revival] apache dead on svlug, restarting after this dumped state

[Rick Moen]

Quoting Daniel Gimpelevich (daniel at gimpelevich.san-francisco.ca.us):

> Moving from the out-of-band list back here.

You just posted, to a publicly archived mailing list, mention of and
a link to a deliberately private forum, by the way.  Everyone else until
now scrupulously avoided making that error.

For the curious, it is:  'Private list for parties' occasional work to
preserve and further SVLUG's operations and protect its ability to
continue operating.  It is also useful in cases when SVLUG's mailing
lists are unavailable for whatever reason.  This mailing list is not an
official SVLUG forum.'

An emergency alert went to that private forum when, following Daniel
screwing around on lists.svlug.org, its Apache instance died that night.
Which led to a followup discussion on the private mailing list.


> This tells us exactly what's being used from the original system. The
> exim is from 2005, which is pretty suboptimal....

But, FWIW, was developer tip source code at that time that Marc Merlin
hand-hacked and compiled.  So, 2005, but pretty advanced for 2005.  You
should do so well today, so to speak.

> but the apache is not only from 2004, it's from the 1.3 branch, which
> I doubt many other people run anymore. 

I certainly hope not.  And yet, you'll notice it's never been h4x0red, 
which is thanks to Marc's security-clueful configuration work, and
avoidance of stupid errors like running PHP.

You will probably have noticed that Mailman was 2.1.6 beta 4.  Again,
some significant amount of local customisation.


> I won't even bother to check what version of PHP is still hooked into
> that.

Um... we never, ever, ever have run PHP on lists.svlug.org.  This is one
of the reasons it never suffered a security compromise -- along with
Marc's manual application of the grsecurity and Pax patchsets.

> Proof of concept time: I've now gone ahead and installed lighttpd
> outside the chroot, listening on port 8080. I hooked it into the
> existing mailman instance with a few lines of C code.

Good work, and thank you.  Actually, let me back up and try again:
Excellent work, very impressive, and I would not have thought to try
that.

I also like that you made Lighty open to the desired Mailman listinfo
page rather than to a meaningless httpd document root.


If you don't mind, please document what you did (including the few lines 
of C code) before doing further anything, though.  I leave it to your
discretion where, but copying the basic idea we use on www.svlug.org
(the Linode virthost) couldn't hurt, and honours the Principle of Least
Surprise.  To show what I mean, here are dir listings on www.svlug.org,
and please note informational symlinks:

rick at gruyere:~$ ls -al /etc/skel/
total 20
drwxr-xr-x  2 root root 4096 May  3 00:13 .
drwxr-xr-x 82 root root 4096 May  5 16:21 ..
-rw-r--r--  1 root root  220 Apr 21  2006 .bash_logout
-rw-r--r--  1 root root 3486 Mar 28  2013 .bashrc
-rw-r--r--  1 root root  675 Apr 18  2010 .profile
lrwxrwxrwx  1 root root   24 Mar 31  2011 site-docs -> /usr/local/src/site-docs
lrwxrwxrwx  1 root root   19 Mar 31  2011 www -> /var/www/svlug-main
rick at gruyere:~$ 

rick at gruyere:~$ ls -al /usr/local/src/site-docs/
total 82
-rw-rw-r-- 1 www-data www-data 21401 May  4 11:29 ChangeLog
lrwxrwxrwx 1 www-data www-data    15 Mar 31  2011 SENSITIVE-DATA-DIRECTORY -> /root/sensitive
-rw-rw-r-- 1 www-data www-data   176 May  3 17:23 TODO
lrwxrwxrwx 1 www-data www-data    19 May 27  2015 lists.svlug.org-documentation -> svlug.xen.prgrm.com
-rw-rw-r-- 1 www-data www-data  2218 Nov 12  2013 new-user
-rw-rw-r-- 1 www-data www-data  7587 Jul 20  2015 nsd-instructions
drwxrwsr-x 2 www-data www-data  4096 May  2 01:05 obsolete
-rw-rw-r-- 1 www-data www-data  5359 May  2 02:29 package-operations
-rw-rw-r-- 1 www-data www-data   519 Oct  1  2015 socialnet-info
-rw-rw-r-- 1 www-data www-data  6819 Dec 24 18:10 svlug.xen.prgrm.com
-rw-rw-r-- 1 www-data www-data  4413 May  2 00:00 svn-instructions
-rw-rw-r-- 1 www-data www-data 11573 May  3 00:10 symantec-site-contact
rick at gruyere:~$ 

rick at gruyere:~$ sudo ls -l /root
[sudo] password for rick: 
total 8
lrwxrwxrwx 1 root     root        9 Mar 31  2011 SENSITIVE-DATA-DIRECTORY -> sensitive
drwxr-xr-x 2 www-data www-data 4096 May  2 12:51 sensitive
lrwxrwxrwx 1 root     root       24 Jul 21  2008 site-docs -> /usr/local/src/site-docs
drwxr-sr-x 3 www-data www-data 4096 May  7  2013 unused
lrwxrwxrwx 1 root     root       19 Mar 31  2011 www -> /var/www/svlug-main
rick at gruyere:~$

rick at gruyere:~$ ls -l /
total 80
drwxr-xr-x   2 root root   4096 May  3 02:37 bin
drwxr-xr-x   2 root root   4096 Jul 15  2011 boot
drwxr-xr-x   7 root root   8192 May  3 22:47 dev
drwxr-xr-x  82 root root   4096 May  5 16:21 etc
drwxrwsr-x  10 root staff  4096 Mar 20  2015 home
drwxr-xr-x  13 root root   8192 May  3 02:34 lib
drwx------   2 root root  16384 Jan 26  2005 lost+found
drwxr-xr-x   2 root root   4096 Sep 25  2004 mnt
dr-xr-xr-x 180 root root      0 May  3 15:46 proc
drwxr-xr-x   8 root root   4096 May  5 20:44 root
drwxr-xr-x  13 root root    500 May  5 20:42 run
drwxr-xr-x   2 root root   8192 May  3 02:37 sbin
drwxr-xr-x   2 root root   4096 Mar  5  2012 selinux
lrwxrwxrwx   1 root root     24 Mar 30  2011 site-docs -> /usr/local/src/site-docs
dr-xr-xr-x  13 root root      0 May  3 15:46 sys
drwxrwxrwt   2 root root   4096 May  5 20:39 tmp
drwxr-xr-x  11 root root   4096 Feb  8  2006 usr
drwxr-xr-x  13 root root   4096 May  3 15:46 var
lrwxrwxrwx   1 root root     19 Mar 31  2011 www -> /var/www/svlug-main
rick at gruyere:~$ 


Idea is that the most important places people need to be able to find on
a machine that is 99% a Web server are (1) the HTML document root, and
(2) the site docs.  So, system setup makes sure there are breadcrumbs 
in all users' homedirs (including root's), and in the root directory.
I didn't want anyone to be able to say 'I didn't document my work
because I didn't know where to put it.'

For the mailing list machine, I guesstimate three critical
locations that merit breadcrumb symlinks:  (1) site docs, (2) location
of the chroot, and (3) Mailman basedir.  Daniel, you agree?  Did I miss
any?

(I gratefully bow to your judgement on this, as you have recent
knowledge of this machine and I do not.)


> I'd like to invite folks to try it out and kick the tires. If it's
> deemed robust, apache can be decommissioned and lighttpd moved to
> ports 80 and 443, being "better, faster, cheaper" and with no more
> PHP.

There is no PHP on lists.svlug.org.  Never was.  Thank God.  But I like
Lighty well enough!  httpd performance is not a relevant concern, but 
security definitely is. 

On those grounds, we should run a small, simple, well maintained httpd
able to run Mailman's Python CGIs (which, remember, is this specific
httpd's sole function) and is distro-packaged.
https://wiki.debian.org/WebServers suggests natural candidates are
Lighty and nginx.  Both are widely used and well maintained.  Both are
in Debian-stable.  Eh, Lighty's fine.  ;->  Good choice.

(This also means only one httpd flavour to know how to administer.)

Back in the day at Linuxcare (1999) when we needed a small, highly
secure httpd to serve just images, we used Boa -- as did Slashdot.
However, these days Boa's long unmaintained.


> With that done, similar things could be done with the other services…

The tough one is the MTA.  We really cannot just switch to a default MTA
configuration, as the antispam is no good.  Also, I'm guessing we'd need
to move both the MTA and Mailman at the same time.  (Setting up a
parallel Mailman instance outside the chroot isn't hard.)

Er, I _guess_ we could run a second instance outside the chroot with the
port binding being to some other port besides 25/tcp.  This is something
you can set in Debian's dc_local_interfaces directive in
update-exim4.conf.conf .  That would permit semi-realistic testing
during construction.

Anyway, building a bespoke antispam setup is meticulous manual work.
One aid to doing so is J.P. Boggis's set of prepackaged configuration,
filter files, modifications, and instructions for the Exim4 MTA to make
it reject and eliminate spam.  I used it to construct my own antispam 
setup.  http://www.jcdigita.com/eximconfig/  It is not, unfortunately,
just a 'unpack this tarball and you're done' affair -- but EximConfig does 
carry out some of the work for you.