[volunteers] Fwd: [web-team] Linode Support Ticket 5649987 - Critical Xen Maintenance / Reboot Schedule

Rick Moen rick at linuxmafia.com
Sat Apr 30 23:01:21 PDT 2016

Daniel just called my attention to his reply to this thread, which I
hadn't noticed because it was way back (though it was indeed marked New
in my mailing lists mbox).

I'm going to deliberately break threading (by removing the In-Reply-To:
header) to make it more likely I'll see any follow-on traffic.

Quoting Daniel Gimpelevich (daniel at gimpelevich.san-francisco.ca.us):

> On Sun, 2015-12-13 at 18:36 -0800, Rick Moen wrote:
> > Micah Dowty originally (2006) had the host run other stuff:  snmpd,
> > svnserve, inetd.   Over the years, I've carefully pared down the
> > bloat,
> > and fixed a number of sysadmin errors and omissions.  (For example,
> > svnserve wasn't actually needed, but was enabled somewhat lazily by
> > someone (er, Lisa), so I turned it and xinetd off, realised our svn
> > setup broke, fixed the svn serup to no longer need svnserve, and then
> > everything was happy.
> Almost: Under the old svn setup, check-ins from any authorized user were 
> communicated to svnserve, which recorded them while running as the "svn" 
> user.

The current workflow does not rely on svnserve running as a 
_network daemon_, which is what it did when Lisa first enabled it and was
the reason why I found it and xinetd among the running processes.  Just
trying to figure this out on the fly:  I'm guessing you mean that
svnserve is still involved even if its good offices as a network daemon
aren't required -- and that's related to the privilege problem you speak
of.  Oy.

svn can do several access methods:

file:///   direct repository access (on local disk)
http://    Access via WebDAV to Subversion-aware httpd
https://   [obvious]
svn://     Access via custom svenserve daemon
svn+ssh:// Tunneled over sshd

As I understand it, Lisa expected that remote users would use the
'svn://' access protocol, something like this:

svn checkout svn://www.svlug.org/var/svn/web/

Her assumption that everyone would be using the 'svn://' access protocol
created the requirment that daemon svnserve (managed by xinetd) be on
tap to handle incoming connections.

I thought, wait, it's kinda dumb to run two additonal network daemons
(svnserve + xinetd) when there's an altnerative, so I turned off the
extraneous network daemons and verified that this substitute workflow

  mkdir yourlocaldirname
  svn checkout svn+ssh://www.svlug.org/var/svn/web/ yourlocaldirname

make changes in local working area.  Then:

   svn ci -m "Checkin comments" [file1 file2...]

But honestly, I cannot really remember anything.  Above is partly based
on the docs that I wrote at the time, site-docs/svn-instructions, which
someone last touched in 2013 and I'm pretty sure I myself haven't really
done anything to since the 2000s.

And yes, I really haven't checked anything into that svn repo in a very
long time, though it _did_ work fine last time I did.

Back in those days:

1.  We were trying to work with super-small amounts of RAM and virtual
disk:  80MB RAM and, IIRC, 3 GB disk.  I'd done everything possible to
keep both process sizes and disk footprint tiny.  And I couldn't help
noticing that the svn repo was very, very bloaty on disk usage.  I
worried that the repo would eat up our disk.  

Don't forget, at that time, the aim was to migrate onto Linode all of
the remaining lists.svlug.org functions, including the enormous Mailman
archives.  So, I was doing my best to conserve disk any way possible.

2.  I also thought, this is stupid: Why are we having the svn repo on
the production host, anyway?  The repo should be elsewhere, because it's
a repo!  Your version control repo should _not_ be on the deployed host.
That's a SPoF to begin with, and also the repo should be on a host with
very high reliability and security, i.e., not a frelling Web server.

3.  I _also_ thought, there's a shitload of files within the system HTML
tree, like the var/www/svlug-main/prev subtrees holding prior speakers' 
presentation slides and such, that are big-ass binary files that make _no_ 
sense to be in a version control repo, especially on a host that has (or
had at that time) very limited disk space.

4.  As a temporary measure, I _thought_ I had scripted backup to offhost
of five critical things:

o snapshot of /etc
o output of dpkg --get-selections
o contents of the /root/sensitive tree
o contents of the site-docs tree /usr/local/src/site-docs
o contents of the /var/www/svlug-main tree (system HTML root)

FYI, all but the last item are tiny, and the last tree (system HTML) is
about 127MB gzipped.

It turns out, I had never scripted that backup.  (Eek.)   That's what
comes of too many crises and nobody helping for long periods.  Instead
of setting up a script, turns out all I did was a one-time backup of all
the above items as they were as of 2013-04-11.

I still haven't scripted it, but I've just done _another_ manual backup
of all of those to linuxmafia.com.  (Um, technically, my backup of
/var/www/svlug-main is at this writing 59% transferred to my server,
because of, y'know, my having slow service over the aDSL to my house.)

A proper scripted backup would use rsync to save time/bandwidth.  At the
moment, just to get a spot-backup done, I made tarballs and am just
scp'ing them over.

IMPORTANT:  Before doing any substantive work including package
maintenance on www.svlug.org, it would be wise to do a system backup of
some sort.  (With my spot-backup that is just now completing, I believe
I could reconstruct the system if it blew up with no lossage, but you're
advised to take your own preventative measures, too.)

> Under the new setup, check-ins require root privilege (a local
> bug) with "root" as the only recorded contributor, because the
> now-unused "svn" user still owns the repository. I suspect that the
> long-ish list of edits that have not been checked into version control
> might be related.

I really have no idea.  I don't know a thing about the svn user, really,
except whatever's in the svn-intructions file is what I used to know.
Frankly, everything else uses git, and I haven't even done anything at
all with svn anywhere since last decade.

At the time Lisa picked svn, that was entirely her prerogative, but if
I'd been asked, I'd have said 'I'd rather we use git', just as if Micah
Dowty had asked anyone before picking Ubuntu Server, I'd have said 'I'd
rather we use Debian'.  Talking things over in advance before major 
system decisions is something I'd prefer everyone do -- myself included,
of course.  Less unilateralism followed by 'Oh, BTW, there's a thing I

> Completing the fix _should_ simply be a matter of: chgrp -R www-data
> /var/svn/web chmod -R g+w /var/svn/web

Could be.  You could try that.  Warning:  I really don't know much at
all about svn internals, and never did.

Please note that site-docs/TODO has a note that the _intention_ of
Lisa's design was that a nightly cron job would autopopulate the
Lighttpd system HTML tree (/var/www/svlug-main) by automated checkout of
the head versions of all files from svn.

site-docs/svn-instructions also has interim suggestions to manually
chown files before checkin (while using /var/www/svlug-main as an svn
working directory).  Which please see, too.

If you rejigger the svn workflow and make, in your view, things that you
believe broken work, would you please do a corresponding rewrite to
site-docs/svn-instructions?  Posterity would be grateful.  ;->

Also, not that I think you're likely to do so today, but rather because
this once did come up, please note that I'd really rather you not move
directories around because, e.g., you prefer /srv , especially not
leaving forests of symlinks around in the old locations.  Once, some
years back, when you did that, I had to do a lot of work figuring that
out and putting things back.  

The biggest problem was the symlinks, which are a long-term nightmare,
and I'll go to some lengths not to have to deal with a bunch of those
again.  I could detail why, but suffice it to say that lots of symlinks
become a major maintenance problem, in addition to a small performance
hit.  I had no problem as such with your sudden attraction to the
most-fashionable FHS opinions about where things should be, _but_ I'd
personally have not made such a change without checking with other
people on the Web team, first.  And you didn't:  You just moved them
first, and then pronounced it as 'I did this' afterwards.

Though, to be honest, changes like that (moving subtrees around) ought
to have some significant advantage beyond 'FHS says so', _because_
there's value in subtrees being where people already know them to be,
and expect them to be.

Also, please describe all significant changes with the date and your
initials in file site-docs/ChangeLog.  Please, please! Thanks!

More information about the volunteers mailing list