[volunteers] Fixed ns1.svlug.org's secondary DNS service; distro upgrade needed

[Rick Moen]

tl;dr version:  1.  Our ability to do secondary DNS nameservice on
Linode virthost ns1.svlug.org AKA www.svlug.org has been somewhat
broken.  I'm pretty sure I just fixed it.  2.  A major system software
upgrade remains needed.


Item 1 of 2.  Fixing AXFR (zone transfer).

Right.  SVLUG's operations remain split between two hosts.[1]  The
antique and likely-to-fail lists.svlug.org host runs all our e-mail /
mailing list operations + Mailman's Web pages, + sshd + ntp.

The virthost www.svlug.org generously donated to the group by Linode
hosts SVLUG's Web pages (Lighttpd w/PHP5), svn repository, ntp, sshd.
It also does authoritative-only DNS nameservice (NSD 2.x software) 
to serve up SVLUG's domains as master/primary nameserver and (as a
courtesy) also do slave/secondary service for some people's domains, in
which functional capacity the host is known as ns1.svlug.org.

Last year, I realised that although ns1.svlug.org served up master and
slave nameservice with high reliability, it was failing to pick up 
changes at remote masters/primaries for the DNS zones it served in
slave/secondary capacity.  The domain owner would say 'Hey, Rick, why
isn't ns1.svlug.org updating to zone S/N 2012060100?  For some reason,
it's still serving 2012051500 data.'  I would go to ns1.svlug.org and do

 zonec -v nsd.zones  #compile all zones to the binary database format used
 nsdc reload

And then 'dig -t soa somedomain.com. @ns1.svlug.org. +short' would 
prove that AXFR had been pulled down, the zone had been recompiled and
reloaded, that thus S/N 2012060100 was now being served, and all was well...
until the next time a master/primary updated something.

Obviously, I hadn't fixed the real problem.  So, today, I poked around,
and eventually it occurred to me to look in the package's docs
directory.

ns1.svlug.org runs Ubuntu Server (not my choice; that was foisted on us
by then-VP Micah Dowty, who didn't care to administer the machine but 
had no problem making choices complicating the lives of those who did).
Currently, that's...

root at gruyere:~ # cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04.4 LTS"
root at gruyere:~ # 

It's set to track LTS but is still on 10.04 Lucid Lynx; newer LTS
release 12.04 LTS is out and mature (12.04.2 being out).  I'll get back
to that matter further down below.

/usr/share/doc/nsd/README.gz (provided by Ubuntu Server nsd package
2.6.7-3) says:


  3.3 Keeping in sync the secondary zones with ``nsdc update''

  Originally NSD was not designed to be run as a secondary server. That
  means that keeping and updating the local master zone files is outside
  of the scope of the package and should be done either out of band or
  with help of separate tools. However starting with 1.0.1-beta4 NSD
  provides such functionality to a limited extent.

  It makes no difference for zonec(8) and nsd(8) whether the master zone
  file was created and maintained locally or was obtained via AXFR/IXFR
  or out of band. However it is possible to specify the master servers
  in nsd.zones file by adding ``masters'' keyword followed by ip address
  of the master to instruct nsdc(8) update to update these zones with
  nsd-xfer(8) utility if necessary.

  ; zone name file [ masters ip-address ]
  zone nlnetlabs.nl secondary/nlnetlabs.nl masters 213.53.69.1

  Given the line above nsdc(8) update will invoke nsd-xfer(8) to check
  whether the zone was modified and if so will AXFR the zone from the
  master 213.53.69.1

  Due to the static nature of nsd(8) database it has to be rebuilt
  completely whenever one of the zones changes. Therefore it is
  recommended to do the database rebuilds at certain scheduled time, for
  example every hour.  nsdc(8) update will rebuild the database and
  reload the daemon if necessary.

  the follow cron(8) command will take care of synchronizing the
  database with the master servers:

  # nsd zone update every hour
  1       *       *       *       *       /usr/local/sbin/nsdc update


Our /etc/nsd/nsd.zones files _does_ have the 'masters [IP]' qualifier
data spoken of on the lines for which NSD does slave/secondary service, 
so it seems all that's missing is the regular rnning of 'nsdc update'.
So, I've now added this to /etc/crontab:

# nsd zone update every hour
1 *     * * *   root    /usr/sbin/nsdc update


Web-searching reminded me of a small cloud on the horizon, though:  
Now that I've finally fixed NSD 2.6.x, we need to get ready to throw it
away, because 3.x is current and 4.x is arriving soonish.

Anyway, the Linode host has a site-docs directory everyone maintaining
the box is supposed to consult and keep current.  So, I updated
site-docs/ChangeLog:

Mo 2013-04-07   Rick Moen <rick at linuxmafia.com>
           Wrote /etc/crontab entry to run '/usr/sbin/nsdc update' every
           hour as suggested by /usr/share/doc/nsd/README.gz's notes
           concerning AXFR access.  Essentially, this cronjob is needed
           in order to act on data received from DNS masters concerning
           available zone update information.

           Note:  Whenever we upgrade from NSD 2.6.x to the
           current 3.x series, there will be major changes inclduing
           entirely new tools with different binary names (e.g., 
           new nsd-control replacing old nsdc) and a new
           conffile format.  Also, I have a feeling we will have to
           make a conscious jump to a separate (newer) 'nsd3' package as
           opposed to the (older) 'nsd' package we're now using.




Item 2 of 2.  System software update.

I've held off from changing from 10.04.4 LTS Lucid Lynx to 12.04.2 LTS 
Precise Pangolin -- for a good reason.  This is Linode.  You have to be
really careful about upgrade or you can blow them up.  The main reason
is that Linode virthosts run on special kernels that are NOT managed by
the distro in the virthost.  In general, it is strongly advised to
consult Linode's own docs about how to proceed, and follow them
carefully.  Web-searching "linode upgrade ubuntu' finds the right page,
http://library.linode.com/upgrading/upgrade-to-ubuntu-12.04-precise

  Preparing to Upgrade

  Prepare your Linode for Ubuntu 12.04 LTS by installing available
  updates, backing up your Linode, checking your kernel, stopping
  services, and starting a screen session.

  Installing Available Updates

  You should install all available updates for Ubuntu 10.04 LTS or Ubuntu
  11.04 before upgrading to Ubuntu 12.04 LTS. Here's how:

  Update your package lists by entering the following command:

  sudo apt-get update
  Install the updates by entering the following command:

  sudo apt-get upgrade
  Any available updates for Ubuntu 10.04 LTS or Ubuntu 11.04 will be
  installed on your Linode.

  Backing Up Your Linode

  It's a good idea to back up your Linode before performing a major
  upgrade. That way, you can restore from backup if anything goes wrong
  during the upgrade process. If you subscribe to the Linode Backup
  Service, we recommend that you take a manual snapshot before upgrading
  to Ubuntu 12.04 LTS. If you use another backup service or application,
  we recommend that you make a manual backup now.

  Checking Your Kernel

  Verify that your Linode is using the latest supported kernel. See
  Applying Kernel Updates for more information.

  Stopping Services

  We recommend that you stop as many services as possible before upgrading
  to Ubuntu 12.04 LTS. This includes web server daemons (Apache and
  nginx), database servers (PostgreSQL and MySQL), and any other
  non-critical services. To stop a service, enter the following command,
  replacing apache2 with the name of the service you want to stop:

  sudo service apache2 stop

  Starting a Screen Session

  We recommend that you start a screen session to ensure that the updates
  will continue to install in the unlikely event you are disconnected from
  the Linode during the upgrade process. Here's how to install screen and
  start a screen session:

  Install screen by entering the following command:

  sudo apt-get install screen
  After installation has completed, start a screen session by entering the
  following command:

  screen
  If you are disconnected from your server, you can reconnect to the
  screen session by entering the following command:

  screen -Dr
  You are now ready to install Ubuntu 12.04 LTS on your Linode.

I have reached in effect the point where it says to back everything up, 
and not yet gone beyond that.



[1] I am often asked:  Why not migrate lists.svlug.org's functions 
over to the Linode host?  Simple, inadequate RAM and disk.

root at gruyere:~/site-docs # free
             total       used       free     shared    buffers
cached
Mem:        250492     205716      44776          0      18456
153784
-/+ buffers/cache:      33476     217016
Swap:       262140       4332     257808
root at gruyere:~/site-docs # df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda             7.7G  960M  6.7G  13% /
root at gruyere:~/site-docs #