[svlug] kernel.org breach, four years later

Rick Moen rick at linuxmafia.com
Sat Nov 21 03:10:28 PST 2015

Quoting Chaiken, Alison (alison at she-devel.com):

> There is no need for air-gap security when source code is managed with
> git.   The reason is that all up-to-date copies of a git repository
> are identical, and all patches submitted to the kernel are signed with
> the developer's cryptographic key, as well usually with the keys of
> the maintainer who merged the patch.   Thus copies of a git repository
> are verifiably correct and there is no need for special safeguards.

You mean copies of a git repository's changesets are the same as they
were upon submission, i.e., they were accurately conveyed from repo to
repo, and are cryptographically signed (SHA1-hashed) to catch errors or
meddling.  _However_, that does not in any way mitigate the risk of bad
new changesets being introduced by, say, a developer's local coding
workstation having been root compromised and being under someone else's

Those are two different threat models.  git's SHA1 hashing addresses the
first threat model, but not the second.

Torvalds restricting his core source development to an airgapped machine
addresses (at least, as to him) the second problem.

> > Bottom line, folks were downloading distributed kernel software from
> > a compromised server for 17 days. Ouch!
> There was little potential for harm, as outlined above.

Actually, no, there _very much_ was quite _serious_ harm potential --
for people fetching kernel source via old-school tarball download,
instead of using git checkout.  And this is a _big_ issue.

People downloading tar-bzip2ed / tar-gzipped kernels from kernel.org
during those 17 or more[1] days -- as opposed to doing 'git clone
linux-git' -- probably[2] got, to my knowledge, dangerously defective,
deliberately sabotaged kernel source tarballs.  Or rather, I suspect
they did.

Because of the almost complete absence of forensics information about
the August 2011 compromise, we do not know this for certain, but I'd
think it likely enough that the download directory was sabotaged, that
I'd bet money on it.  So, if that's true, then thousands of members of
the public downloading kernel and other kernel.org-published tarballs
(potentially[2]) got sabotaged source trees.

So, yes, Alison, it's true and quite reassuring that people getting
their code by using modern source-control management software for checkout
were protected through storage and vetting of hashes, and this is yet
another good reason to prefer git, etc., but that's not enough, because
those who weren't using git, remained at risk.

when they discovered the breach.  This pisses me right royally off,
actually, because basic system administration, and basic protection
of the public, requires it.  The kernel.org admins knew that a large
number of their machines had been rooted, and I'm pretty sure the
knowledge was sent only to the 'users at kernel.org' e-mail alias (and
certainly not to the general public), and to nobody else, for two
additional days, until the actual public announcement on August 31st.
So, the rest of us weren't deemed worth telling, nor protecting by
pulling the plug.

That is Rule Zero of root compromise:  You pull the plug.  Forensics and
recovery later, but first you kill power on the affected hosts.  (Not 
'/sbin/shutdown -h now', as such may trigger planned badness.  You yank

Not doing that puts people at risk.  In this case, it put _us Linux users_ 
at risk.

Here's a pastebin (leaked copy) of the August 29th very restricted e-mail 
by John H. 'Warthog9' Hawley, chief kernel.org sysadmin:  
You can sense his extreme fatigue, even at a distance of four years, so,
dude, I feel your pain.

(Does anyone know which roles kernel.org machines hera, odin1, demeter2,
zeus1, and zeus2 served in August 2011?)

The Rick Report (since our kernel.org friends are still keeping hush-hush)

Entry started (predictably) using a compromised developer's ssh
credential.  Since H. Peter Anvin's machine personal colo machine was
the first found to be compromised, it could have been his, but might
have been anyone's.  Anyway, some chain of stolen ssh keys was followed
to get into kernel.org machines hera, odin1, demeter2, zeus1, and zeus2.
Then the interesting part occurred, the local escalation to root.

Since Linux Foundation did a, frankly, sucky job on this matter and have
done a miserable job of being publicly accountable, e.g., spiking the 
long-promised forensic autopsy, I'm going to go out on a limb and
speculate that _The Register's_ Dan Goodin was correct on August 31st,
2011, here:

  Linux kernel maintainers didn't respond to an email seeking comment
  for this story, but two security researchers who were briefed on the
  breach said the infected systems were hit by a self-injecting rootkit
  known as Phalanx, variant of which has attacked sensitive Linux systems
  before. [link]

Quotation is from
http://www.theregister.co.uk/2011/08/31/linux_kernel_security_breach/ .

At this point, I must be a bit pedantic on the matter of security
terminology:  It is a misnomer to speak of a 'self-injecting rootkit'.
Like trojans and worms[3], a rootkit is (definitionally) not in itself a
means of entry.  A rootkit is a set of tools to hide the intruder's
presence, by replacing major system tools by ones modified so as to not
report the intruder's presence and processes, replacement versions that
the intruder installs _after_ system entry and compromise of root

'Phalanx' _is_ a rootkit, but one that was also _bundled_ with canned
attack code that exploited a particularly bad, embarrassing design error
in 2.6 Linux kernels of the day:  According to the Phalanx coders, Linux
device file /dev/mem was, at that time, permitted to read and write _all
memory_ -- including the image within memory of the running kernel.[4]
Quoting the README:

  You are reading the "READMEFILE" for the brand spanking new
  PHALANX rootkit. Phalanx is a self-injecting kernel rootkit designed
  for the Linux 2.6 branch.

  It uses /dev/mem to inject hostile code into kernel memory and hijack 
  syscalls.  AFAIK, this is the first rootkit to utilize the /dev/mem 
  interface.  So, why /dev/mem and not the traditional and considerably 
  more handy /dev/kmem?  It was recently ruled out as a "security risk",
  and has therefore been disabled by default on recent linux kernels. 
  Somehow the retarded 10-year olds who maintain the Linux kernel felt 
  that being able to access kernel memory in a device is a risk, but 
  having another device that can access ALL MEMORY, including kernel 
  memory, is fine.

Yeah, you read that right.  Effectively, those 2.6 kernels had about the
same level of local security as did Windows 3.11 for Workgroups.

You can download Phalanx's beta 6 here:
Notice that it's from _2005_.

If Goodin and the two security researchers he consulted were correct,
then our best-and-brightest folks were reportedly rooted in 2011 by a
six-year-old attack tool that exploited a stupid memory error.

It was therefore pretty trivial, with kernels of that era, to write an
attack tool to use /dev/mem to hijack system calls, which, therefore,
when run locally, basically completely broke system security and gave
immediate root access via an included 'phalanx client' backdoor utility.
The bundled rootkit then got installed to hide this unauthorised
presence.  The intruders would then logically spend their time of
concealed godhood gimmicking everything else they could (e.g.,
corrupting the downloadable tarballs), and (according to Hawley)
replaced the compromised hosts' OpenSSH daemon and clients on August
17th, 2011, with ones that thereafter stole developers' ssh credentials
whenever they used those on the compromised hosts to ssh/scp.

There you have it, my best attempt at reconstructing the real story.

And yeah, I'd call that embarrassing.  

o  Making /dev/mem about to access _all memory_ was embarrassing.
o  Doing nothing about this ghastly error for _six years_ after
   it became automatically exploitable was embarrassing.
o  Not pulling power upon receipt of proof of root compromise was 
o  Not bothering to inform or protect the public for two 
   additional days was embarrassing.
o  Promising a forensics report, failing to deliver one for four
   years, and leaving it to the likes of me to try to discover the
   truth, is kind of sad and embarrassing.

Some of those items are 100% certain; the rest are a pretty good guess
on my part.

[1] Intrusion occurred _no later_ than August 12, 2011, but maybe
significantly earlier.  It was discovered only on the 29th.  So, at
_least_ 17 days until _discovery_ of the root compromise, and at least
19 days until announcement to the public. 

[2] Only the custodians of the root-compromised machines can say whether 
downloadable kernel source tarballs were also security-compromised, but
this would be a logical target of any kernel.org intruder.  E.g., on
January 21, 1999, ftp.win.tue.nl at Eindhoven University was
site-compromised and downloadable tarballs of Wietse Venema's key TCP
Wrappers package were trojaned.  Around July 30, 2002, the OpenBSD
Foundation's ftp server was site-compromised and OpenSSH 3.2.2p1, 3.4p1,
and 3.4 development source code was trojaned.  On September 28, 2002,
ftp.sendmail.org server was site-compromised and downloadable source
tarballs of sendmail 8.12.6 were trojaned.  On December 13, 2007,
www.squirrelmail.org was site-compromised and downloadable source
tarballs of SquirrelMail 1.4.11 and 1.4.12 were trojaned.  On November
8, 2010,  ftp.proftpd.org was site-compromised and downloadable tarballs
of ProFTPD 1.3.3c were trojaned.  On June 30, 2011, vsftpd.beasts.org
was (somehow) entered and downloadable tarballs of vsftpd 2.3.4 were
trojaned.  In August, 2011, would kernel.org's downloadable Linux kernel
source tarballs have been judged worth trojaning?  I rather think so. 

[3] Covered here:  http://linuxmafia.com/~rick/faq/#virus5.

[4] Somewhere in, I'm pretty sure, a later part of the 2.6 series,
CONFIG_STRICT_DEVMEM started making the kernel check addresses in
/dev/mem with devmem_is_allowed(), restricting access to the first 1MB
of RAM only, thus closing this hole.  I note that my 2015 systems 
have /dev/mem readable only by user root and group kmem, and writable
only by user root.

Cheers,                           (morganj): 0 is false and 1 is true, correct?
Rick Moen                         (alec_eso): 1, morganj
rick at linuxmafia.com               (morganj): bastard.
McQ! (4x80)                                     -- seen on IRC

More information about the svlug mailing list