[svlug] Source Code Walk-through -- Analysis Tools

Karen Shaeffer shaeffer at neuralscape.com
Sat Nov 3 21:28:37 PST 2007


Hi,
Well, I think cscope is a really effective tool for working with the
kernel. I wouldn't think of using anything else for myself. One thing
is for sure -- you need tools to work with the kernel.

Thanks,
Karen

On Sat, Nov 03, 2007 at 09:54:27PM -0700, Kristian Erik Hermansen wrote:
> My Linux developer friend Peter Petrakis, who works at Stratus on
> fault-tolerant OS/driver hardening, gave some advice.  I think the
> presenter and attendees may find it useful :-)
> 
> You can see his LinkedIn profile below.  He used to work on AlphaLinux
> many years ago...
> http://www.linkedin.com/pub/3/790/645
> 
> 
> ---------- Forwarded message ----------
> From: Peter Petrakis <peter.petrakis at gmail.com>
> Date: Nov 3, 2007 9:42 PM
> Subject: Re: [svlug] Please read and respond - I suggest postponing
> 1st code walk-thru mtg
> To: Kristian Erik Hermansen <kristian.hermansen at gmail.com>
> 
> 
> Opengrok is about the best thing I've ever
> used,http://www.opensolaris.org/os/project/opengrok/. It's like using
> LXR but with syntax highlighting and better search capabilities. He
> should consider scripting his presentation using javascript or
> something similar. That or re-examine the whole code walk through
> think entirely and prepare the code he'll examine in advance.
> 
> Problem with these fancy data mining tools is sometimes they mess up
> and it takes you a couple clicks to find what you're looking for.
> That's OK when you're working in your office but could be awkward and
> stressful when you're on stage IMHO. The most rock solid and least
> sexy way to browse source code is using cscope. Also, sometimes the
> best way to understand a new feature is to look at the diffs. meld is
> really good for that.
> 
> Peter
> 
> On 11/3/07, Kristian Erik Hermansen <kristian.hermansen at gmail.com> wrote:
> > Any ideas on how to best browse Linux sources in front of an audience
> > for this guy who is going to be filmed @ Google doing Linux source
> > code walkthroughs?
> >
> >
> > Forwarded Conversation
> > Subject: [svlug] Please read and respond - I suggest postponing 1st
> > code walk-thru mtg
> > ------------------------
> >
> >  From: Darlene Wallach <freepalestin at dslextreme.com>
> > To: SVLUG <svlug at lists.svlug.org>
> > Cc: Leslie Hawthorn <lhawthorn at google.com>, Tiffany Griffith
> > <tgriffith at google.com>
> > Date: Sat, Nov 3, 2007 at 6:30 PM
> >
> > All,
> >
> > I have a concern that we are not ready to start
> > the Linux kernel code walk-through meetings. I
> > want to make sure we are putting the very generous
> > offer of Google to host us to the best possible use.
> > I also want to make sure we are making the best use
> > of our time to participate and have our meetings
> > filmed.
> >
> > I have not seen a syllabus/curriculum for the
> > upcoming Linux kernel code walk-through.
> >
> > I assert we should have a suggested
> > syllabus/curriculum, which specific kernel code
> > we will use, a specific editor, etc. so that
> > everyone is looking at the same code and seeing
> > the same line numbers - this would seem to especially
> > benefit people who will view via the video. What
> > else should be common for everyone? What am I not
> > listing?
> >
> > I suggest we have a link on svlug.org which lists
> > everything so that people who want to participate
> > can look there to see where we are - what we have
> > covered, what we plan on covering (in case they
> > want to come to a specific meeting).
> >
> > Are there people interested in the code walk-through
> > that want to propose different aspects? If different
> > people take on the tasks it won't all land in the
> > hands of one person. I assume most people have day
> > jobs, family/relationship responsibilities. So most
> > people will not have a lot of extra time on their
> > hands.
> >
> > Are there people who concur with my assertions and
> > suggestions?
> >
> > Are there people who offer to take on doing some of
> > the work?
> >   - which specific kernel should we use
> >   etc.
> >
> > Here is what I have seen posted in regards to what
> > to cover/do:
> >
> > Srihari Raghavan
> >     Sort of a junior-level Linux experience guy here.  I was wondering,
> > if there is currently, and if not, is there any interest...in having
> > intensive source code walk-through sessions for Linux kernel.
> >
> >      To get an idea of what I am talking about, some thing along the
> > lines of:
> > http://www.mckusick.com/courses/2006.advdescrip.html
> >
> > and in another post Srihari wrote:
> >
> > My 2c w.r.t suggestions...
> >
> >      a. Filming it for youtube is great, but it could be worthwhile to
> > do it after a few sessions, as that would give us time to make the
> > sessions more structured, settle down on topics, identify ways of
> > conducting these sessions etc.,
> >
> >
> >      b. Regarding topics, these are some thoughts...IMHO.  In general,
> > it would be good to understand "Design Patterns in Kernel
> > software"...how does Linux solve (in terms of in-depth data structures,
> > algorithms) the many well known aspects of OS/kernels....some examples:
> >          1. How does Linux work with diff. HW...essentially kernel
> > startup, machine dependent/independent code, working in a small
> > processing/memory footprint device, logical volumes etc.,
> >          2. Process creation/handling, system calls, interrupts,
> > interrupt handlers, BH handlers, kernel synchronization methods etc.,
> >          3. Scheduler - how does it handle interactive vs. CPU intensive
> > processes..O(1) scheduler using priority queues/red-black trees, process
> > synchronization etc.,
> >          4. Memory handling - slab allocator and other types, virtual
> > memory, memory alloc/dealloc tracking, handling leaks,
> > paging/segmentation etc.,
> >          5. Linux kobjects, sysfs, modules, driver interfaces etc.,
> >          6. Networking stacks, Filesystem internals, SMP,
> > virtualization...and other advanced topics.
> >
> >
> >      c. I personally would be fine with pre-meeting prep, but if the
> > group decides to go with homework or both, its fine by me.
> >
> >
> >      d. Typical kernel source code archives are
> >          http://fxr.watson.org/fxr/source/?v=linux-2.6
> >          http://lxr.linux.no/
> >
> > ====================================
> >
> > Andrew Wilcox
> > I have an old book that describes the Linux 2.2 kernel, "Linux Core
> > Kernel Commentary".  Maybe someone could write an update to 2.6 for
> > it.  It would seem like an interesting project (I personally don't
> > have the time for it right now, however).  It's a fantastic book -- it
> > covers some of the "stranger" parts of the kernel (i.e. SMP,
> > scheduling) in more understandable terms than just the source code.
> > It also contains a full print out of the whole core of the kernel
> > code, cross-referenced with the commentary.
> >
> > Here's a link to the book on Amazon:
> > http://www.amazon.com/gp/product/1576104699
> > Also, there seems to be a 2nd edition for the 2.4 kernel at
> > http://www.amazon.com/gp/product/1588801497 -- I do not personally
> > have this edition, but it may be worth looking in to.
> >
> > and in another post Andrew wrote:
> > Not sure if we might want to discuss the multiboot record or not?
> > Fairly trivial, but probably worth mentioning.
> > (this was in response to Bruce Coston)
> >
> > ====================================
> >
> > Akkana Peck
> > I don't mind pre-session assignments like that. But depending on the
> > module we're reading, it might help to have someone give some
> > background in what the module does if it's not obvious, so we're
> > not just blindly reading lines of code and trying to guess.
> >
> > It would be really cool to have homework like "Now try tweaking this
> > code to do foo" and see what people come up with and what problems
> > they have.
> >
> > That would be excellent if people wanted to cover different types of
> > modules.  Some might be more interested in scheduler details, some
> > in network algorithms, some in USB drivers. But that'll become
> > clearer after several sessions -- we're probably fine with one group
> > initially.
> >
> > and in another post Akkana wrote:
> > Several people have expressed interest in startup and init.
> > I'm not sure if there's a lot of interesting actual kernel code
> > involved there, but if there is, I'd be interested too.
> >
> > How the process table is managed might be another good place
> > to start (and we'd get a better understanding of things like
> > those pesky zombie processes).
> >
> > Specific kernel areas I'd love to learn about:
> >
> > - ACPI and power management, how suspend (to RAM or disk, but
> >    especially RAM) and ACPI sleep states are managed, and what
> >    happens on resume. And what's up with those unkillable daemons
> >    running in kernel space?
> >
> > - Udev.
> >
> > - The "completely fair scheduler" and how it can help desktop
> >    processes (and it would be interesting to hear ways that the CK
> >    scheduler might have differed, if anyone knows).
> >
> > - A USB driver or two -- device drivers are one of the areas where
> >    it might be most possible to contribute, and so many devices are USB.
> >    (Maybe choose some USB class where a lot of devices still don't
> >    have drivers, like webcams.)
> >
> > - The new wireless infrastructure, and analyze any specific wireless
> >    driver -- as with USB drivers, it's an area that needs help, so if
> >    we can learn how the system works, we might be able to contribute.
> >
> >
> > ====================================
> >
> > Larry Colen
> > I suspect that just doing
> > anything along these lines will have the very significant effect of
> > giving other people the idea of holding these code walkthroughs.
> >
> > I would like to suggest that at each meeting we ask someone to
> > volunteer to note down any potential bugs, or documentation errors.
> > Code walkthroughs are helpful for more than just people learning the
> > code.
> >
> > http://www.linuxjournal.com/article/4388
> >
> > and in another post Larry wrote:
> > I find that having context really helps me learn things like this, so
> > I think that a good place to start would be a high level overview. The
> > general structure of the kernel.
> >
> > Online supplements would be URLs of good places to look for info, and
> > maybe some faqs for etiquette on the lkml etc.
> >
> > For digging into the code, I'd probably want to start with a fairly
> > high level look at what happens when init() is called.
> >
> > ====================================
> >
> > bruce coston
> > Beginning should be easy, Boot - probably with GRUB as its the most
> > common method now. Or start just after that as GRUB is not the kernel
> > itself.
> >
> > ====================================
> >
> > Philip Martin
> > I think there's some interesting stuff there.  All the PCI bus
> > initialization, which would provide a nice segue into device driver
> > loading and ACPI.  The whole init process (as in /sbin/init) is
> > outside the kernel, but would be a great place to start looking at the
> > linux process lifecycle, process table and scheduler.
> >
> > Maybe we should group things into umbrella concepts which can be
> > covered in a number of sessions.  Each session should be as atomic as
> > possible, however, to enhance retention.  Something like:
> >
> > major kernel data structures (atomic_t, kernel_size_t, etc),
> > conventions (locking), utility functions, etc.
> >
> > booting and device discovery (ACPI, bus initialization, device
> > discovery, device driver loading, udev)
> >
> > process management (process lifecycle, process table, process scheduler,
> > etc)
> >
> > device drivers (general framework, specific types, etc)
> >
> > VFS (VFS layer, IO scheduler, ext2, ext3, etc)
> >
> > VM
> >
> > etc.
> >
> > The progression should be logical and should hopefully build upon what
> > has come before.
> >
> > ====================================
> >
> > Brian J. Tarricone
> > is there a need to standardise on a
> > particular version that everyone will follow?  Presumably 2.6 will be
> > covered, but, for example, 2.6.12 is very different from 2.6.23, since
> > active development is occurring on the 2.6 series.  Things like the
> > recent addition of a new CPU task scheduler, a new default memory
> > allocator, and even many of the VM changes occurring early in the 2.6
> > series might make for some confusion if people are on different versions.
> >
> > _______________________________________________
> > svlug mailing list
> > svlug at lists.svlug.org
> > http://lists.svlug.org/lists/listinfo/svlug
> >
> > --------
> >  From: Tejas Kokje <tejas.kokje at gmail.com>
> > To: svlug at lists.svlug.org
> > Date: Sat, Nov 3, 2007 at 7:13 PM
> >
> > [Quoted text hidden]Darlene,
> >
> > For curriculum, how about following chapter order of one of the published
> > kernel books like "Understanding the Linux Kernel" ?  We need not to follow
> > it strictly, but it can always act as a guideline.
> >
> > As for kernel version, how about 2.6.23 since it has new CFS scheduler ? We
> > all will be on same page if we get same kernel. Something from kernel.org.
> > Like
> >
> > ftp://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.tar.gz
> >
> > If I understand correctly, this tarball is not going to change even when
> > future versions of kernels are released.
> >
> > For editor, I don't think it matters. Good old vi with ctags will do it (yeah,
> > emacs users ..flame me). Everybody is free to use whatever they like.
> >
> > I strongly feel that we should meet on Tuesday. If not for kernel
> > walkthroughs, atleast to discuss about the issues you raised. We can divide
> > responsibilites when we all meet. First meeting can remind us of first day of
> > course at school (only discuss logistics, course overview etc with dates for
> > midterms & finals :-D)
> >
> > I think discussing these issues asynchronously through mailing list might not
> > be the best way and end up taking more time.
> >
> > Everybody else, chime in.
> >
> > Tejas Kokje
> > [Quoted text hidden]
> > --------
> >  From: Akkana Peck <akkana at shallowsky.com>
> > Reply-To: svlug at lists.svlug.org
> > To: svlug at lists.svlug.org
> > Date: Sat, Nov 3, 2007 at 7:36 PM
> >
> > Darlene Wallach writes:
> > > I have a concern that we are not ready to start
> > > the Linux kernel code walk-through meetings.
> > [ ... ]
> > > I have not seen a syllabus/curriculum for the
> > > upcoming Linux kernel code walk-through.
> >
> > I'm not too worried. I'm very much aware this is a volunteer thing
> > thrown together very quickly -- I'm guessing Paul is still trying
> > to decide what's best to do first, and how fast we'll be able to
> > follow.  It might take a few meetings to figure that out.
> > I'm all for jumping right in and learning some basics while
> > we all figure out how these sessions should be run.
> >
> > Once things are rolling and there's a plan several weeks in
> > advance, I love your idea of having all the info on svlug.org
> > so people will know where we are and what to prepare for next.
> > I'm certainly willing to help with that.
> >
> > > which specific kernel code we will use,
> >
> > I do agree that we should decide on a specific kernel version
> > beforehand, so we don't have to spend the whole first session
> > fighting with wi-fi configuration, waiting for downloads,
> > configuring and building.
> >
> > Paul, is 2.6.23.1 (the latest stable version) a reasonable choice
> > for us to have ready on our laptops?
> >
> > > a specific editor, etc.
> >
> > Oh, now you're talking about restricting our choice of religion! :-)
> >
> > Seriously, I expect most people will use whatever editor is
> > most familiar to them on the privacy of their own laptops.
> > However, for future documentation on svlug.org, it might be
> > good to pick a favorite web source viewer, e.g. LXR.  I found
> > http://fxr.watson.org/fxr/source/?v=linux-2.6 for the git version, but
> > nothing that has 2.6.23. Are there any better search/display tools?
> >
> > --
> >     ...Akkana
> >     "Beginning GIMP: From Novice to Professional": http://gimpbook.com
> > [Quoted text hidden]
> > --------
> >
> 
> 
> --
> www.alphalinux.org
> del.icio.us/peter.petrakis
> 
> 
> -- 
> Kristian Erik Hermansen
> 
> _______________________________________________
> svlug mailing list
> svlug at lists.svlug.org
> http://lists.svlug.org/lists/listinfo/svlug
---end quoted text---

-- 
 Karen Shaeffer
 Neuralscape, Palo Alto, Ca. 94306
 shaeffer at neuralscape.com  http://www.neuralscape.com




More information about the svlug mailing list