[svlug] Katrina Relief Effort - Days 4, 5 and 6

[Chris Verges]

It seems like work comes in a two- or three-day push, since I can only 
find time to update my weblog every other night. This unfortunately 
means that many details are lost since things change so rapidly out here 
in the field. I'll do my best to recall the events of the past 48 hours.

On Saturday 9/24, we had to return to Hancock County on an emergency 
call from the Navy NOC. They said that the entire network was down 
again, and were requesting assistance immediately. Unfortunately, the 
Cisco group that was in the RV before us blew out the electrical system, 
so we had to spend a few hours to get that fixed. A very kind service 
crew on US-29 in Pensacola stayed an extra two hours after they normally 
quit to assist us. Jamie (the on-site manager for Cisco) called his boss 
and relayed this news, who in turn relayed it up the chain to John 
Chambers, CEO of Cisco. Chambers sent the message back down to get the 
service station's contact information, as he will be personally sending 
a thank-you for their help!

As we started to load up the RV around 2:30pm, I helped Jamie put one of 
the two Swe Dish satellite dishes into the truck. (He needed to ship it 
to Virginia on next-day delivery ... at 150 lbs, that would come out to 
be quite expensive!) While lifting that, I twisted funny and dislocated 
my top-most left rib, right under the shoulder area. Now I've felt this 
before and just tried to work through it, but since most of the 
equipment was very heavy, that proved to be impossible.

Phillipe Hoster, the engineer that I was relieving, and I went through 
all the satellite setup procedures so that I would know how to deploy 
Internet connectivity for the RV once we were on-site again. Immediately 
after that, around 6:30, I headed to the Sacred Heart Health System ER 
in Pensacola. Once I explained what I was doing and where I would be 
heading, Dr. Jeanne Madden -- one of the kindest people I have ever met 
-- did her best to rush my prognosis and get me back into the field. The 
staff at the hospital did an outstanding job, and I walked out of there 
around 8:45pm.

While on I-10 heading towards Mississippi, I got a phone call from Lt. 
Cmdr. Chris Gaucher of the Navy NOC. He just rotated in to command of 
the NOC, and wanted to touch base to find out our status. When I 
informed him that we were heading in to the area and would arrive in two 
hours to start diagnosing the problem. I thought it would be a late work 
night, but he said that the previous NOC staff had a 99.999% up-time 
expectation, and he understood that it was a harsh environment where 
even 90% up-time was damn good. So we got to relax more on the drive and 
had a good night's sleep on Saturday.

Sunday 9/25 started with me getting badged in at the Hancock County 
Emergency Operations Center (EOC). Afterwards, we headed to Pearlington 
to check on the Red Cross site and Phil's friend, Buster. Buster is 
between 7 and 10 years old, a local who is being raised by his 
grandparents, either of whom seem fit to be a guardian. Phil first met 
Buster when this grown-up child was shopping for his entire family at 
the distribution center in the camp. Since then, we all have a keen 
interest in making sure Buster makes it through this hard time with as 
much ease as possible.

I stayed at Pearlington to wire the site for 2 permanent IP phones on an 
outside kiosk, 1 IP phone for the doctor's office, and Internet 
connectivity for laptops and PCs through a Catalyst 3550 switch. This 
took a little longer than expected due to some physical wiring hang-ups, 
but I managed to get it done in 3 or 4 hours. Unfortunately, the Harris 
engineers who were responsible for setting up the new satellite dish for 
that site still haven't gotten the data portion of the feed working, so 
the site is limited to plain ol' telephone service (POTS) for now.

 From there, I went to the Navy Network Operations Center (NOC) based at 
Hancock Medical Center in Waveland, MS. Petty Officer (1st Class) Mark 
Mollere was having a wireless issue with some new mesh-based access 
points called /Breadcrumbs/. (DISCLAIMER: The Breadcrumbs are not Cisco 
gear, but rather that of a company known as Radjant.) The idea is that 
you can just distribute them in an area you want to light up with 
wireless access, and the individual devices form both client and bridge 
links on their own. They worked OK until the internal backup battery on 
them died. For some reason, when this battery dies, the device dies. :-\ 
So we're going through and replacing all of the deployed Breadcrumbs 
with Cisco Aironet equipment for a more stable network. To begin this 
process, I replaced a wireless bridge link in the northern wing of the 
building with an Aironet 1300 and a 5.2 dB omnidirectional antenna. The 
next phase of the project involves getting about 30 access points 
shipped from Cisco. (More on this later.)

On the morning of Monday 9/26, Mollere and I went on a full site survey 
of the areas supported by the Navy NOC. To give you a basic idea, here's 
what the topology is like:

ISP --- [satellite] --- Navy NOC
                           |
                           |
                       [802.16]
                           |
                           |
                       Waveland
                  Fire/Police Station
                           |
                           |
                +----------+----------+
                |                     |
                |                     |
            [802.16]              [802.16]
                |                     |
                |                     |
        223rd Engineering         2nd Street
            Battalion         Elementary School

*All 802.16 equipment is non-Cisco, and is used as a high-speed,
wireless backbone for the network.

There are several stories to tell from each site, and I could easily be 
typing for the rest of the night if I attempt to. The elementary school 
is definitely the most moving, as it is a food/water distribution 
center. People from all over the local community come there almost daily 
to get the basic supplies needed to live. At the fire/police station, I 
got to meet Eddie Favre, the mayor of Bay St. Louis and distant relative 
to NFL quarterback Brett Favre. When Jamie presented him with a couple 
of Cisco polo shirts, some Cisco baseball hats, and an IP phone system 
for the fire/police station, he started to tear. It's hard to think of 
the emotions the locals of these devastated areas are feeling, since the 
volunteers leave every two weeks or so -- the locals have to live it 
until it ends.

But the basic plan is to fully upgrade the Navy network with Cisco 
Aironets. We're going to be deploying the brand-new Aironet 1240 series 
at all sites, and extending the fire/police station's wireless coverage 
to encompass a senior citizen's center about two blocks away. (The 
actual technology used will be two very-high-gain antennae connected to 
Aironet 1400s.) Currently, the center has been running all 
communications through the director's cell phone, which can only dial 
calls due to damage incurred during Katrina. We're going to light up the 
entire building, providing them with a phone bank of wired IP phones and 
one or two wireless IP phones so the administration team can take 
communications to the bedside of those who cannot make it to the wired 
system.

The latter part of today was figuring out how to make all this possible. 
I came up with some Visio diagrams that listed what I hoped to 
accomplish, and itemized exactly which parts I need for each site. The 
request went in to Cisco management with Priority 1 flagging, but for 
now, we wait. I'm going to try and visit each site tomorrow and at least 
start running the necessary wired infrastructure so we have a jump start 
when the gear actually arrives. However, this shouldn't take too long, 
and will leave me with a day mostly without headaches.

As we were getting ready to head to bed tonight, we noticed the lights 
in the RV significantly dimmed. Given that we've had power distribution 
issues due to a CF from the original Cisco team, we were quite worried 
and decided to investigate further. It turns out that we were low on 
gas. :-\ In the course of two days, we had burned through almost 40 
gallons of fuel for the on-board generator! So Jamie and I had to 
disconnect all external moorings and head down to the FEMA fuel 
distribution site in Waveland to refill. The tank holds 80 gallons when 
full, just to give you an idea. Then I got the satellite link back up 
within 20 minutes, and the 802.11 wireless up in another 5 or so. One of 
the police officers who patrols the area walked by and commented on our 
sudden disappearance/reappearance act. I told him what happened, and 
then explained what the satellite dish was for when asked. He said, 
"Ya'll have free Internet?! Oh, I'm going to bring my laptop by here 
tomorrow!" Gotta love technology!

Chris Verges
-- 
chverges at cisco.com
408 525-0401