[svlug] Need help with home network numbers

Dagmar d'Surreal dagmar at dsurreal.org
Sun Jun 24 21:45:02 PDT 2001

On Thu, 21 Jun 2001, Dan Copeland wrote:

>   I use a Linux 2.2 box as a gateway and firewall (by which I mean packet
> filter) between my local network and the Big Bad World on the other side
> of my DSL connection.
>   I recently switched DSL providers and a few things have changed.  Previously
> I had everything working well - local machines were on 10.0.0.* addresses,
> as was the firewall's internal interface.  The external interface had
> the one static IP my ISP gave me.  Outgoing traffic was masqueraded and
> all was well.
>   Now, however, I want a few internal machines to have their own static
> IPs.  My new ISP has given me a block of 4 IPs - x.y.z.144-147 or x.y.z.144/28.
>  I'd like to give one to my firewall's external interface and one to
> each of three main user machines.
>   I tried to configure basic networking (packet filtering aside for now)
> so that the three machines can exist behind my firewall and became quite
> confused.  Am I actually creating a 4 IP subnet?  If so, don't I really
> only have 2 IPs usable for hosts?  Or do I configure each machine (firewall
> and hosts) as if it's alone on the DSL connection and make the firewall
> machine do promiscuous bridging?

Okay.  I understand your problem.  I've got 8 IPs from Speakeasy, and
they're bridged instead of routed, just like your setup.  FORTUNATELY this
gives us an excuse (heh) to use static host routes.  You will not be
losing any internet-capable IP addresses to the internal interface of the
router/firewall this way, either.  :)

To keep it simple, I'm going to outline what I set up here (excepting the
firewalling rules, since they aren't particularly tied to the problem
at hand) and explain the how and why of it.  Expanding this to multiple
IPs is not a problem.

The network topology I have in place here for a testing firewall (I'm
trying to write some web interfaces for the stuff) is this... (a fixed
width font would be necessary right here)

             |          (plain ol' ethernet)    
     |                                             |
     |e                                            |a
  +--+--+                                       +--+--+        +-----+
  |     |                                       |     |        |     |
  | FWa |                                       | FWb +--------| WWW |
  |     |                                       |     |b      d|     |
  +--+--+                                       +--+--+        +-----+
     |                                             |c
     |   (more ethernet)                           |   (yet more ethernet)
 ----+-------------------                  --------+------------------

I really don't have more descriptive names for the three network segments
than what are shown above, but the letters at the junctions merely where
IP addresses are being used.  "f" is the first gateway out to the internet
at the x.x.x.1 address.  I did not put the IP addresses in there, because
we're going to be dealing with both IP and MAC addresses, and I don't want
to make the diagram that huge.  I'm also working my way up in terms of
complexity/arcaneness.  Most people don't normally have to muck around
with arp tables like this, but it's *easy* once you get the hang of it.
The first three octets of the hosts involved have been obscured until
proven guilty of something to avoid nasty lawsuits (don't you hate it when
your machines sue you?).

Okay... The first bastion host (FWa) is the perfectly normal kind.  It's
sitting on x.x.x.223, quietly servicing nameservice for everything, mail,
and www, and masquerading a private network segment in the typical
fashion.  The only reason it's in this picture is because I think it's
important to point out that routing across the "plain ol' ethernet"
segment is pefectly normal stuff and can be expected to behave normally.  
This is where all the public IP addresses (.223-.230) lived until the
machines on the right were added.

The newer machine which is *only* a firewall (FWb, sitting with x.x.x.229
on interface a) has three NIC cards in it.  Interface c is the internal
private interface, which for all intents and purposes serves the private network masqueraded.  It's IP address is and
there's nothing fancy about it other than that.  Interface b connects via
a crossover cable straight to the web server, and interface b has the IP

The Web Server (WWW) uses the x.x.x.230 address, and all packets to it
must pass through the firewall.  We'll start the detailed bits here.
It's interface (d) is configured as if it were a /32... That is, the IP
address, network address, and broadcast address are all x.x.x.230, and the
netmask of course is  If you're doing static routing in
this fashion for more than one IP, you will need to put in host routes
for the other hosts on that segment... but heres the basic routing

Destination     Gateway         Genmask         Flags Metric Ref   Iface UH    0      0     eth0       U     0      0     lo         UG    1      0     eth0
(no one panic, yes, I deleted a field from the output)

The only abnormal thing that has to be done on WWW to support this is that
you have to place a *static* route for the host that is the gateway onto
the interface *before* you lay down the route that makes it the default
gateway...  Linux's ifconfig will fight you otherwise.  There's probably a
less messy way to do it, but at the end of my rc.inet1 is...

/sbin/route add ${GATEWAY} eth0
/sbin/route add default gw ${GATEWAY} netmask metric 1

Okay.  Now that the firewalled host that's getting routed statically is
able to talk to the internal (b) interface of the firewall without a
problem, we'll move on to the way the internal interface of the firewall
is configured...

I'm lazy, so I made it the normal /16, since it doesn't matter otherwise,
but the internal interface (b) of the firewall is initialized fairly
normally.  However, directly after it's initialized (in my rc.inet1.eth1)
there are a few other things that needed to be done...  Specifically, I
needed to make a static route for that host address pointing to the
internal interface, since the external interface of the firewall thinks
it's on a /24, and that address would be looked for on interface a instead
of b if we didn't...

/sbin/route add x.x.x.230 eth1

With this static route in place, the kernel will properly route packets to
that _one_ host inside, instead of thinking it should be on eth0.
Next we need to enable packet forwarding on that interface or all is for

echo "Enabling packet forwarding on eth1"
echo 1 >/proc/sys/net/ipv4/conf/eth1/forwarding

The external interface (a) on FWb is also configured normally (well, as
normally as one gets with a bridged segment) with it's IP address of
x.x.x.229, with the network and broadcast addresses being x.x.x.0 and
x.x.x.255 respectively.  It also has it's default route pointing to the
x.x.x.1 address on this interface.  There are THREE things that need to be
done on this interface to make it all work...

First off, we need to enable packet forwarding.  Normally you might think
this would come second or last, BUT changes to this particular
configuration item are required to *reset* the other proc values by yet
another obscure RFC.  Enabling forwarding after enabling proxy_arp may
*disable* proxy_arp as a side-effect and you need that.

echo "Enabling packet forwarding on eth0"
echo 1 >/proc/sys/net/ipv4/conf/eth0/forwarding

This enables proxy_arp.  Proxy_arp lets a host interface respond to ARP
queries on behalf of a host that resides on another of it's interfaces,
and since you *want* FWb to answer when an arp query to find x.x.x.230
happens on the "plain 'ol ethernet" segment, you'll need to turn this
feature on...

echo "Enabling proxy_arp on eth0"
echo 1 >/proc/sys/net/ipv4/conf/eth0/proxy_arp

Okay.  Now that you have given the kernel the go-ahead to be answering ARP
queries for more than just it's own interface on that IP address, we need
to make it clear to the kernel WHICH IP addresses it's supposed to be
answering for... hence, we are now doing something with the arp command
most people are blessed with never having to do unless they work with
firewalls all the time. 

# Publish the arp entry for the proxied host
/sbin/arp -i eth0 -s x.x.x.230 00:60:B0:EB:D6:12 pub

Okay.  THAT command tells the kernel that it is to answer ARP queries on
behalf of x.x.x.230 on it's external interface (a) and basically say
"Yeah, that's me alright".  The 00:60:B0:EB:D6:12 part is the only
non-obvious part of the lot... That is the MAC address of the _eth0_
interface to which it's being applied, which to some people would seem
quite redundant, but you *must* specify this on every *nix I've worked
with.  (I suppose on machines with a far more flexible NIC card you could
make up an ARP address, but the Linux kernel doesn't go for that kind of
foolishness, and neither should)  The common mistake is to try to publish
the MAC address of the IP address we're trying to route for, which is very
very wrong.  The "pub" simply means "publish this", i.e., "stick this into
your ARP table as us".  To do this with more than one host, you just need
to invoke it again with each additional IP.  Proxying arp for actual
*blocks* of IP addresses is not done this same way, and relies on some
other features we won't go into at the moment.  ;)

So... now with all this in place... Here's how it functions...  A packet
comes in from the internet from, looking for x.x.x.230 on the f
side...  The device at f doesn't know which ethernet host it should
deliver the packet to, it just knows that it's supposed to be on this
segment due to the bridging, so it makes an ARP query for x.x.x.230 on the
"plain ol' ethernet" segment.  FWb seeing this, and knowing that it's
supposed to respond for both x.x.x.229 (normal behaviour) as well as
x.x.x.230 on interface a (eth0) responds and says "Yup.  That's me" so f
hands the packet to it (a, eth0 on FWb) and considers the matter closed.

FWb's kernel gets the packet, and notes that it is *not* the ultimate
destination, so it would need to route the packet to go futher.  It's been
given permission to forward packets that reach this interface, so it looks
into it's routing table and sees that x.x.x.230 resides on eth1 (b)... It
then delivers the packet to the host responding to ARP for x.x.x.230
(Yes, FWb will have to make an arp query on that segment, unless it's
already done this and has the result cached) on eth1 (b) which is the d
interface on WWW.

To turn things around, a packet going to *from* WWW (d, x.x.x.230)
gets handed to the host that's acting as it's default gateway, which to it
is (i.e, interface b, eth1 on FWb).  FWb has been told it can
forward packets that hit this interface, and it concludes that
is not in any of it's local networks, so it hands the packet off to *it's*
default gateway of x.x.x.1, and with any luck it will reach
in a few moments.

To those who might be irritated with the length of this description, ARP
tomfoolery is something many people tend to take for granted, so when they
*do* have to use it, it can really cause some teeth grinding if they don't
have a _crystal_ clear idea of the difference between MAC (second from the
bottom layer) and IP (third from the bottom layer) addresses.  Hopefully
this will sort things out in short order for Mr. Copeland, as well as give
him some ideas of how to troubleshoot problems with ARP tomfoolery in the
future.  (People who deal with routers for a living are often and
repeatedly made painfully aware of the difference, so if you're one of
those people this whole thing might seem exceptionally redundant *g*)

Oh, and don't worry about having to do anything this complex with
ipchains/iptables if you intend to add in-kernel firewalling to your
firewall/routing host.  This won't affect how you build your firewall
rules much at all, although with this configuration reverse path filtering
(rp_filter) _may not behave in the manner you expect it to_.


--+ Dagmar

More information about the svlug mailing list