ARP Issue: Bridging, Routing, and FreeBSD LAGGs

[jasonvp]

I've already posted this on the FreeBSD forums but figured I'd get some more eyes on it. This is more network-related than security related, but the answer may lie in ipfw. I'm just not sure yet.

The diagram to reference:


The "router" and "server" in this case are both running FreeBSD. The router is bridging interfaces em0 and re0 together, represented on the router as the black line (em0) and the red line (re0). Verizon uses a single /24 broadcast domain for their business class customers, and out of that, they allocate a set number of public IPs per customer. In my case, I have 13: .210-.222.

The router has an IP address on the bridge0 interface of .222. The server has a 2xGigE LACP-enabled lagg interface, with an IP of .210. The server also has a few jails on it, each with a public IP.

The basic problem is that the router receives ARP information from Verizon's upstream (a Juniper router) for all of the IPs on that server. So, for instance, the ARP entry on my router for .210 will be Verizon's MAC address, not my server's lagg0 interface. They're on the same (bridged) VLAN, but for some reason the router hears it from VZ and ignores it from the server.


From a machine on the private VLAN (note the green network in the diagram), I telnet to xx.yy.zz.215 port 80, because it's a jail running apache. The first one succeeds:

deadshot$ telnet xx.yy.zz.215 80
Trying xx.yy.zz.215...
Connected to somehostname.
Escape character is '^]'.

And if I look on the router, the arp entry is incorrect:

lateapex-gw# arp -n xx.yy.zz.215
? (xx.yy.zz.215) at 54:e0:32:be:cf:c1 on bridge0 expires in 1197 seconds [bridge]

That MAC is the Verizon router on the same broadcast domain. Now that my router has the incorrect ARP entry, further telnets to xx.yy.zz.215 port 80 fail:

deadshot$ telnet xx.yy.zz.215 80
Trying xx.yy.zz.215...

The proper MAC entry should be the lagg0 interface from the jail host:

joker$ ifconfig lagg0 | grep ether
    ether 0c:c4:7a:31:e3:d8

If I hard-set the ARP entry on the router for IP xx.yy.zz.215 -> 0c:c4:7a:31:e3:d8, it's good and stays that way:

lateapex-gw# arp -S xx.yy.zz.215 0c:c4:7a:31:e3:d8
xx.yy.zz.215 (xx.yy.zz.215) deleted
lateapex-gw# arp -n 1.2.3.4
? (xx.yy.zz.215) at 0c:c4:7a:31:e3:d8 on bridge0 permanent [bridge]

After that, I can repeatedly get to xx.yy.zz.215's port 80 from the private VLAN:

deadshot$ telnet xx.yy.zz.215 80
Trying xx.yy.zz.215...
Connected to somehostname.
Escape character is '^]'.
^]
telnet> Connection closed.
deadshot$ telnet xx.yy.zz.215 80
Trying xx.yy.zz.215...
Connected to somehostname.
Escape character is '^]'.
^]
telnet> Connection closed.

I'm kind of lost here. The work-around for now is a series of static ARP entries on the router pointing each of the public IP addresses on joker's lagg0 interface to the proper MAC. But that seems like a bad solution and I'm going to have to remember those are there in case I ever re-use one of those IPs somewhere else. It would be better to figure out how to block the ARP replies coming back from Verizon's router, but only for those IP addresses that belong to me. I can't just blindly block all ARPs from the VZ router; doing that will mean I'll never learn his MAC.

Any ideas?