DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD Security

FreeBSD Security Securing FreeBSD.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 28th November 2015
jasonvp's Avatar
jasonvp jasonvp is offline
Real Name: Jason
Port Guard
 
Join Date: Nov 2015
Location: Northern VA
Posts: 15
Default ARP Issue: Bridging, Routing, and FreeBSD LAGGs

I've already posted this on the FreeBSD forums but figured I'd get some more eyes on it. This is more network-related than security related, but the answer may lie in ipfw. I'm just not sure yet.

The diagram to reference:


The "router" and "server" in this case are both running FreeBSD. The router is bridging interfaces em0 and re0 together, represented on the router as the black line (em0) and the red line (re0). Verizon uses a single /24 broadcast domain for their business class customers, and out of that, they allocate a set number of public IPs per customer. In my case, I have 13: .210-.222.

The router has an IP address on the bridge0 interface of .222. The server has a 2xGigE LACP-enabled lagg interface, with an IP of .210. The server also has a few jails on it, each with a public IP.

The basic problem is that the router receives ARP information from Verizon's upstream (a Juniper router) for all of the IPs on that server. So, for instance, the ARP entry on my router for .210 will be Verizon's MAC address, not my server's lagg0 interface. They're on the same (bridged) VLAN, but for some reason the router hears it from VZ and ignores it from the server.


From a machine on the private VLAN (note the green network in the diagram), I telnet to xx.yy.zz.215 port 80, because it's a jail running apache. The first one succeeds:
Code:
deadshot$ telnet xx.yy.zz.215 80
Trying xx.yy.zz.215...
Connected to somehostname.
Escape character is '^]'.
And if I look on the router, the arp entry is incorrect:
Code:
lateapex-gw# arp -n xx.yy.zz.215
? (xx.yy.zz.215) at 54:e0:32:be:cf:c1 on bridge0 expires in 1197 seconds [bridge]
That MAC is the Verizon router on the same broadcast domain. Now that my router has the incorrect ARP entry, further telnets to xx.yy.zz.215 port 80 fail:
Code:
deadshot$ telnet xx.yy.zz.215 80
Trying xx.yy.zz.215...
The proper MAC entry should be the lagg0 interface from the jail host:
Code:
joker$ ifconfig lagg0 | grep ether
    ether 0c:c4:7a:31:e3:d8
If I hard-set the ARP entry on the router for IP xx.yy.zz.215 -> 0c:c4:7a:31:e3:d8, it's good and stays that way:
Code:
lateapex-gw# arp -S xx.yy.zz.215 0c:c4:7a:31:e3:d8
xx.yy.zz.215 (xx.yy.zz.215) deleted
lateapex-gw# arp -n 1.2.3.4
? (xx.yy.zz.215) at 0c:c4:7a:31:e3:d8 on bridge0 permanent [bridge]
After that, I can repeatedly get to xx.yy.zz.215's port 80 from the private VLAN:
Code:
deadshot$ telnet xx.yy.zz.215 80
Trying xx.yy.zz.215...
Connected to somehostname.
Escape character is '^]'.
^]
telnet> Connection closed.
deadshot$ telnet xx.yy.zz.215 80
Trying xx.yy.zz.215...
Connected to somehostname.
Escape character is '^]'.
^]
telnet> Connection closed.
I'm kind of lost here. The work-around for now is a series of static ARP entries on the router pointing each of the public IP addresses on joker's lagg0 interface to the proper MAC. But that seems like a bad solution and I'm going to have to remember those are there in case I ever re-use one of those IPs somewhere else. It would be better to figure out how to block the ARP replies coming back from Verizon's router, but only for those IP addresses that belong to me. I can't just blindly block all ARPs from the VZ router; doing that will mean I'll never learn his MAC.

Any ideas?
Reply With Quote
  #2   (View Single Post)  
Old 28th November 2015
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,975
Default

Is the bridge a requirement? If ARP is the problem, it would go away if you were able to provision a solution without a bridge, and just forward IP packets where they are needed.

I understand you have multiple external IP addresses, so perhaps a bidirectional NAT (binat) configuration would enable you to provision em0 and re0 on separate Ethernets.
Reply With Quote
  #3   (View Single Post)  
Old 28th November 2015
jasonvp's Avatar
jasonvp jasonvp is offline
Real Name: Jason
Port Guard
 
Join Date: Nov 2015
Location: Northern VA
Posts: 15
Default

Quote:
Originally Posted by jggimi View Post
Is the bridge a requirement? If ARP is the problem, it would go away if you were able to provision a solution without a bridge, and just forward IP packets where they are needed.
Thanks for the suggestion. Bidirectional NAT'ing is another work-around, but I'd rather not use NAT where it's not needed (NAT sucks, the end).
Reply With Quote
  #4   (View Single Post)  
Old 1st December 2015
jasonvp's Avatar
jasonvp jasonvp is offline
Real Name: Jason
Port Guard
 
Join Date: Nov 2015
Location: Northern VA
Posts: 15
Default

Quote:
Originally Posted by jggimi View Post
I understand you have multiple external IP addresses, so perhaps a bidirectional NAT (binat) configuration would enable you to provision em0 and re0 on separate Ethernets.
Allow me to apologize for the seemingly dismissive last reply. I've never really been a fan of the obfuscatia that comes along with NAT. It does create some interesting troubleshooting problems; I see that sort of thing in production environments all the time.

I am, however, warming up to the idea of doing it, because it'll allow the router to ... well... route. Sorta.

I'm proposing a series of changes to my pf.conf that will look like this:
Code:
vz_int = "em0"
ext_int = "re1"
int_int = "re0"

host1_priv = "192.168.0.210"
host1_pub = "XX.YY.ZZ.210"
host2_priv = "192.168.0.211"
host2_pub= "XX.YY.ZZ.211"
# .
# .
# .
# until host_11
#
# NAT rules
pass on $vz_int from $host1_priv to any binat-to $host1_pub
pass on $vz_int from $host2_priv to any binat-to $host2_pub
# .
# .
# .
# until host_11
Here's where it gets a bit confusing for me, but it probably is pretty easy. I've been avoiding NAT for my existing RFC1918 LAN when the destination is my public servers. In other words: I want the public servers to see SRC==192.168.xx.yy, not the NAT IP. If a public server has to start a conversation with the private LAN, I also want that to be native. I only want the private LAN to be NAT'd when it leaves the building.

See this post for explanation. I'm going to have to change the NAT lines, and I think this will accomplish what I'm after?
Code:
no nat on $ext_int from $local_ipv4_lan to $external_ipv4_lan
nat on $vz_int from $local_ipv4_lan to any -> XX.YY.ZZ.221
My gut tells me the first line is superfluous because the second line only says NAT on outgoing on the Verizon interface. Thoughts? This should allow the now BiNAT'd IPs to get to the RFC1918 space with their respective public IPs, right (assuming it they pass further rules in PF of course)?
Reply With Quote
  #5   (View Single Post)  
Old 1st December 2015
jasonvp's Avatar
jasonvp jasonvp is offline
Real Name: Jason
Port Guard
 
Join Date: Nov 2015
Location: Northern VA
Posts: 15
Default

Quote:
Originally Posted by jasonvp View Post
In other words: I want the public servers to see SRC==192.168.xx.yy, not the NAT IP. If a public server has to start a conversation with the private LAN, I also want that to be native. I only want the private LAN to be NAT'd when it leaves the building.
Here I go replying to myself; this isn't going to work, is it? The bi-nat lines will only be applied for traffic going through the $vz_int. If I left the rules as written, the hosts in $int_int wouldn't even be able to talk to the public IP addresses, would they? Would the router know how to respond to packets source from $int_int and destined for the BiNAT'd IPs? Or would it just drop them since they never traverse the $vz_int? Would I need to change the interface in the binat-to rules to "any"?
  • Each public IP should be able to talk to other public IPs via the router.
  • Each private IP on $int_int should be able to talk to the public IPs via the router.
  • Each public IP should be able to talk to the private IPs on $int_int via the router.

This is getting interesting and a little daunting.
Reply With Quote
  #6   (View Single Post)  
Old 2nd December 2015
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,125
Default

What is the purpose of your setup? What are you trying to achieve?
Why did you choose this topology instead of a classic DMZ setup?
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #7   (View Single Post)  
Old 2nd December 2015
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,975
Default

Briefly, the only place I would suggest to use (bidirectional) NAT is on your external, public network. If the router is routing, rather than bridging, your external facing network is only attached to the router, and none of your servers would have an external address -- the mapping to Internet-facing addresses would be done by the router, and only the router.

---

J65nko reminds me that there are at least two "classic" DMZ topologies that might be considered, so I'll briefly describe them:
  • DMZ subnet sits between two firewalls, an "inner" and an "outer."
  • DMZ subnet resides in isolation via a single firewall.
While I personally prefer the dual firewall topology as I believe it offers a better defense in depth, either would provide a better defensive governance than you are currently deploying or considering among your solution-set.
Reply With Quote
  #8   (View Single Post)  
Old 2nd December 2015
jasonvp's Avatar
jasonvp jasonvp is offline
Real Name: Jason
Port Guard
 
Join Date: Nov 2015
Location: Northern VA
Posts: 15
Default

Quote:
Originally Posted by J65nko View Post
What is the purpose of your setup? What are you trying to achieve?
Why did you choose this topology instead of a classic DMZ setup?
Per my previous posts: the basic problem is the way my ISP allocates IPs (static block of XX.YY.ZZ.210 - .222) and the network those IPs reside on. Specifically: I have to use their router on their premises as the default route (XX.YY.ZZ.1). They don't allocate me an actual CIDR block and route it towards me.

So, I have 3 basic choices:
1. DMZ where each server is on the same broadcast domain as the .1 address, meaning the servers are wide open and it's up to their local packet filters (or whatever) to protect themselves. I'm not interested in DMZs, so that's not a consideration.

2. L3 Bridged Interface such as what I started this thread with: Bridge the ISP side and broadcast domain that my servers are on, and use PF (iptables in the case of Linux) on the router to protect the network. The router isn't routing in this case, it's just a pass-through. The servers still set their default routes to XX.YY.ZZ.1.

3. A binat setup where the server's have an RFC1918 address that gets 1-to-1 NAT'd to the public IPs that I own. In this case, every packet is routed (sort of... a NAT isn't really a router) through the router as opposed to bridged/passed-through.

I'm doing choice #2 right now, but it's presenting me with all sorts of ARP problems on the FreeBSD router. The router is learning the ARPs for my IPs from the ISP's router, not from my servers. So the work-around for the time being is a collection of static ARPs on the router.

What's important to note is that my router is also the termination point for my existing RFC1918 subnet (wireless network, NAS, etc). So packets from that LAN to the world need to be NAT'd. Packets from it to the public side of my network should not be NAT'd. And packets from the public side of my network to the private side should also not be NAT'd.

If I go with choice #3, my concern is what will happen with the existing private LAN when it tries to talk to one of my public IPs? Will the right thing happen regarding the binat?
Reply With Quote
  #9   (View Single Post)  
Old 2nd December 2015
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,975
Default

Quote:
...If I go with choice #3, my concern is what will happen with the existing private LAN when it tries to talk to one of my public IPs? Will the right thing happen regarding the binat?
No. If you use NAT on your Internet-exposed subnet, you will have to route traffic to local servers (whether in a DMZ or not) via their local, untranslated network addresses.

See the Redirection and Reflection section of the PF User's Guide. The chapter hasn't changed much since the fork, other than using different PF syntax. It does discuss DMZs, but not as a direct solution.

http://www.openbsd.org/faq/pf/rdr.html#reflect

Last edited by jggimi; 2nd December 2015 at 02:13 AM. Reason: clarity
Reply With Quote
Old 2nd December 2015
jasonvp's Avatar
jasonvp jasonvp is offline
Real Name: Jason
Port Guard
 
Join Date: Nov 2015
Location: Northern VA
Posts: 15
Default

Quote:
Originally Posted by jggimi View Post
If the router is routing, rather than bridging, your external facing network is only attached to the router, and none of your servers would have an external address -- the mapping to Internet-facing addresses would be done by the router, and only the router.
Yep, I understand that. I've been a network engineer and architect for over 2 decades. What I don't understand (yet) is how FreeBSD handles all of these things. To be honest, Linux does this part really well and I'm actually considering converting just the router back to it. But, before I throw in the towel:

In a binat scenario, the router's interfaces:
  • em0: XX.YY.ZZ.222/24 # Public side, Verizon IPs. Of those, I have 13.
  • re0: 172.16.0.0/24 # RFC1918 side of "public" servers. Binat sources.
  • re1: 192.168.100.0/24 # Private LAN for wireless, NFS server, etc.
So, packets coming in em0 destined for one of the public IPs will get binat'd out towards re0. Conversely, packets from the 172.16.0.0/24 LAN outbound on any of the interfaces also need to be binat'd via the same 1-to-1 mapping.

Packets coming in re1 and destined to the Internet should be NAT'd via the last NAT rule I listed. The many-to-1. Packets coming in re1 and destined to XX.YY.ZZ.210 - .220 should not be many-to-1 NAT'd, but should trigger the 1-to-1 binat.

This is where it gets confusing. Theoretically, if the router gets a packet destined for XX.YY.ZZ.210-.220 (the binat IPs) from any interface, it should know that it owns those 11 IPs, and do the binat appropriately. Correct?

For instance, let's say I have the NAT'ing rules applied as previously posted. And...

source: 192.168.100.10
destination: XX.YY.ZZ.210

The packet from 192.168.100.10 should hit interface re1, and then... what? Will the binat, which is sitting on interface em0, get triggered? Will the router know to send the packet, translated, to 172.16.0.210? Or will it just get dropped?
Reply With Quote
Old 2nd December 2015
jasonvp's Avatar
jasonvp jasonvp is offline
Real Name: Jason
Port Guard
 
Join Date: Nov 2015
Location: Northern VA
Posts: 15
Default

Quote:
Originally Posted by jggimi View Post
No. If you use NAT on your Internet-exposed subnet, you will have to route traffic to local servers (whether in a DMZ or not) via their local, untranslated network addresses.
OK, thanks. That's what I figured. Ignore my previous reply to you. We were co-writing, apparently.
Reply With Quote
Old 2nd December 2015
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,975
Default

Most commonly, we use a split-horizon DNS. As an example, your authoritative nameserver would provide resolution for Internet-facing clients, and your caching nameserver would provide local resolution to local clients, resolving to your RFC1918 addresses.
Reply With Quote
Old 2nd December 2015
jasonvp's Avatar
jasonvp jasonvp is offline
Real Name: Jason
Port Guard
 
Join Date: Nov 2015
Location: Northern VA
Posts: 15
Default

Quote:
Originally Posted by jggimi View Post
Most commonly, we use a split-horizon DNS. As an example, your authoritative nameserver would provide resolution for Internet-facing clients, and your caching nameserver would provide local resolution to local clients, resolving to your RFC1918 addresses.
Yep, I figured that was going to be the next suggestion, and I'm not game for that either. I'm going to drop this binat idea on the floor and stick with the bridging. If it continues giving me problems, I'm going to have to revert back to Linux for the router.

Thanks for your suggestions and guidance.
Reply With Quote
Old 2nd December 2015
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,975
Default

Good luck, no matter what decisions you eventually make, or find yourself driven to by an ISP that is unwilling or unable to deploy TCP/IP as designed.
Reply With Quote
Old 5th December 2015
jasonvp's Avatar
jasonvp jasonvp is offline
Real Name: Jason
Port Guard
 
Join Date: Nov 2015
Location: Northern VA
Posts: 15
Default

Quote:
Originally Posted by jggimi View Post
Good luck, no matter what decisions you eventually make, or find yourself driven to by an ISP that is unwilling or unable to deploy TCP/IP as designed.
I'm almost ashamed to admit that: proxy arping solves the issue and allows the router to act as a router. As a network architect/engineer, I despise the idea of proxy arping. But in a small, well-defined and contained network such as my premises, it's not (that much of) a problem.

The basic steps:
1. Address the Verizon-facing interface on the router out of my static IP allotment, with the default route to .1
2. Address the public server facing interface on the router out of a completely phony IP block (I used 10.0.0.1/8).
3. Enable proxy arping on the router. In /etc/rc.conf:
Code:
# Set Proxy ARP
arpproxy_all="YES"
And then from the CLI:
Code:
# service routing restart
4. Set each of the public servers' default route to the IP address on the router, not Verizon's .1.

Voila. It works. It's sad that I have to deploy a broken network technology to fix a broken network. Yay Verizon!
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
FreeBSD NGINX issue scrummie02 FreeBSD Ports and Packages 6 13th February 2015 09:33 PM
Redundant Bridging and STP igy01 OpenBSD General 5 23rd January 2013 11:40 PM
vsftpd port issue on FreeBSD 8.2 x64 sparky FreeBSD Ports and Packages 7 29th March 2012 11:50 AM
FreeBSD FreeBSD 8 is getting new routing architecture clone News 0 10th November 2009 06:38 PM
FreeBSD ping issue wooki FreeBSD General 1 2nd October 2009 04:10 PM


All times are GMT. The time now is 02:15 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick