rdomains and multiple interfaces

[ssh2ksh]

Hi everyone,

I am happy to make my first post, these forums are full of knowledge and I hope to contribute. I have been using OpenBSD since 4.9 and have enjoyed using it on a variety of machines. I have a new configuration I'd like to try out but need a little help finding the best approach.

I am using a PCE APU as my home router and recently I've been learning about rdomains with the goal of hosting two internal networks -- one routed through a VPN service and one routed directly through the service provider. The router is running 5.6-stable amd64.

Currently, the internal interfaces are bridged together and vether0 is provided as a gateway. All internal traffic is routed and NATed through tun0 by default. Here's a quick diagram:

  re0          tun0
   |            |
<---- bridge0 ----->
vether0 re1 re2 ral0
   192.168.2.x/24

(egress is tun0 for the internal network)

Exceptions are made by adding static routes in rc.local then performing NAT on those destinations to re0.

My goal is to add another internal network where the default route will always be through re0. I could then NAT the traffic separately and bypass the tunnel based upon source network. Diagram attempt:

      re0            tun0
       \              /
         re1 re2 ral0
       /              \
192.168.2.x/24   10.0.2.x/24
    to tun0         to re0

(egress based on source network)

I've been studying the these two articles to help understand rdomains:

1) packetmischief.ca/2011/09/20/virtualizing-the-openbsd-routing-table
2) spacebugs.nl/index.php?option=com_content&task=view&id=259&Item id=33

The first article utilizes two discrete physical interfaces, but in my configuration the traffic could arrive from re1, re2, or ral0. I think this would necessitate additional virtual interfaces to function as gateways. The second article (watch out for the URL space) uses VLANs but I'm not sure yet if they are helpful for this task.

Ultimately I know that rdomains will provide a simple and secure way to separate the traffic, but I'm unsure how use rdomains with my current interface configuration. Do I need vether, vlans, or even new rdomains at all?

I feel I still have a lot of reading to do but am hoping a push in the right direction will get me started on an implementation.

Thanks for reading, and hello!

ssh2ksh

[jggimi]

Hello, and welcome.

1. You're using bridge(4) because you lack Ethernet switches for your LAN?
2. You're using vether(4) to prevent deletion of the bridged IP address upon physical disconnection of an Ethernet cable?

If you have two Ethernet switches on the LAN, or a single 802.1Q (VLAN) capable switch, you could then operate two Ethernets, each with its own IP subnet.

If you want to retain your current single-Ethernet infrastructure, you can still run two separate IP subnets over the same Ethernet. A DHCP server might only be able to supply addresses to one of the two subnets, requiring static IP assignments for the other. And, there are security implications to a single Ethernet sharing multiple IP subnets to be considered, since the Ethernet is shared by all devices.

[jggimi]

One addition: The ral(4) driver does not support VLAN tagging, but the re(4) driver does. So here are the various options I believe you can consider:

• Bridged, single Ethernet segment, as you have today, with the addition of multiple IP subnets.
• Separate Ethernet segments, with the deployment of Ethernet switches. Your ral0 NIC can either be bridged with one of the Ethernet segments, or, can be on its own, third subnet.
• An 802.1Q switch configured with multiple VLANs. As above, ral0 could remain its own subnet. It may also be bridged with an applicable vlan(4) pseudo device.

[ssh2ksh]

Hi jggimi and thank you,

You're absolutely right about 1) and 2) -- I'm using the bridge(4)/vether(4) combination essentially to mimic an off the shelf router where every interface is on the same Ethernet segment. I understand it might be a naïve configuration and I'm open to changing it, I just found it easy to understand and write my pf rules around. I've seen it recommended here to just deal with each interface separately and ditch the bridge(4).

One last complication is that due to the router's awkward location, everything currently connects to it wirelessly through ral(4) and the internal re(4) interfaces are mostly unused. This will change eventually but for now using re(4) requires an Ethernet cable across the room. It's an option but I'd like to avoid it.

Unfortunately I don't have any other switches currently available (sorry, this should have ruled out VLANs from the start) but with the help of your suggestions:

1. Do you think it would be best to remove the bridge(4)/vether(4) combination and handle traffic between the interfaces in pf?
   
2. With this limited wireless configuration, am I still able to create the additional IP subnets even if all connections originate on one interface, ral(4)? Would this be a job for vether(4)?

[jggimi]

Originally Posted by ssh2ksh

...I'm using the bridge(4)/vether(4) combination essentially to mimic an off the shelf router where every interface is on the same Ethernet segment.

These are routers that contain an embedded switch.

I understand it might be a naïve configuration and I'm open to changing it...

Its limiting, as it limits your Ethernet connections to one-per-NIC. Convenient? Sometimes. In a home? Certainly.

The price of 802.1Q switches has been dropping at the low end, and, if I were in the market today I'd be buying them for personal use, rather than using separate unmanaged switches as I do today. There's increased flexibility, as I can provision Ethernet segments within a single switch, and don't have to re-cable to move a device from one segment to another, and a reduction in NIC use on platforms like OpenBSD that support 802.1Q tags.

I've seen it recommended here to just deal with each interface separately and ditch the bridge(4).

Probably from me.

One last complication is that due to the router's awkward location, everything currently connects to it wirelessly....

Understood. I have a switch located among the media players and set top boxes in my living room, so that only one Ethernet cable needs to be run. WiFi won't move HD video due to interference here.

Do you think it would be best to remove the bridge(4)/vether(4) combination and handle traffic between the interfaces in pf?

This is a decision best left to you.

Note that if you don't bridge, each NIC will need to be on a separate subnet. But if you write rules based on networks rather than interfaces (e.g: use from and to rather than on), your rules shouldn't be too complicated. Example: you might use 10.0.1/24, 10.0.2/24, and 10.0.3/24 for three subnets without bridging. A rule that refers to 10.0/16 would encompass all three subnets.

With this limited wireless configuration, am I still able to create the additional IP subnets even if all connections originate on one interface, ral(4)? Would this be a job for vether(4)?

You can have as many subnets as you want within a single Ethernet segment. You may have difficulty crafting a workable DHCP server configuration, however. Here's an example of a DHCP configuration with multiple subnets on the same segment. While a bridge(4) wasn't involved, something similar might be used.

[ssh2ksh]

I really appreciate the assistance, jggimi, thanks! I've been working on this and wanted to make sure I had done more research before asking any more questions.

I spent some time with each interface on its own subnet but have moved back to bridge(4)/vether(4) for now. I also got much further along with rdomains but I'm thinking maybe I can do this without them.

I tried to create a simplied goal: pass some traffic out re0, some out tun0, and let the decision be based upon source address or source interface.

I have already accomplished this but in reverse. For example, I listen to a streaming music service and there's no need to tunnel that traffic, so I added static routes to rc.local(8) and rules to pf.conf(5) to accomplish this:

# /etc/rc.local
echo -n 'rc.local:'

echo -n ' static-routes'
/sbin/route add 70.42.73.18 192.168.0.1 > /dev/null 2>&1
/sbin/route add 70.42.73.32 192.168.0.1 > /dev/null 2>&1

echo '.'

# /etc/pf.conf
streams = "{ 70.42.73.18 70.42.73.32 }"
lan_net = "10.0.2.0/24"

set skip on lo

match out log on re0  from $lan_net to $streams nat-to re0
match out log on tun0 from $lan_net to any      nat-to (tun0)

block log

pass in  log on vether0
pass out log on re0 proto tcp from re0 to $streams port www
pass out log on tun0 from (tun0) to any

I'd like to do the same thing just in the other direction. I added a new vether(4) interface to the bridge(4) to act as a DMZ* with a 10.0.3.1/24 address and updated pf.conf(5). I also moved my LAN from 192.168.2/24 to 10.0.2/24, less typing!

* I think I am using the term DMZ incorrectly/loosely here

# /etc/pf.conf (updated)
streams = "{ 70.42.73.18 70.42.73.32 }"
lan_net = "10.0.2.0/24"
dmz_net = "10.0.3.0/24"

set skip on lo

match out log on re0  from $lan_net to $streams nat-to re0
match out log on re0  from $dmz_net to any      nat-to re0
match out log on tun0 from $lan_net to any      nat-to (tun0)

block log
block out log quick on tun0 from $dmz_net to any

pass in  log on { vether0 vether1 }
pass out log on re0  from  re0   to any
pass out log on tun0 from (tun0) to any

A laptop has a static address in the DMZ range and this is working fine. I can ping vether0/1, re0, the 10.0.2/24 LAN network, and my ISP gateway at 192.168.0.1. However, if I ping an internet address, it attempts to route through tun0 and is blocked (correctly).

# tcpdump -nei pflog0
rule 7 pass in on vether1: 10.0.3.2 > 192.168.0.1: icmp: 8 0
rule 2 match out on re0: 192.168.0.8 > 192.168.0.1: icmp: 8 0
rule 8 pass out on re0: 192.168.0.8 > 192.168.0.1: icmp: 8 0

rule 7 pass in on vether1: 10.0.3.2 > 8.8.8.8: icmp: 8 0
rule 5 block out on tun0: 10.0.3.2 > 8.8.8.8: icmp: 8 0

I think this is due to the routing table. OpenVPN adds a 0/1 route to itself when the tunnel comes up, and the inbound packets from the DMZ network hit the 0/1 route before they use the default route, and so the "match out on re0" NAT rule is never evaluated because it's going out tun0 instead.

# route -n show (excerpt)
Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
0/1                10.0.88.88         UGS       63   328932     -     8 tun0
default            192.168.0.1        UGS        1   553500     -     8 re0
70.42.73.18        192.168.0.1        UGHS       0     1153     -     8 re0
70.42.73.32        192.168.0.1        UGHS       1   204395     -     8 re0

This is where I have been getting stuck. I tried adding this rule:

pass in log on vether1 from $dmz_net to !re0 route-to re0

The packets are now going out re0 but they are not coming back to 10.0.3.2.

# tcpdump -nei pflog0
rule 8 pass in on vether1: 10.0.3.2 > 8.8.8.8: icmp: 8 0
rule 2 match out on re0: 192.168.0.8 > 8.8.8.8: icmp: 8 0
rule 9 pass out on re0: 192.168.0.8 > 8.8.8.8: icmp: 8 0

man 5 pf.conf says:

When a route-to rule creates state, only packets that pass in the 
same direction as the filter rule specifies will be routed in this way. Packets 
passing in the opposite direction (replies) are not affected and are routed 
normally.

Perhaps I need another rule to match the return traffic? I looked at reply-to but I'm starting to stretch my pf(4) experience and I'm not sure this is the right way to tackle this. All I can think of right now is:

• Different pf.conf(5) rules to accomplish the source address based routing
• Some way to have vether1 exclusively use the 192.168.0.1 default route
• Try again with rdomains and have separate routing tables for each interface

Thanks again for your time, jggimi, and anyone else who has been reading.

[jggimi]

Originally Posted by ssh2ksh

...I added static routes to rc.local(8) and rules to pf.conf(5) to accomplish this...

As you mentioned in your first post.

...
/sbin/route add 70.42.73.18 192.168.0.1 > /dev/null 2>&1
/sbin/route add 70.42.73.32 192.168.0.1 > /dev/null 2>&1...

I noticed, through whois(1) that audioaddict.com (also branded as sky.fm, and radiotunes.com) has the /23. You might want to use the netblock rather than the individual addresses. This way, you won't have to adjust your rules if they switch servers -- only if they switch netblocks.

* I think I am using the term DMZ incorrectly/loosely here

I believe its a correct use. Your network requires complete trust, of course, because any machine can use any IP address it wants, and join your DMZ subnet. [1]

OpenVPN adds a 0/1 route to itself when the tunnel comes up, and the inbound packets from the DMZ network hit the 0/1 route before they use the default route, and so the "match out on re0" NAT rule is never evaluated because it's going out tun0 instead.

Now, here's where I get lost, because I haven't used OpenVPN in 20 years or more. I interpret a "0/1" route as 0.0.0.0/1, which are the IP addresses 0.0.0.0-127.255.255.255. That's half of the IP addresses. Any IP address above this range won't use the VPN. That's fine for the higher range RFC1918 netblocks 192.168/16 or 172.16/12, but coul be a problem for the 10/8 range you're using. Granted, it's less typing ... but if you used higher subnet addresses above the 0/1 range, you may be able to avoid the 0/1 tunnel route.

I've no personal experience with route-to and reply-to, and don't have any guidance for you with PF routing options. The only "fancy" routing I've ever done has been equal cost multipath routing, and that didn't involve PF directed routing. (I just followed the FAQ.)

---

[1] Recent discussion of trusted vs. untrusted LAN tiers on the OpenBSD misc@ mailing list: http://marc.info/?t=141711266800001&r=1&w=2