MacVTap VEPA with OpenBSD router/firewall, need bridge to reflect on same segment

[rbigm101]

Hello community! Long time thread reader, first time poster. Here is what I would like some help with if someone is available to do so.

I'm using OpenBSD 5.0 i386

I had followed this guide posted below, and I had successfully set up a couple VEPA connections using MacVTap on my Debian KVM server running libvirt. If you are unfamiliar with this type of VM connection, it does not allow the VM's or the host to talk to each other, all frames get sent to the LAN directly.

The advantage of this is that I can use my existing firewall rules in pf, and simply add the VM to the appropriate table and enjoy the benefits of OpenBSD security. All while still having my VM's isolated from the host and each other exactly the way I want them to be, managed from one location.

The issue is (according to the bridge(4) man page) that "If the destination is on the same segment as the origin segment, the bridge will drop the packet because the receiver has already had a chance to see the frame."

Now this is my issue here, I'm having a difficult time getting a work around together (that I'm assuming involves removing the interface in question from the bridge), mostly because I'm not sure what components of OpenBSD (like a !route line in hostname.if) are operating in the layer that I'm trying to deal with (layer2?). I have tried to disable stp on the port in question (using a line in the hostname.if), I have tried bridging vether devices together so I can merry-go-round the frames back to the interface, but the bridges won't seem to allow a vether device on more than one bridge. I have also tried integrating trunks and vlans, to no success.

The biggest thing that has brought me to the forum with this issue is that my experiments with the various psudo-devices are not doing what I am predicting they will in any way. I am guessing this is a result of the psudo-devices being designed to work concisely, and not in the way that I'm using them.

Linux has a 'Hairpin mode' and some physical switches offer ‘Reflective Relay’ mode, I had assumed that I could implement this in OpenBSD.

If someone could describe to me a theory that I could try to implement, I would me most appreciative.

https://seravo.fi/2012/virtualized-b...g-with-macvtap

Thank you,

Mike

[ocicat]

Originally Posted by rbigm101

I'm using OpenBSD 5.0 i386

Are you using OpenBSD 5.0 or 6.0?

[jggimi]

Hello and welcome!

Is there a way to meet your topology requirements without using bridges? An external managed switch, for example? Or perhaps vxlan(4) will do what you need. The vxlan driver was designed to tunnel complete virtual networks over cloud services, and was added with OpenBSD release 5.5.

[rbigm101]

Originally Posted by ocicat

Are you using OpenBSD 5.0 or 6.0?

I'm using OpenBSD 5.0

[rbigm101]

Originally Posted by jggimi

Hello and welcome!

Is there a way to meet your topology requirements without using bridges? An external managed switch, for example? Or perhaps vxlan(4) will do what you need. The vxlan driver was designed to tunnel complete virtual networks over cloud services, and was added with OpenBSD release 5.5.

I believe I need to use bridges, as the switch will allow the VM's to talk to each other unless I implement a second firewall (which I am trying to avoid), and I don't know of any other way to connect my interfaces besides a bridge, which are as follows:
I have a WAP on one interface, my Debian server with the VM's on another interface, my Desktop on a third interface, and the internet on a fourth interface. The WAP is running dd-wrt.

I will look into the vxlan possibility. Which means I will be upgrading to OpenBSD 6.0, and that may take me a few days. Thanks you for the information. I'll update the thread with my results.

[jggimi]

See the vxlan(4) man page, and then this presentation.

[jggimi]

I should add, that's a FreeBSD presentation but it gives a detailed explanation of the tunneling mechanism. I looked for an OpenBSD-specific presentation, as I recalled seeing one, but perhaps I am thinking of the video of reyk@'s presentation in 2014.

[ocicat]

Originally Posted by rbigm101

I'm using OpenBSD 5.0

OpenBSD 5.0 was released in November 2011, & support ended in November 2012.

[rbigm101]

Originally Posted by ocicat

OpenBSD 5.0 was released in November 2011, & support ended in November 2012.

I'll update the thread after I've updated to 6.0. It'll take me a few days.

[rbigm101]

Alright so I'm up and running with OpenBSD 6.0. Right now I have my interfaces bridged, with DCHP running off of a vether interface.

My thoughts in the meantime on getting my VEPA working have been as follows:

1. I can't think of a way to attach the vxlan to my interface (without a bridge) in a way that seems like it would work.

2. I could remove the interface to the KVM host from the bridge, and do some kind of "rdr-to and nat-to combination" as outlined here, http://www.openbsd.org/faq/pf/rdr.html#rdrnat , but this is going to involve getting this to work, while also restoring "bridge like" functionality.

So my question to the forum now is, is #2 my best option, and what is the general idea behind how this would be implemented? I'm finding a working theory very challenging to develop.

[LeFrettchen]

Originally Posted by ocicat

OpenBSD 5.0 was released in November 2011, & support ended in November 2012.

Who cares ?
I still use OpenBSD 5.1 on my old laptop (Pentium 3), and it rocks.

[jggimi]

Originally Posted by rbigm101

I can't think of a way to attach the vxlan to my interface (without a bridge) in a way that seems like it would work.

Keep in mind, vxlan(4) must be used by all participating VMs. These tunnels might be able to "bridge" using multicasting, as described in section 2.4 of Reyk's paper, which begins on page 91 of the AsiaBSDCon 2014 proceedings. https://2016.asiabsdcon.org/papers/abc2014-proc-all.pdf

Originally Posted by LeFrettchen

Who cares ?
I still use OpenBSD 5.1 on my old laptop (Pentium 3), and it rocks.

In this particular case, rocking doesn't matter. The vxlan(8) driver was not available until 5.5.

As for your 5.1-release laptop, there have been 71 critical security and reliability fixes published since it was released. Of these 71, only 1 is available for 5.1. Should you care? Well, that's entirely up to you.

[TronDD]

Originally Posted by LeFrettchen

Who cares ?

Maybe the people who volunteer their time to provide support.

Run whatever you want but time moves on. If you didn't keep up, you can't expect many people remember how the system worked 5 years ago, or to have a setup where they can help investigate your problem.

[rbigm101]

Originally Posted by jggimi

Keep in mind, vxlan(4) must be used by all participating VMs. These tunnels might be able to "bridge" using multicasting, as described in section 2.4 of Reyk's paper, which begins on page 91 of the AsiaBSDCon 2014 proceedings. https://2016.asiabsdcon.org/papers/abc2014-proc-all.pdf)

I hadn't considered that I might be able to connect the xvlan(4) directly to the VM (or any other type of implementation that did not include using bridge(4)), as the example implementation of xvlan(4) that I found to help me understand what xvlan(4) fundamentally was, seems to show that is needs to be a part of a bridge(4) to be useful. That refrence is here: http://www.echothrust.com/blogs/usin...ay-remote-lans

I'll look through that paper to see if I can discover an implementation of xvlan(4) that will relate to my situation. Thank you for your reply.

[jggimi]

Originally Posted by rbigm101

I'll look through that paper to see if I can discover an implementation of xvlan(4) that will relate to my situation.

His video (link above) may be helpful, and if you need to, you might consider reaching out to him directly.

[rbigm101]

Update:
I have been configuring my VM’s to use the vxlan. I have the vxlan that is attached to the VM’s bridged to my eth0 interface. I have the other vxlan on my OpenBSD router, on the same bridge as all the interfaces. The purpose of this experiment was to gain/demonstrate the functionality of having one of the VM’s talk to its host system. The VM talkes to to xvlan, the xvlan repeats on the OpenBSD bridge, the bridge sends the frame to the KVM host ethernet port.

Here’s what keeps happening:
1. I have my OpenBSD router running with the vxlan ready to go.

2. I turn on my KVM host and configure the vxlan with these commands.

ip link add vxlan0 type vxlan id 1 remote 172.168.1.1 local 172.168.1.11 dev eth0 dstport 4789
ip addr add 172.168.1.200/24 dev vxlan0

Then I start my VM’s

3. I run tests with ping, telnet, and tcpdump, while using X11 forwarding to view/use the running VM’s, so I can do ping, and telnet tests on those, while watching my tcpdump.

4. At some point in my testing, the OpenBSD kernel will crash. I have noticed that these kernel crashes will not reoccur if I restart the OpenBSD router after a kernel crash and continue testing, without having restarted my KVM host along with it. Meaning I just ssh into the KVM host again and continue my testing. If I restart the KVM host, shortly after I start testing, I'll do a ping or try to ssh and the router will crash. Photos of all of the information I could produce from the crashes are below.

I have not been able to get the xvlan to work at any point durring this (not even an ip assignment to the VM’s) , and I’m worried that the functionality I’m looking for is not available while the router is the last thing that has been booted. Although it could be just poor vxlan configuration and/or firewall issues.

Tomorrow I’m going to submit a bug report on the kernel crashes, and a bug report on the bridge(4) for not supporting a “hairpin” mode, which I believe is the most elegant solution to using VEPA with OpenBSD. I’m also going to look into different ways to connect the vxlan to the VM’s.

http://s12.postimg.org/x2vfmecf1/IMG...003886_HDR.jpg
http://s12.postimg.org/rsqgv3s65/IMG...547944_HDR.jpg
http://s12.postimg.org/ke152q6al/IMG...023029_HDR.jpg
http://s12.postimg.org/riiybrdjx/IMG..._185802405.jpg

[jggimi]

Originally Posted by rbigm101

...a bug report on the bridge(4) for not supporting a “hairpin” mode...

A lack of a feature is not considered a bug, even if you determine the feature is required for your particular implementation.

[rbigm101]

Originally Posted by jggimi

A lack of a feature is not considered a bug, even if you determine the feature is required for your particular implementation.

Then I'll title it feature request, I've seen feature requests in bug reports before.

Also, I had left the vxlan configured on my KVM host overnight, after the last kernel crash. This morning I saw that it had crashed again (pic below), so the "KVM post OpenBSD crash reboot theory" is out.

I need to move on with my VM based project, so I'm going to try and make an OpenBSD firewall/switch VM that is bridged to my physical router with vxlan. The VM's will be on an isolated network. I hope this will be possible without havingto put the VM's on a different subnet. I may try to add the functionality that I need to bridge(4) myself in the near future.

http://s17.postimg.org/homs04inj/IMG..._112525508.jpg