DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD General

OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 28th June 2021
discostew discostew is offline
Port Guard
 
Join Date: May 2021
Posts: 14
Default OpenBSD virtio driver (vio / macvtap) on VM strangeness bridged to bonded host NIC

Hi Daemon Forums,

I'm really stumped with this one..

I recently upgraded a number of my virtual machines to OpenBSD 6.9 / 6.8 (running on an Ubuntu 20.04 server host machine), and noticed some networking behavior I can't really figure out. (Apologies in advance if this thread is a better fit for a Linux / QEMU forum - I'm not really sure where the the issue is in this chain.)

The 6.9 VM has a macvtap NIC connected to a linux host interface that is bonded (i.e. two 1 GBE interfaces bonded in Round Robin mode to a managed switch) using the Bridge source model. The interface on the OpenBSD side is using the virtio (vio) driver.

It is my understanding that this configuration (macvtap / bridge) should NOT support host to guest network communication (big warning in QEMU when you set this up). This is by design in the macvtap, is my understanding, and macvtap is not designed to pass traffic between host / guess - only to other non-host destinations on the network.

In this configuration macvtap does in fact appear to be passing (almost all) of the traffic between guest and host, regardless of which side originates the connection.

To make things even more confusing, it doesn't appear to be passing _ALL_ of the traffic in the connection (randomly dropping some packets?), and even worse, it appears it might be sending duplicate packets at times as well! For example, when sniffing traffic on either the host or guess side, I am routinely seeing duplicate ACKs.

Basically, any TCP sessions between guess and host that are short (e.g. DNS queries, etc.) appear to work like business as usual. Longer sessions, like requests to a web server running on the host, appear to work well for about 4-5 seconds before the TCP session quickly melts down. For these longer sessions, the first few packets go by without any problems, however after a few dropped packets and duplicate ACKs the TCP session quickly grinds to a halt and melts down.

Anybody have any idea why 1) macvtap is allowing traffic between guest and host in the first place, and 2) why I might be seeing duplicate / dropped / TCP session meltdown in this configuration?

I can't help but think this is related to the host's interface (bridged device) being a bonded interface.. I setup the bonded interface about 6 months ago, and I vividly remember that macvtap was (properly) not facilitating guest <-> host traffic BEFORE setting up the bond. However, I also recently upgraded both the virtual machines and the host OS, so there have been a lot of moving parts since then

Last edited by discostew; 28th June 2021 at 08:27 PM.
Reply With Quote
  #2   (View Single Post)  
Old 28th June 2021
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,416
Default

While you may have something mis-provisioned on your host, note that the vio(4) man page discusses two flags you can set. It's possible neither of them would apply to your situation:
Code:
     Setting the bit 0x2 in the flags disables the RingEventIndex feature.
     This can be tried as a workaround for possible bugs in host
     implementations of vio at the cost of slightly reduced performance.
Code:
     Setting the bit 0x100 in the flags forces the interface to be always in
     promiscuous mode.  This can be used as a workaround for a bug in QEMU
     before version 1.7.2 that prevents packets with a VLAN tag from being
     sent to the guest.
To my understanding, these kernel flags are your only provisioning options other than standard TCP/IP tools such as ifconfig(8). They are provisioned into a bootable kernel with config(8), which will disable kernel re-linking on each boot. For quick testing, see boot_config(8) for setting flags temporarily at boot.
Reply With Quote
  #3   (View Single Post)  
Old 28th June 2021
discostew discostew is offline
Port Guard
 
Join Date: May 2021
Posts: 14
Default

UPDATE: I fired up one of my old, pre-upgrade OpenBSD 6.1 VMs with an identical configuration on the same host (macvtap / bridge / virtio to bonded host interface), and this configuration appears to be happily facilitating guest <-> host traffic without issue! No dropped / duplicate packets, and no TCP session meltdown. In fact it sustains a nice 19 MB/s transfer rate through https to the host, which isn't to shabby!

Perhaps it has always been working well on the pre-6.9 upgrade hosts, which is why I didn't notice anything peculiar about this setup

At any rate, since I can reproduce the TCP meltdown on OpenBSD 6.8 and OpenBSD 6.9, and I cannot reproduce it on OpenBSD 6.1, something has changed since 6.1 (new bug?)

Interested to hear your thoughts!
Reply With Quote
  #4   (View Single Post)  
Old 28th June 2021
discostew discostew is offline
Port Guard
 
Join Date: May 2021
Posts: 14
Default

Awesome, thank you for the quick reply jggimi! Sorry I was typing my reply when you posted yours and didn't see it. Let me give that a try and see if that fixes the issue.
Reply With Quote
  #5   (View Single Post)  
Old 28th June 2021
discostew discostew is offline
Port Guard
 
Join Date: May 2021
Posts: 14
Default

jggimi - I just tried both the 0x2 and 0x100 vio flags on the OpenBSD 6.8 VM and there was no change: same duplicate packets / dropped packets / TCP session melt down.

The same https transfer that gives me a sustained 19 MB/s to my OpenBSD 6.1 VM achieves a jaw dropping 4 KB/s to 6 KB/s transfer rate to my OpenBSD 6.8 machine...

Very strange
Reply With Quote
  #6   (View Single Post)  
Old 28th June 2021
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,416
Default

If one of these flags doesn't resolve the issue, then you likely discovered a regression. Discovering the root cause has a process to locate the commit which caused it.

  1. Install 6.2 and/or newer releases until you find the first release with the regression.
  2. Using AnonCVS (see the sidebar at www.openbsd.org), build from source and test kernels bisecting by date/time until you can locate the commit which introduced the regression.
Yes, its iterative, and takes lots of time, but it makes a formal report to bugs@ really helpful.
Reply With Quote
  #7   (View Single Post)  
Old 28th June 2021
discostew discostew is offline
Port Guard
 
Join Date: May 2021
Posts: 14
Default

jggimi - I just tried changing from the vio driver to the rtl8139 driver for the QEMU model type on the guest, and the problem is reproducible with with the rtl8139 driver as well. Same performance.

Does this imply that vio might still be the issue, or perhaps it is somewhere else (pf maybe?)
Reply With Quote
  #8   (View Single Post)  
Old 28th June 2021
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,416
Default

I don't have enough information to answer.
Reply With Quote
  #9   (View Single Post)  
Old 29th June 2021
discostew discostew is offline
Port Guard
 
Join Date: May 2021
Posts: 14
Default

I took a quick pcap of the guest (10.0.0.12) <-> host (10.0.0.7) traffic on the 6.8 guest from the macvtap interface (attached). Switched back to virtio. This capture is fairly late in the connection 'melt down'.

My biggest question is why is macvtap passing any traffic at all in the first place! Overall the connection eventually achieves a few KB/s even with all of these errors, so if you wait long enough you can kinda get something out of the effort.

Also, I just did a fresh / generic 6.5 VM install - reproducible on 6.5 as well.
Attached Images
File Type: png Screenshot from 2021-06-29 10-07-20.png (997.4 KB, 10 views)
Reply With Quote
Old 29th June 2021
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,416
Default

You can also capture from your various guests with tcpdump(8), which may provide some insights too. You asked:
Quote:
...why is macvtap passing any traffic at all in the first place!
I wouldn't know, but that question indicates to me that the root cause of the problem may be host-related rather than guest. How does the capture compare to one with the known-good 6.1 guest?
Reply With Quote
Old 2nd July 2021
discostew discostew is offline
Port Guard
 
Join Date: May 2021
Posts: 14
Default [SOLVED] OpenBSD virtio driver (vio / macvtap) on VM strangeness bridged to bond nic

Jggimi, I think I've found a solution to this issue and in the process answered both of the questions regarding macvtap:

Quote:
Anybody have any idea why 1) macvtap is allowing traffic between guest and host in the first place, and 2) why I might be seeing duplicate / dropped / TCP session meltdown in this configuration?
First, the warning message that QEMU gives when setting up a macvtap interface, "In most configurations, macvtap does not work for host to quest network communication.", is a little ambiguous / misleading.. Macvtap's behavior for guest <-> host communication changes somewhat based on what type of interface it is bridged to. Here are the details:

When macvtap is connected to a physical ethernet, it sounds like the traffic it receives is injected too low in the stack to facilitate guest <-> host communication. Basically once macvtap hands a packet to a bridged physical interface, that packet is destined to egress from that physical interface no matter what, and can't go retrograde up the TCP stack back to the host.

When macvtap is connected to a virtual ethernet, such as a 'bridge' interface or macvlan interface (and i'm guessing ipvlan as well), the interface can make smarter decisions on where to forward the packet (e.g. guest to host, guest to guest, or guest to lan).

When macvtap is connected directly to a BONDED interface, which it appears is somewhere inbetween the previous two scenarios, things get kinda funky.. I haven't found this discussed / documented anywhere to date, but it appears that connecting macvtap directly to a bonded interface yields undefined / unpredictable behavior. In my case, my bond uses the low-level balance-rr bonding mode, and without knowing the low level details / inner-workings of the bond driver I'm guessing that packets introduced at the level macvtap injects them at were getting shuffled around incorrectly (i.e. leading to dropped / duplicate packets in the session - theory seems reasonable to me).

So the solution was to add another logical layer in the network configuration on the guest side, make a macvlan interface connected to the bonded interface, and connect macvtap on the guess to macvlan on host, instead of directly to the bond, i.e:

Guest side | macvtap | Host side
VM Guest -> vio0 interface -> | macvtap | -> macvlan0 -> bond0 [enp1s0f3, enp1s0f4] -> LAN

With this configuration all traffic is routed correctly from guest <-> host, guest <-> guest, and guest <-> LAN.

And this configuration is fast! I'm now seeing ~160 MB/s scp transfer from host to guest (single core 6.9 VM, i'm guessing this might be the 'bottleneck' here), while maintaining ~140 MB/s NFSv4.1 transfers from host to LAN over the bonded link (2x 1 GBE nics, all interfaces configured 9000 MTU jumbo frames).

I also tried using 'bridge' instead of macvlan. This appears to work for guest <-> host, and guest <-> LAN traffic, however it did not permit guest <-> guest traffic (for reasons I have not yet contemplated, but I'm sure are perfectly inline with intended behavior for the bridge interface). Bridge also has the added disadvantage of not passing ARP traffic from guest <-> LAN, which made establishing connections reliability (i.e. without setting up some sort of proxy-arp or hard-coded ARP tables) a nightmare.

Quote:
How does the capture compare to one with the known-good 6.1 guest?
This is embarrassing, but the way name resolution was setup on the 6.1 guest the connection was silently being resolved to the router, which was in turn routing back to the host.. Therefore while the connection's ingress / egress traffic was being routed from / to the macvtap->bond bridge, having the traffic go through the router was bypassing the direct guest <-> host route, and thus, passed traffic (albeit at slower transfer rates).

Hope this helps.

Thank you so much for your help Jggimi!

Last edited by discostew; 2nd July 2021 at 10:02 PM.
Reply With Quote
Old 3rd July 2021
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,416
Default

I'm glad you have found a solution!
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
MacVTap VEPA with OpenBSD router/firewall, need bridge to reflect on same segment rbigm101 OpenBSD Security 17 20th September 2016 04:03 PM
OpenBSD 5.6 runs into a panic on boot [virtio] jkl OpenBSD General 18 4th February 2015 02:53 PM
Virtio Support :: OpenBSD liquidshane OpenBSD General 3 23rd June 2012 10:13 PM
VMT driver (VMware Tools driver) on OpenBSD 4.8 xinform3n OpenBSD General 7 9th December 2010 10:48 PM
syslog strangeness on freebsd 8.0 and 8.1-RC vikashb FreeBSD General 0 6th July 2010 04:31 AM


All times are GMT. The time now is 05:31 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick