DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD Security

OpenBSD Security Functionally paranoid!

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 7th February 2022
Reeshar Reeshar is offline
Real Name: Richard L
Port Guard
 
Join Date: Feb 2022
Location: London, UK
Posts: 14
Default Issue with WireGuard and routing domains on OBSD 7.0

TL;DR: WireGuard acting as a client under fully-patched OBSD 7.0 on an RPi4 works perfectly when using a single routing domain, but then fails when the external interface is put into a separate routing domain - rdomain 1. tshark shows handshake initiation packets leaving the external interface but no response being received from the server. Has anyone successfully implemented a WireGuard client with the latest release of OBSD and split routing domains?
__________________


I've been running WireGuard for some time now on a server connected to the Internet, initially using wireguard-go and more recently using hostname.wg0 under OBSD 7.0. The sole purpose of the server is to turn traffic around and send it back out to the Internet when I'm abroad and make the traffic appear to be coming from the UK. Standard stuff to be able to access UK streaming services.

It all works perfectly with a variety of clients: WireGuard on Android, Windows, Linux and various appliances. It's also a lot faster than using OpenVPN which I also have running on the server as a backup, although I'll probably decommission that soon.

I've recently been playing with a WireGuard client gateway using OBSD 7.0 on a Raspberry Pi 4. In its basic form using a single default routing domain, WireGuard works fine. I use different priority routes for direct traffic to the VPN router and all other traffic intended to pass through the wg0 tunnel. tshark shows WireGuard handshakes and encrypted traffic passing through the external interface bse0 exactly as expected.

Next step...

To introduce greater separation between the direct and tunneled routes to the server I have then made two modifications: I've added an rdomain 1 line to the start of the hostname.bse0 external interface file and wgrtable 1 to the wgpeer line in the hostname.wg0 file. (In addition I've modified sshd_config to listen to addresses from rdomains 0 and 1, although that's not pertinent to WireGuard behaviour. I've also not bothered with DNS mods at this stage as I'm testing using pings with IP addresses and dig with a specified external DNS.)

On reboot, I've made wg0 the default interface for rdomain 0 traffic. With that, WireGuard has stopped working on the RPi4. tshark shows outbound handshake requests going out to the server but no response from the server. If I switch back to single-domain operation everything works OK again. This seems to suggest that the handshake request has become corrupted such that the server doesn't recognise it, but I would love someone else to find some other explanation!

Has anyone any experience of WireGuard on OBSD 7.0 running with split routing domains? Or is anyone willing to have a go themselves at replicating the situation? It doesn't need to be an RPi: any host system would do!

___________________


Here's a copy of the tshark output monitoring the external interface for traffic to/from the WireGuard server.

With a single default routing domain:

Code:
-bash-5.1$ doas tshark -f 'host x.x.x.x' -i bse0
Capturing on 'bse0'
    1   0.000000 192.168.1.125 ? x.x.x.x WireGuard 190 Handshake Initiation, sender=0xFBFAD38C
    2   0.004860 x.x.x.x ? 192.168.1.125 WireGuard 134 Handshake Response, sender=0x47722CB5, receiver=0xFBFAD38C
    3   0.006097 192.168.1.125 ? x.x.x.x WireGuard 170 Transport Data, receiver=0x47722CB5, counter=0, datalen=96
    4   0.006649 x.x.x.x ? 192.168.1.125 WireGuard 74 Keepalive, receiver=0xFBFAD38C, counter=0
    5   0.023525 x.x.x.x ? 192.168.1.125 WireGuard 170 Transport Data, receiver=0xFBFAD38C, counter=1, datalen=96
    6   1.005879 192.168.1.125 ? x.x.x.x WireGuard 170 Transport Data, receiver=0x47722CB5, counter=1, datalen=96
    7   1.024303 x.x.x.x ? 192.168.1.125 WireGuard 170 Transport Data, receiver=0xFBFAD38C, counter=2, datalen=96
    8   2.005833 192.168.1.125 ? x.x.x.x WireGuard 170 Transport Data, receiver=0x47722CB5, counter=2, datalen=96
    9   2.020512 x.x.x.x ? 192.168.1.125 WireGuard 170 Transport Data, receiver=0xFBFAD38C, counter=3, datalen=96
   10   3.005815 192.168.1.125 ? x.x.x.x WireGuard 170 Transport Data, receiver=0x47722CB5, counter=3, datalen=96
   11   3.022067 x.x.x.x ? 192.168.1.125 WireGuard 170 Transport Data, receiver=0xFBFAD38C, counter=4, datalen=96
   12   4.005796 192.168.1.125 ? x.x.x.x WireGuard 170 Transport Data, receiver=0x47722CB5, counter=4, datalen=96
   13   4.020740 x.x.x.x ? 192.168.1.125 WireGuard 170 Transport Data, receiver=0xFBFAD38C, counter=5, datalen=96
   14   5.005779 192.168.1.125 ? x.x.x.x WireGuard 170 Transport Data, receiver=0x47722CB5, counter=5, datalen=96
   15   5.020599 x.x.x.x ? 192.168.1.125 WireGuard 170 Transport Data, receiver=0xFBFAD38C, counter=6, datalen=96
   16   6.005758 192.168.1.125 ? x.x.x.x WireGuard 170 Transport Data, receiver=0x47722CB5, counter=6, datalen=96
   17   6.020418 x.x.x.x ? 192.168.1.125 WireGuard 170 Transport Data, receiver=0xFBFAD38C, counter=7, datalen=96
   18   7.005756 192.168.1.125 ? x.x.x.x WireGuard 170 Transport Data, receiver=0x47722CB5, counter=7, datalen=96
   19   7.020781 x.x.x.x ? 192.168.1.125 WireGuard 170 Transport Data, receiver=0xFBFAD38C, counter=8, datalen=96
   20   8.005738 192.168.1.125 ? x.x.x.x WireGuard 170 Transport Data, receiver=0x47722CB5, counter=8, datalen=96
   21   8.020502 x.x.x.x ? 192.168.1.125 WireGuard 170 Transport Data, receiver=0xFBFAD38C, counter=9, datalen=96
   22   9.005714 192.168.1.125 ? x.x.x.x WireGuard 170 Transport Data, receiver=0x47722CB5, counter=9, datalen=96
   23   9.027533 x.x.x.x ? 192.168.1.125 WireGuard 170 Transport Data, receiver=0xFBFAD38C, counter=10, datalen=96
   24  19.025513 192.168.1.125 ? x.x.x.x WireGuard 74 Keepalive, receiver=0x47722CB5, counter=10
^C24 packets captured
With the external interface in rdomain 1 and access to the wg0 tunnel from rdomain 0:

Code:
-bash-5.1$ doas tshark -f 'host x.x.x.x' -i bse0
Capturing on 'bse0'
    1   0.000000 192.168.1.125 ? x.x.x.x WireGuard 190 Handshake Initiation, sender=0xF8014803
    2   5.219900 192.168.1.125 ? x.x.x.x WireGuard 190 Handshake Initiation, sender=0x65088075
    3  10.419816 192.168.1.125 ? x.x.x.x WireGuard 190 Handshake Initiation, sender=0x9A42E713
    4  15.599741 192.168.1.125 ? x.x.x.x WireGuard 190 Handshake Initiation, sender=0xDD48EACC
    5  20.689660 192.168.1.125 ? x.x.x.x WireGuard 190 Handshake Initiation, sender=0xAB5DDADD
    6  25.819580 192.168.1.125 ? x.x.x.x WireGuard 190 Handshake Initiation, sender=0x0226D5DE
^C6 packets captured
Here's the ifconfig output for wg0:

Code:
wg0: flags=80c3<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1420
        description: WireGuard interface
        index 6 priority 0 llprio 3
        wgport 36559
        wgrtable 1
        wgpubkey <client pubkey>
        wgpeer <server pubkey>
                wgendpoint x.x.x.x 51820
                tx: 0, rx: 0
                wgaip 0.0.0.0/0
        groups: wg
        inet 10.0.0.11 netmask 0xffffff00 broadcast 10.0.0.255

Last edited by Reeshar; 7th February 2022 at 11:26 AM. Reason: Added information
Reply With Quote
  #2   (View Single Post)  
Old 7th February 2022
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Hello, and welcome!

I tested but never implemented rdomain separation with WireGuard, as I need to be able to have my OpenBSD client disable / enable the VPN easily by just setting the wg(4) driver down or up. With routing domains, the VPN needed to be used full-time.

OpenBSD developer solene@ has a blog entry with example provisioning: https://dataswamp.org/~solene/2021-1...uard-exit.html
Reply With Quote
  #3   (View Single Post)  
Old 7th February 2022
Reeshar Reeshar is offline
Real Name: Richard L
Port Guard
 
Join Date: Feb 2022
Location: London, UK
Posts: 14
Default

Yes, I've previously read that and also:

https://codimd.laas.fr/s/NMc3qt5PQ

I don't see anything in those that conflicts with what I've done.

Unfortunately there's been very little written on this subject from real practical experience with the latest OBSD. For example Solène writes:

This guide should work from OpenBSD 6.9. (My use of bold).

Hence I'm appealing to others to give it a whirl! I've also tried testing with/without rdomain setups with a free 7-day WireGuard setup with vpnjantit:

(https://www.vpnjantit.com/free-wireguard)

with similar issues. It works without rdomains but not with.
Reply With Quote
  #4   (View Single Post)  
Old 7th February 2022
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

If I have the time this week, I will build a test client environment that uses two routing domains.
Reply With Quote
  #5   (View Single Post)  
Old 7th February 2022
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

I had time to test. I built a lab with 3 systems: "client" - "internet" - "server". The "internet" was just a 2-NIC router so that the client and server systems were using different subnets, in order to test routing. I used static addressing, to ensure my understanding of routing, and started with Solene's blog for configuring the client, primarily because I couldn't remember what I'd tested, and her work is still available. (I think I'd tested rdomains before she'd published her guide.)
  • The tunnel endpoints are the server's IP address - 1.2.3.4 - and the client's IP address - 172.16.10.2.
  • I needed to define two default routes. One in the hostname.vio0 configuration in rdomain 1, and one in the hostname.wg0 configuration in rdomain 0.
/etc/hostname.vio0:
Code:
rdomain 1
172.16.10.2/24
!route -T 1 add default 172.16.10.1
/etc/hostname.wg0:
Code:
wgkey <key>
wgpeer <key> wgendpoint 1.2.3.4 4433 wgaip 0.0.0.0/0
inet 192.168.10.2/24
wgrtable 1
up
!route add default 192.168.10.1
This works for me, and I think it's because there is a default route for rdomain 1.
Reply With Quote
  #6   (View Single Post)  
Old 10th February 2022
Reeshar Reeshar is offline
Real Name: Richard L
Port Guard
 
Join Date: Feb 2022
Location: London, UK
Posts: 14
Default

Great! Thanks for trying that jggimi.

But I have to say I am completely gob-smacked (to use an old English expression)! I've started up my RPi config today - exactly the same as had failed before - and it is now working. Working to my local server and working to a remote WireGuard server (vpnjantit). I haven't changed anything!! I really don't understand because I had rebooted this config several times in the past to no avail. And now on booting it's worked.

For me, the default route for rdomain 1 is already present without needing to add it because I'm using dhcpleased on that interface:

Code:
-bash-5.1$ route -n -T 1 show -inet
Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            192.168.1.1        UGS        1        7     -     8 bse0 
192.168.1/24       192.168.1.125      UCn        2      139     -     4 bse0 
192.168.1.1        00:e0:67:21:cd:21  UHLch      1      151     -     3 bse0 
192.168.1.125      e4:5f:01:4a:00:b9  UHLl       0     3926     -     1 bse0 
192.168.1.139      84:39:be:6d:c9:a7  UHLc       1        7     -     3 bse0 
192.168.1.255      192.168.1.125      UHb        0       59     -     1 bse0
I've just added (as before) a default route for rdomain 0:

Code:
doas route add default -link -iface wg0
and the corresponding routing table (for vpnjantit):

Code:
-bash-5.1$ route -n show -inet
Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            link#6             ULS        5     3405     -     8 wg0  
224/4              127.0.0.1          URS        0      106 32768     8 lo0  
127/8              127.0.0.1          UGRS       0        0 32768     8 lo0  
127.0.0.1          127.0.0.1          UHhl       1        2 32768     1 lo0  
192.168.6/24       192.168.6.111      UCn        0        0     -     4 wg0  
192.168.6.111      wg0                UHl        0       90     -     1 wg0  
192.168.6.255      192.168.6.111      UHb        0        0     -     1 wg0  
192.168.101/24     192.168.101.1      UCn        1        0     -     4 axen0
192.168.101.1      00:50:b6:ec:c0:25  UHLl       0       64     -     1 axen0
192.168.101.100    f8:e4:3b:4f:ad:c2  UHLc       3     3247     -     3 axen0
192.168.101.255    192.168.101.1      UHb        0        1     -     1 axen0
I don't seem to have much success with route commands embedded in the hostname.wg0 file, but now things seem to be working I'll try again!


I'll investigate further with the info you've provided!
Reply With Quote
  #7   (View Single Post)  
Old 10th February 2022
Reeshar Reeshar is offline
Real Name: Richard L
Port Guard
 
Join Date: Feb 2022
Location: London, UK
Posts: 14
Default

And now having moved some physical network connections around it's not working again...

Totally weird.

And yet I can ping the Internet from rdomain 1...
Reply With Quote
  #8   (View Single Post)  
Old 10th February 2022
Reeshar Reeshar is offline
Real Name: Richard L
Port Guard
 
Join Date: Feb 2022
Location: London, UK
Posts: 14
Default

And now I've finally sorted it - I think anyway. It's the clock!

Normally an RPi sets its clock on boot-up as it doesn't have a built-in RTC. But when you create a separate rdomain 1 for the Internet side of the Pi, ntpd can't access any time servers. So the clock isn't set and WireGuard doesn't work. Remove the domain separation and ntpd can once again access time servers and so WireGuard works.

I've tested this by using rdate to set the time against a cloudflare time server:

Code:
doas route -T 1 exec rdate 162.159.200.123
and everything was working again.

Good grief that took a while to work out!!

I've just got to sort out starting ntpd using rtable 1.
Reply With Quote
  #9   (View Single Post)  
Old 10th February 2022
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

See the route(8) man page for # route -T <rtable> exec.
Reply With Quote
Old 11th February 2022
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Sorry, you were already aware of route exec. The easiest way to set this at boot time would be to provision the ntpd_rtable flag via rcctl(8). See the rc.d(8) man page for a description of this flag. There's an example of reviewing this flag in the rcctl(8) man page.

Last edited by jggimi; 11th February 2022 at 04:13 AM. Reason: added last sentence
Reply With Quote
Old 11th February 2022
Reeshar Reeshar is offline
Real Name: Richard L
Port Guard
 
Join Date: Feb 2022
Location: London, UK
Posts: 14
Default

Yup, that's exactly what I've done. I used:

Code:
# rcctl set ntpd rtable 1
which has the effect of adding

Code:
ntpd_rtable=1
to my /etc/rc.conf.local file.

Presumably I could have added the flag manually.

The only remaining thing that puzzles me slightly is dhpleased. If I do

Code:
# ps ax -O rtable
dhpleased is in rdomain 0, yet it has been able to obtain an IP address on the external Internet-facing interface which is in rdomain 1.

So should I move dhcpleased to rtable 1 as with ntpd? Or does dhcpleased work across rdomains?

Once I've sorted all of this, I put together a separate post on how to run WireGuard with rdomains on a RPi in case anyone else in interested.

Incidentally, although I show commands as root, I actually use doas as I always have done. But what I didn't realise was that ifconfig gives different details if you execute it as a user and if you execute it as root. So to get a full description of the wg0 interface you need to execute ifconfig as root:

Code:
bash-5.1$ ifconfig wg0
wg0: flags=80c3<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1420
        description: vpnjantit WireGuard connection
        index 6 priority 0 llprio 3
        wgport 16146
        wgrtable 1
        groups: wg egress
        inet 192.168.6.111 netmask 0xffffff00 broadcast 192.168.6.255
...as against:

Code:
bash-5.1$ doas ifconfig wg0
wg0: flags=80c3<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1420
        description: vpnjantit WireGuard connection
        index 6 priority 0 llprio 3
        wgport 16146
        wgrtable 1
        wgpubkey DRWTID5iUrkEDczOv+Y1KavaUWRnlDCmQMZoYjKyWWE=
        wgpeer DiUst9llpM3ROfXwHueAFu+seOqUw8ihqIKyiKLqbmA=
                wgendpoint 188.119.148.113 1024
                tx: 697980, rx: 1743164
                last handshake: 63 seconds ago
                wgaip ::/0
                wgaip 0.0.0.0/0
        groups: wg egress
        inet 192.168.6.111 netmask 0xffffff00 broadcast 192.168.6.255
I've not found this difference documented anywhere (but I might have missed it!) and existing descriptions of installing WireGuard all seem to have been done as root, so just say "Use ifconfig to see the configuration".

Last edited by Reeshar; 11th February 2022 at 08:19 AM.
Reply With Quote
Old 11th February 2022
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

  • The DHCP protocol does not use IP routing. Access to local subnets is not blocked.
  • Display of public keys is a considered a security risk as it exposes an attack vector even though it does not expose a private key. You may want to change your key pairs, now.
Reply With Quote
Old 11th February 2022
Reeshar Reeshar is offline
Real Name: Richard L
Port Guard
 
Join Date: Feb 2022
Location: London, UK
Posts: 14
Default

The key is only valid for another day or so. vpnjantit allocates free WireGuard for 7 days at a time and I'm only using for testing. It's not my prime WireGuard details. Hence I'm not bothered about exposing the details.

Good to know about DHCP. I thought the process running in rdomain 0 might prevent it from accessing the external interface.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
pf - wireguard vazaro OpenBSD General 10 1st November 2021 05:19 PM
Screenshots of WireGuard Android app J65nko General software and network 1 5th June 2021 12:00 AM
FreeBSD port of Wireguard Code Quality Disaster bashrules News 3 27th March 2021 08:34 AM
WireGuard: replacement for IPsec e1-531g News 15 16th March 2021 09:08 AM
ARP Issue: Bridging, Routing, and FreeBSD LAGGs jasonvp FreeBSD Security 14 5th December 2015 05:35 PM


All times are GMT. The time now is 11:44 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick