Dual ISP Equal Cost Multi-path routing

[perfopt]

Hello

I have an OpenBSD firewall router for my home. Currently it works fine connected to an ISP and my home network. The firewall also runs DHCPD and DNS (Unbound). I followed instructions at https://openbsdrouterguide.net/ along with relevant man pages.

I have two ISPs and want to set up the router to do Equal Cost Multi Path routing. I followed instructions here:

https://www.openbsd.org/faq/faq6.html#Multipath

I first deleted the existing default routes. Then added the two multipath routes to both ISPs.

I enabled multipath option uusing sysctl

I tried the traceroute example in the FAQ but I see traffic going through only one default route.

With both ISPs connected I am able to ping google.com from the firewall but web browsers behind the firewall are unable to access websites. Some websites work and some don't.

How can I debug this problem? I would like to get ECMP working and later use ifstated for handling ISP failure.

[jggimi]

Hello, and welcome!


The only time I tested equal-cost routing, I used two NICs, following the FAQ. A clear difference with your scenario appears to be your single-NIC outbound provisioning. I don't know if that is the reason for the problems you are experiencing, but it is certainly something to consider.

[perfopt]

Hello

My router uses multiple physical NICs - ISO1 is on em0 and ISP2 on em1. My home network is on em3. Currently em2 is unused and will be setup later as a DMZ

[jggimi]

Sorry, I'd misread your post when I had too much blood in my caffeine system. I'd thought I saw two gateway addresses sharing the same subnet. I've looked again, and now see my error of assumption.

[perfopt]

Found the error. I had incorrectly setup NAT

Below is the corrected version:

Excerpts from my pf.conf

ext_if= "{ em0, em1 }" #WAN ports for ISPs
ext_if0="em0" #ISP1
ext_if1="em1" #ISP2

#------------------------#
# NAT
#------------------------#

pass out on $ext_if0 inet from $g_lan:network to any nat-to ($ext_if0)
pass out on $ext_if1 inet from $g_lan:network to any nat-to ($ext_if1)

Earlier I had
pass out on $ext_if inet from $g_lan:network to any nat-to ($ext_if)

That was screwing things up.