I had no difficulty setting up the lab this evening, here, and running a set of tests. I used 4.9-release systems.
My "router" had the following configuration changes. The changes to sysctl.conf are shown as a patch against the 4.9-release code.
I added the following files:
hostname.em0 (connection to ISP #1)
Code:
inet 10.0.1.4/24
!route add -mpath default 10.0.1.1
hostname.em1 (connection to ISP #2)
Code:
inet 10.0.2.4/24
!route add -mpath default 10.0.2.2
hostname.em2 (connection to back-end server)
Code:
Index: sysctl.conf
===================================================================
RCS file: /cvs/src/etc/sysctl.conf,v
retrieving revision 1.49
diff -u -r1.49 sysctl.conf
--- sysctl.conf 16 Feb 2011 10:37:45 -0000 1.49
+++ sysctl.conf 16 Sep 2011 23:03:45 -0000
@@ -4,9 +4,9 @@
# boot time. See sysctl(3) and sysctl(8) for more information on
# the many available variables.
#
-#net.inet.ip.forwarding=1 # 1=Permit forwarding (routing) of IPv4 packets
+net.inet.ip.forwarding=1 # 1=Permit forwarding (routing) of IPv4 packets
#net.inet.ip.mforwarding=1 # 1=Permit forwarding (routing) of IPv4 multicast packets
-#net.inet.ip.multipath=1 # 1=Enable IP multipath routing
+net.inet.ip.multipath=1 # 1=Enable IP multipath routing
#net.inet.icmp.rediraccept=1 # 1=Accept ICMP redirects
#net.inet6.icmp6.rediraccept=0 # 0=Don't accept IPv6 ICMP redirects
#net.inet6.ip6.forwarding=1 # 1=Permit forwarding (routing) of IPv6 packets
I was able to connect ssh sessions from both ISPs, and from my "Internet" user machine, using either ISP as the gateway route. Here's an example of three ssh sessions: 10.0.1.1 and 10.0.2.2 are the ISPs, 10.0.0.3 is the internet user. In this instance, routed through ISP 2.
Code:
# w
7:15PM up 21 mins, 4 users, load averages: 0.28, 0.21, 0.11
USER TTY FROM LOGIN@ IDLE WHAT
root 00 - 6:55PM 0 w
root p0 10.0.2.2 6:57PM 16 -ksh
root p1 10.0.0.3 6:58PM 15 -ksh
root p2 10.0.1.1 7:00PM 0 -ksh
I thought I would post this information for you, and then start on the "local server" and NAT configurations.