DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD General

OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 8th November 2010
SlyM SlyM is offline
Port Guard
 
Join Date: Nov 2010
Posts: 21
Thanked 0 Times in 0 Posts
Default Multi-Path or Route-To?

Hello everyone,

I'd like to ask all of those who have serious in-depth knowledge of how the OpenBSD kernel works and how pf works to answer a possibly simple, possibly complicated question.

Basically what I want to know is how the multipath routing in the OpenBSD kernel works. Is it meant to work for multiple routers available on the same subnet or can they be on separate subnets? I am trying to use only 1 NIC for egress traffic to the Internet. Both routes are actually 2 separate ISP connections (say cable and dsl) over a single NIC with both lines connected to it via a switch and ISP1's IP assigned to the NIC as the primary IP and the IP of ISP2 aliased onto the same NIC and then using multipath default routes to define the default gateways of each ISP connection.

What I'm basically trying to do it load balance 2 Internet connections with my OBSD firewall. But with simple multipath routes it sort of works, but I suffer a whole heck of a lot of packet loss.

What I ultimately found using line snooping Ethernet sniffers is that the firewall would sometimes attempt to send traffic from one ISP's IP address to the default gateway of the other ISP's connection. I assumed this would not happen since each ISP connection has a small 5 IP subnet and I did specify the netmask and the default gateway for each ISP is within its subnet. I made the seemingly logical assumption that it would intelligently route traffic from ISP1's IP to ISP1's default gateway and ISP2's IP to ISP2's default gateway. But it doesn't seem to be doing that, picking a default gateway seemingly at random without taking into consideration the source IP its employing at the moment.

Why is this? Am I misunderstanding what multipath routes are meant for? Or is this a bug in the kernel/my configuration of multipath?

I found a seemingly slightly better working solution with just using Route-to rules in pf. I now no longer suffer from packets going down the wrong default gateway, but I only have the default gateway for ISP1's connection defined for this to work right.

Is multipath routing only meant to work for routers on the same subnet?
Reply With Quote
  #2   (View Single Post)  
Old 8th November 2010
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,801
Thanked 214 Times in 189 Posts
Default

Quote:
Originally Posted by SlyM View Post
I am trying to use only 1 NIC...and ISP1's IP assigned to the NIC as the primary IP and the IP of ISP2 aliased onto the same NIC
Alias IP addresses are not designed for routing. They merely give the NIC multiple IP addresses to respond to. But they will always respond with their "real" address as the origination.

That will -usually- work with stateful connection protocols like TCP, as there are TCP sequence numbers, but will of course fail with any stateless connections -- UDP and ICMP being the two that come immediately to mind.

(I discovered this with UDP on some aliased addresses years ago.)

Add another NIC, and use two separate subnets for Multipath routing. Avoid alias IP addresses except for TCP-only activity, such as web servers. FAQ 6.14 shows an example of equal cost routing -- using two interfaces.
Reply With Quote
  #3   (View Single Post)  
Old 8th November 2010
SlyM SlyM is offline
Port Guard
 
Join Date: Nov 2010
Posts: 21
Thanked 0 Times in 0 Posts
Default

> Alias IP addresses are not designed for routing. They merely give the NIC multiple IP > addresses to respond to. But they will always respond with their "real" address as > the origination.

Thanks, I was not aware of this. Would explain the still somewhat odd behavior I am seeing with UDP traceroutes while other things TCP based (ie web surfing) seems to be working flawlessly.
Reply With Quote
  #4   (View Single Post)  
Old 10th November 2010
SlyM SlyM is offline
Port Guard
 
Join Date: Nov 2010
Posts: 21
Thanked 0 Times in 0 Posts
Default Now with 2 NICs installed...

Now I'd like to understand with 2 NICs installed for egress traffic, 1 for each ISP connection. With multipath default routes to each ISP's respective default gateway next hops, why is it that the box only uses the secondary ISPs connection exclusively?

If I issue a $ping 8.8.8.8

The ping times are appropriate for the usual latency of the secondary connection. This is confirmed by issuing a $traceroute 8.8.8.8 and watching traffic go down that route.

If I issue a $ping -I {IP of the primary connection} 8.8.8.8

The pings work, and have the usual latency that can be expected on that line. This can also be confirmed by issuing a $traceroute -s {IP of the primary connection} 8.8.8.8 and watching the traffic go down the primary line this time.

Then to really make matters worse, if I issue a $ping -I {IP of the secondary connection} 8.8.8.8
This fails....as does $traceroute -s {IP of secondary connection} 8.8.8.8

Why? It should work if it works when not specifying any address. And when not specifying any address, shouldn't it pick a connection to use at random or using a round-robin method with multipath routing enabled, or am I misunderstanding something again?
Reply With Quote
  #5   (View Single Post)  
Old 10th November 2010
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,801
Thanked 214 Times in 189 Posts
Default

It appears you did not follow FAQ 6.14, or did not follow it correctly. Note the section on sysctl settings.

From route(8), highlight mine:
Quote:
The optional -mpath modifier needs to be specified with the add command
to be able to enter multiple gateways for the same destination address
(multipath). When multiple routes exist for a destination, one route is
selected based on the source address of the packet. The sysctl(8)
variables net.inet.ip.multipath and net.inet6.ip6.multipath are used to
control multipath routing. If set to 1, multiple routes with the same
priority are used equally; if set to 0, the first route selected will be
used for subsequent packets to that destination regardless of source
.
Reply With Quote
  #6   (View Single Post)  
Old 10th November 2010
SlyM SlyM is offline
Port Guard
 
Join Date: Nov 2010
Posts: 21
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by jggimi View Post
It appears you did not follow FAQ 6.14, or did not follow it correctly. Note the section on sysctl settings.

From route(8), highlight mine:
Right, I did this...hence my confusion...I did read the that. What it conveniently leaves out is if this is meant to have multiple routes defined for a single source IP, or does it still work if you have more than one IP with default gateways each to the same basic (Internet) destination.
Reply With Quote
  #7   (View Single Post)  
Old 10th November 2010
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,801
Thanked 214 Times in 189 Posts
Default

Did you, by chance, use the same subnet for both interfaces? That's a mistake, if so. Each NIC must be on its own subnet.
Reply With Quote
  #8   (View Single Post)  
Old 10th November 2010
SlyM SlyM is offline
Port Guard
 
Join Date: Nov 2010
Posts: 21
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by jggimi View Post
Did you, by chance, use the same subnet for both interfaces? That's a mistake, if so. Each NIC must be on its own subnet.
NIC1 has an IP from the subnet of the primary ISP. And NIC2 has an IP from the secondary ISP. They are on totally separate subnets from two totally different ISPs.
Reply With Quote
  #9   (View Single Post)  
Old 10th November 2010
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,801
Thanked 214 Times in 189 Posts
Default

OK. There are a few possibilities for misconfiguration, such as a manual sysctl(8) command but a forgotten edit of sysctl.conf(5) and then a reboot. But just to be sure, post the outputs of:
  • ifconfig -A
  • route -n show -inet
Also post the contents of your hostname.* files
Reply With Quote
Old 10th November 2010
SlyM SlyM is offline
Port Guard
 
Join Date: Nov 2010
Posts: 21
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by jggimi View Post
OK. There are a few possibilities for misconfiguration, such as a manual sysctl(8) command but a forgotten edit of sysctl.conf(5) and then a reboot. But just to be sure, post the outputs of:
  • ifconfig -A
  • route -n show -inet
Also post the contents of your hostname.* files
Here is $ifconfig -A


lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33200
priority: 0
groups: lo
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
re0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,S IMPLEX,MULTICAST> mtu 1500
lladdr 00:14:d1:1c:ee:a8
priority: 0
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet6 fe80::214:d1ff:fe1c:eea8%re0 prefixlen 64 scopeid 0x1
re1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,S IMPLEX,MULTICAST> mtu 1500
lladdr 00:14:d1:1c:f3:f7
priority: 0
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
inet 192.168.xxx.yyy netmask 0xfffffe00 broadcast 192.168.xxx.255
inet6 fe80::214:d1ff:fe1c:f3f7%re1 prefixlen 64 scopeid 0x2
msk0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MU LTICAST> mtu 1500
lladdr 00:1b:b9:b0:07:80
priority: 0
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet6 fe80::21b:b9ff:feb0:780%msk0 prefixlen 64 scopeid 0x3
enc0: flags=0<>
priority: 0
groups: enc
status: active
pfsync0: flags=41<UP,RUNNING> mtu 1500
priority: 0
pfsync: syncdev: re1 maxupd: 128 defer: off
groups: carp pfsync
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33200
priority: 0
groups: pflog
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:02
priority: 0
carp: MASTER carpdev re0 vhid 2 advbase 1 advskew 0
groups: carp egress
status: master
inet6 fe80::200:5eff:fe00:102%carp0 prefixlen 64 scopeid 0x6
inet 97.67.xxx.yyy netmask 0xfffffff8 broadcast 97.67.xxx.yyy
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:01
priority: 0
carp: MASTER carpdev re1 vhid 1 advbase 1 advskew 0
groups: carp
status: master
inet6 fe80::200:5eff:fe00:101%carp1 prefixlen 64 scopeid 0x7
inet 192.168.xxx.yyy netmask 0xfffffe00 broadcast 192.168.xxx.255
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:03
priority: 0
carp: MASTER carpdev msk0 vhid 3 advbase 1 advskew 0
groups: carp egress
status: master
inet6 fe80::200:5eff:fe00:103%carp2 prefixlen 64 scopeid 0x8
inet 68.213.xxx.yyy netmask 0xfffffff8 broadcast 68.213.xxx.yyy

Here is $route -n show -inet

Routing tables

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 68.213.xxx.yyy UGSP 0 17 - 8 carp2
default 97.67.xxx.yyy UGSP 0 9 - 8 carp0
68.213.xxx.yyy/29 link#8 UC 1 0 - 4 carp2
68.213.xxx.yyy 00:22:10:a3:22:e0 UHLc 1 0 - 4 carp2
97.67.xxx.yyy/29 link#6 UC 1 0 - 4 carp0
97.67.xxx.yyy 00:a0:c8:5b:1c:bb UHLc 1 0 - 4 carp0
127/8 127.0.0.1 UGRS 0 0 33200 8 lo0
127.0.0.1 127.0.0.1 UH 1 60 33200 4 lo0
192.168.xxx/23 link#2 UC 9 0 - 4 re1
192.168.xxx.yyy 192.168.xxx.yyy UH 0 0 - 4 carp1
224/4 127.0.0.1 URS 0 0 33200 8 lo0

Here are the /etc/hostnames.*


/etc/hostname.carp0
inet 97.67.xxx.yyy 255.255.255.248 97.67.xxx.yyy vhid 2 pass password1 carpdev re0
!route add -mpath default 97.67.37.89

/etc/hostname.carp1
inet 192.168.xxx.1 255.255.254.0 192.168.xxx.255 vhid 1 pass password2 carpdev re1

/etc/hostname.carp2
inet 68.213.xxx.yyy 255.255.255.248 68.213.xxx.yyy vhid 3 pass password3 carpdev msk0
!route add -mpath default 68.213.215.9

/etc/hostname.msk0
up

/etc/hostname.pfsync0
up syncdev re1

/etc/hostname.re0
up

/etc/hostname.re1
inet 192.168.xxx.11 255.255.254.0 192.168.xxx.255
up
rtsol

I changed the IP addresses for the usual privacy reasons.
Reply With Quote
Old 10th November 2010
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,801
Thanked 214 Times in 189 Posts
Default

I believe your carp(4) implementation might be getting in the way. As configured, carp is not being utilized for load balancing.. You might consider using real interfaces. See if it makes a difference.

If not, or for some real help instead of my fumblings, consider posting to misc@.

Next time, please enclose such pasted content in code tags. It makes things like routing tables much easier to read. As posted, I'm having trouble with it.
Reply With Quote
Old 10th November 2010
SlyM SlyM is offline
Port Guard
 
Join Date: Nov 2010
Posts: 21
Thanked 0 Times in 0 Posts
Default

Sorry about not using the code tags...not familiar with that. Will have to spend some time on the forums guides to learn how to do this.

As for the carp interfaces, I'm using them between to similarly configured boxes to not only load balance 2 Internet connections, but also to be redundant firewalls of each other. So technically they are being used. If they are working the way I expect...that's another thing. What makes you say they aren't being used, as is currently configured?
Reply With Quote
Old 10th November 2010
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,801
Thanked 214 Times in 189 Posts
Default

I wasn't clear. Currenlty configured for load balancing (ARP or IP).
Reply With Quote
Old 10th November 2010
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,801
Thanked 214 Times in 189 Posts
Default

To be even clearer, I don't see a "carpnodes" configuration clause in your hostname.carp* files.
Reply With Quote
Old 10th November 2010
BSDfan666 BSDfan666 is offline
Real Name: N/A, this is the interweb.
Helpful companion
 
Join Date: Apr 2008
Location: Ontario, Canada
Posts: 2,223
Thanked 193 Times in 184 Posts
Default

This forum uses BBcode, which is similar to HTML markup.. simply place the text between the tags.

[code]Formatted output goes here.[/code]
Reply With Quote
Old 10th November 2010
SlyM SlyM is offline
Port Guard
 
Join Date: Nov 2010
Posts: 21
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by jggimi View Post
I wasn't clear. Currenlty configured for load balancing (ARP or IP).
I'm not entirely sure what you mean by that. If I understand correctly, I'm trying to achieve IP load balancing...
Reply With Quote
Old 10th November 2010
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,801
Thanked 214 Times in 189 Posts
Default

Search for "carpnodes" in the carp(4) man page.
Reply With Quote
Old 10th November 2010
SlyM SlyM is offline
Port Guard
 
Join Date: Nov 2010
Posts: 21
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by jggimi View Post
Search for "carpnodes" in the carp(4) man page.
Oh I apologize...I misunderstood. I am not trying to use carp for any load balancing. Just for fail-over...should one firewall die/get disconnected/administratively shutdown for whatever reason...the other takes over. This does work as expected..am quite happy with this.

I'm attempting to use Equal Cost Multipath routing to load balance traffic from the Internal LAN (and the firewall itself I would assume) between the two Internet connections. This is what is not working.
Reply With Quote
Old 10th November 2010
SlyM SlyM is offline
Port Guard
 
Join Date: Nov 2010
Posts: 21
Thanked 0 Times in 0 Posts
Default

Should I just do away with multipath routes and just use pf route-to rules for this?
Reply With Quote
Old 10th November 2010
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,801
Thanked 214 Times in 189 Posts
Default

Thank you for the clarification.

I would assume that each server at each end of a carp cluster must have the same routing tables.

If they do, then you have exceeded my capabilities to assist you further. Since no one else has jumped in, I recommend reaching out the the broader support community at the misc@ mailing list. If you have not used it before, please read http://www.openbsd.org/mail.html before posting there.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
How to add static route using virtual NIC bsdplus Solaris 1 22nd August 2010 02:10 AM
add to path carpman FreeBSD General 4 19th August 2010 06:13 AM
ping: sendto: No route to host joostvgh OpenBSD General 2 29th April 2010 12:34 PM


All times are GMT. The time now is 05:34 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick