DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD Security

OpenBSD Security Functionally paranoid!

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 5th December 2011
alpha202ej alpha202ej is offline
Port Guard
 
Join Date: Dec 2011
Posts: 10
Thanked 0 Times in 0 Posts
Default PF Dual WAN Fail Over Issue (one box)

Hello,

Just as a fair warning, I am an OpenBSD noobie. If your still reading this, thank you

We have two internet connections at our location and one OpenBSD box with PF installed. We want it to replace our current firewall but this issue I am seeing is that if I try to simulate one of the connections going down the internet stops working. I have my PF setup already setup to do round robin for the two connections and it works great when they are both connected.

My assumption going into this is that it would simply draw on the one connection for the internet traffic until the connection comes back up but this doesn't seem to be the case.

What would I need to do to have it fail over to the working connection?

Thanks!
Reply With Quote
  #2   (View Single Post)  
Old 5th December 2011
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,640
Thanked 214 Times in 189 Posts
Default

Hello,and welcome!

Two possibilities for you to consider:
  • Equal Cost Multipath Routing -- ECMR -- described in FAQ 6.14.
  • ifstated(8) to alter your routing tables even when ECMR is not used.
Reply With Quote
  #3   (View Single Post)  
Old 5th December 2011
alpha202ej alpha202ej is offline
Port Guard
 
Join Date: Dec 2011
Posts: 10
Thanked 0 Times in 0 Posts
Default

Thanks for the information! I will learn how these work and try to use those.

I'll post back with my findings! Thanks!
Reply With Quote
  #4   (View Single Post)  
Old 8th December 2011
alpha202ej alpha202ej is offline
Port Guard
 
Join Date: Dec 2011
Posts: 10
Thanked 0 Times in 0 Posts
Default

Still having issues with dual wan with fail over. I was able to implement Equal Cost Multipath Routing but I am not sure how to implement it in conjunction with ifstate. As far as I can tell the ifstate is a sample configuration meant for use with CARP. I followed the example on OpenBSD's FAQ 6.14.

Below is my pitiful pf.conf

Code:
############
## MACROS ##
############

# Interfaces
int_if  = "vr0" #Internal Interface
ext_if1 = "vr1" #External Interface T1
ext_if2 = "axe0" #External Interface WiMAX
ext_gw1 = "x.x.x.x"
ext_gw2 = "y.y.y.y"
vpn_if  = "tun0" #VPN Tunnel to FSJ
vlan100 = "vlan100" #VLAN Interface 10.0.2.0/24

# Misc
localnet = "10.0.0.0/24"
vlan100net = "10.0.2.0/24"
SH_Ext = "z.z.z.z"
nonroute = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, \
              10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, \
              0.0.0.0/8, 240.0.0.0/4 }"
snglextports = "{ 10000, 443, 1194, 22 }"


# Addresses
MFC8780 = "10.0.0.5"

# Ports
RDP = "{ 3389 }"

############
## TABLES ##
############


####################
## GLOBAL OPTIONS ##
####################

set skip on lo0

###########################
## TRAFFIC NORMALIZATION ##
###########################


####################
## QUEUEING RULES ##
####################


#########################################
## TRANSLATION RULES (NAT) & SCRUBBING ##
#########################################

match in log
match in all scrub (no-df max-mss 1440)
match out on $ext_if1 from $localnet nat-to ($ext_if1)
match out on $ext_if2 from $localnet nat-to ($ext_if2)
match out on $ext_if2 from $vlan100net nat-to ($ext_if2) # Limiting VLAN 100 to WiMAX only

###################
## FILTERS RULES ##
###################

antispoof for $int_if
antispoof for $ext_if1
#antispoof for $ext_if2

block in log
#block out log

# Block Non-Routable Addresses
#block drop in quick on $ext_if from $nonroute to any
#block drop out quick on $ext_if from any to $nonroute

pass inet proto { tcp, udp } from { self, $localnet, $vlan100net }

#pass out on $int_if to $localnet
#pass in quick on $int_if from $localnet to $int_if

# Single WAN
#pass out on $ext_if1 proto { tcp udp icmp } all

### Dual WAN Configuration
pass in on $int_if from $localnet route-to { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin
pass in on $vlan100 from $vlan100net route-to { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin

# Single External Connection Path for Secure Ports
pass in on $int_if proto { tcp, udp } from $localnet to port $snglextports route-to ($ext_if1 $ext_gw1)

# General "Pass Out" Rules For External Interface
pass out on $ext_if1
pass out on $ext_if2

# Route Packets From any IPs on on $ext_if1 to $ext_gw1 and the same for $ext_if2 and $ext_gw2
pass out on $ext_if1 from $ext_if2 route-to ($ext_if2 $ext_gw2)
pass out on $ext_if2 from $ext_if1 route-to ($ext_if1 $ext_gw1)

pass in on $int_if from $vlan100net to $MFC8780
#pass in quick on $ext_if1 inet proto tcp from any to any port 80 rdr-to 10.0.0.11 port 8008
#pass in on $ext_if1 inet proto tcp to $ext_if1 port 8008 rdr-to 10.0.0.11
#pass on $int_if inet proto tcp to 10.0.0.11 port 8008
pass in quick on $ext_if1 inet proto tcp from any to any port 8008 rdr-to 10.0.0.11
pass on $int_if inet proto tcp to 10.0.0.11 port 8008
pass in inet proto tcp to port ssh
pass inet proto icmp

# VPN
pass quick on $vpn_if
pass in quick proto udp from any to port 1194

block in on ! lo0 proto tcp to port 6000:6010
Any guidence with my configuration to accomplish this failover (on the same box) would be greatly appreciated.

Thank you!
Reply With Quote
  #5   (View Single Post)  
Old 8th December 2011
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,640
Thanked 214 Times in 189 Posts
Default

Here's a non-carp ifstated example from an article (circa 2007) in the OpenBSD Journal:

http://undeadly.org/cgi?action=artic...&mode=expanded
Reply With Quote
  #6   (View Single Post)  
Old 12th December 2011
alpha202ej alpha202ej is offline
Port Guard
 
Join Date: Dec 2011
Posts: 10
Thanked 0 Times in 0 Posts
Default

So using multiple examples I was able to make this:

Code:
net0 = '"ping -q -c1 -w1 x.x.x.x >/dev/null" every 3'
net1 = '"ping -q -c1 -w1 y.y.y.y >/dev/null" every 3'

init-state zero

state zero {
    init {
        run "route add -mpath default x.x.x.x"
        run "route add -mpath default y.y.y.y"
    }
    if ! $net0
        set-state one
    if ! $net1
        set-state two
        if $net0 {
                if $net1
                        set-state zero
                }

}

state one {
    init {
        run "route delete -mpath default x.x.x.x"
        run "route add -mpath default y.y.y.y"
    }
        if ! $net0
            set-state one
    if $net0 {
                if $net1
                        set-state zero
                }
}

state two {
    init {
        run "route delete -mpath default y.y.y.y"
        run "route add -mpath default x.x.x.x"
    }
    if ! $net1 {
        if $net0
            set-state one
    }
}
x.x.x.x is the first hop for my default gateway on my first ISP.
y.y.y.y is the first hop for my default gateway on my second ISP.

The script does a ping test of each of the ISPs and if one stops responding, it removes the route. Once it responds, it adds it back on.

It serves its purpose quite well in terms of removing the route when it stops resolving but in practice, I still can't get the fail over to work. Not only that, when browsing many web pages half load. Ping tests work just fine but if I am going to replace this I cannot have web pages half loading

As always, any help or ideas that would help with these issues would by greatly appreciated.

Thanks!
Reply With Quote
  #7   (View Single Post)  
Old 14th December 2011
alpha202ej alpha202ej is offline
Port Guard
 
Join Date: Dec 2011
Posts: 10
Thanked 0 Times in 0 Posts
Default

After putting my box into our environment, I was able to get it to work (reboot probably helped a bit too!). Thanks!
Reply With Quote
  #8   (View Single Post)  
Old 20th April 2012
raiashish raiashish is offline
New User
 
Join Date: Apr 2012
Posts: 1
Thanked 0 Times in 0 Posts
Default

Hello alpha202ej i am having exactly the same problem as you stated in the starting of this post and i would like to ask you that how you solved it ....... hope you will help

.... Thanks in advance
Reply With Quote
  #9   (View Single Post)  
Old 22nd April 2012
neant neant is offline
New User
 
Join Date: Apr 2012
Posts: 4
Thanked 0 Times in 0 Posts
Default

Just did the same thing, this thread helped so I thought I'd share, I took it a small step further.

In pf.conf I'm using these two lines:
Code:
pass out log on $ext1_if label "out_$if"
pass out log on $ext2_if label "out_$if"
in order to mark outgoing traffic on each interface. Then, when that interface goes down, I flush the routes on it and
Code:
pfctl -k label -k "out_$ext1_if"
kill all states associated with the interface. This way some connections get reinitiated and the user never knows something has gone wrong.

I'm pinging the gateway and a host outside my ISP that is likely to always be online, like google.com, so if google doesn't answer, the ISP probably has a problem and I bring down the interface. This has the potential to cause problems if the wrong host is chosen, but I can't figure out a better way to do it and only one interface is brought down at a time.

I'm using a single file with macro definitions that are common to both pf.conf and ifstated.conf, like $ext1_if, $ext1_gw, etc., this way if I have to change something I only have to change it in one place. Also, my run statements in ifstated.conf execute shell scripts, these have the macro file included in them too and send root an email when the state changes.

Tried to be brief, detailed here.

Last edited by neant; 24th April 2012 at 09:47 AM.
Reply With Quote
Old 24th April 2012
neant neant is offline
New User
 
Join Date: Apr 2012
Posts: 4
Thanked 0 Times in 0 Posts
Default

Well, I have a problem with it, my Google-Fu is failing me and I can't seem to find anything relevant.

The load balancing is done by the OS, using equal-cost multipath routing. I do NAT using match rules:
Code:
match out log on $ext1_if from <nated> nat-to $ext1_nat
match out log on $ext2_if from <nated> nat-to $ext2_nat
ext1_if = re0 and ext2_if = re1, these are the external interfaces and there's no restriction on outgoing traffic on egress.

The problem is, packets that get NATed to $ext1_if get routed on $ext2_if. For example, I have a webserver on the internal network, port 80 on $ext1_if gets redirected to the webserver:
Code:
pass in on $ext1_if proto tcp to $www_out port $www_tcp_ports rdr-to $www_in
This works fine, but after the connection is established I'm seeing (tcpdump) incoming packets to the server on re0 (ext1) and the outgoing responses on re1 (ext2), NATed to ext1's IP address. So the server gets queried on one interface and responds on the other one. The packets get to destination fine, but performance is horrible when this happens. OpenVPN, still on an internal server, is pretty much unusable.

I've been searching for what exactly happens inside the kernel, when is routing and NATing done, but can't find anything, I suppose it happens because the connection has state established so pf doesn't see the packets anymore and OS's routing doesn't know anything about what NAT is gonna do, but I'm not sure how to fix it, short of maybe changing everything and doing the load balancing in pf, which I'm kinda reluctant to do.

Can someone point me in the right direction?
Reply With Quote
Old 24th April 2012
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,640
Thanked 214 Times in 189 Posts
Default

neant, I recommend you prepare complete documentation of your configuration and post your question to the Project's misc@ mailing list. The number of active users who respond to OpenBSD threads on this forum is very small. I'm not aware of any of us who are using multipath routing with NAT and that is why I recommend opening a discussion with a wider audience.

Personally, I have only ever tested multipath with NAT in a laboratory environment and did not experience the results you are seeing. That could be due to the limited nature of my tests. But ... even if it were due to a component of your configuration that you did not share with us, should you share it here it is possible that no active member of this forum will be able to point you towards a root cause.

If you have never posted to misc@ before, please review http://www.openbsd.org/mail.html and http://www.openbsd.org/report.html for recommendations.
Reply With Quote
Old 24th April 2012
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,640
Thanked 214 Times in 189 Posts
Default

I described the multpath/NAT testing and configuration used in this post in another thread. FWIW.
Reply With Quote
Old 24th April 2012
neant neant is offline
New User
 
Join Date: Apr 2012
Posts: 4
Thanked 0 Times in 0 Posts
Default

Indeed, I've only seen it happen after we got the system into production, when there were a lot of connections going through. As I understand from quickly reading the RFC, the mpath routing algorithm tries to keep the same path, unless things change, so it's likely not gonna show up in a lab.
Anyway, I think I found the answer, it's in the thread you linked too and pretty clearly explained in pf.conf's man page (the *Routing* section even, who would've guessed, eh?):
Code:
reply-to
           The reply-to option is similar to route-to, but routes packets that
           pass in the opposite direction (replies) to the specified
           interface.  Opposite direction is only defined in the context of a
           state entry, and reply-to is useful only in rules that create
           state.  It can be used on systems with multiple external
           connections to route all outgoing packets of a connection through
           the interface the incoming connection arrived through (symmetric
           routing enforcement).
Maybe I should test more, but for now it seems to work fine. So, basically, this line:
Code:
pass in on $ext1_if proto tcp to $www_out port $www_tcp_ports rdr-to $www_in
becomes:
Code:
pass in on $ext1_if proto tcp to $www_out port $www_tcp_ports rdr-to $www_in reply-to ($ext1_if $ext1_gw)
or, in my actual case:
Code:
match in on $ext1_if proto tcp to $www_out port $www_tcp_ports rdr-to $www_in

[... other rules ...]

pass in on $ext1_if proto tcp to $www_in port $www_tcp_ports reply-to ($ext1_if $ext1_gw)
$www_out is the public IP of the webserver, $www_in is the private one. In this second case the pass rule has the internal address of the webserver, the packet is already headed to 192.168.x.x by the time it gets to it. Guess match rules actually change the packets, didn't know that, thought it would only happen after the pass rule.

Well, maybe it'll help some future googler.

Thanks, all seems good for now, even though I didn't get to do much testing, I'll se what else pops up. Mpath + NAT works for me.
Reply With Quote
Old 24th April 2012
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,640
Thanked 214 Times in 189 Posts
Default

Excellent!
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
DHCP Cable Connection Fail fmgil OpenBSD General 4 1st September 2009 05:18 PM
Mounting ext2 partitions seems to fail Sunsawe FreeBSD Installation and Upgrading 2 17th June 2009 01:38 PM
Basic networking fail. diw OpenBSD General 13 31st March 2009 09:29 AM
Dual/Quad NIC schrodinger General Hardware 12 28th February 2009 11:27 PM
RAID-1 over NFS with fail-over PatrickBaer FreeBSD General 0 12th October 2008 12:03 AM


All times are GMT. The time now is 12:29 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick