OpenBSD-5.4::PF config issue

[Atlantis]

greetings All !

I encounter problems while setting up a multihomed FW/GW under OpenBSD 5.4
To make things easier, i enabled multipath into sysctl.conf (tried both equal and unequal).
Setup is as follow

INTERNET ===|(dsl:82.22X.XX.XX) FREE-DSL-BOX (int:192.168.1.254) |===| (rl0)             (em1) |=== LAN
								|	     OPENBSD BOX       | 
INTERNET ===|(fttb:81.xx.xx.xx) NC-FTTB-BOX (int:192.168.0.1) |======| (re0)             (em0) |=== DMZ

=======

Now the question

Setup is working, but not so good. I would need to have all traffic to pass through NC in priority, and , in case of failure, switch to FREE (NC connection bandwidth is 200MBps, Free is 14MBps).
In parallel, i would need the GW to be reachable from FREE as well as NC networks (both providers boxes do NAT traffic as well...)
I am able to proceed when i disable either re0 or rl0, but when both are enabled, only one is working...
I believe it has something to do with reply-to sent to the wrong interface (?)
In parallel, i have a lot of packets losses, for an unknown reason (i log everything for now - debugging purposes)
Any ideas about how to setup this conf correctly and/or to optimize this setup ?
BTW: i tried to setup -mpath using both equal and unequal weights, without success, pb remains the same...

Thanks for your help !

sincerely
___________________
here are the interfaces and setup description:

===

# ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33192
        priority: 0
        groups: lo
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr XX:XX:18:XX:XX:7d
        description: Connexion Free
        priority: 0
        groups: egress
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 192.168.1.253 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::2e0:18ff:XXXX:XXXX%rl0 prefixlen 64 scopeid 0x1
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr XX:XX:6e:XX:XX:XX
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex,master)
        status: active
        inet 10.0.0.254 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::XXX:6eff:XXXX:XXXX%em0 prefixlen 64 scopeid 0x2
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:30:6e:XX:XX:XX
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet 172.16.1.254 netmask 0xffffff00 broadcast 172.16.1.255
        inet6 fe80::230:XXXX:XXXX:8e81%em1 prefixlen 64 scopeid 0x3
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:1e:XX:XX:XX:XX
        description: Connexion Numericable
        priority: 0
        groups: egress
        media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
        status: active
        inet 192.168.0.253 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 feXX::21e:XXXX:feXX:83XX%re0 prefixlen 64 scopeid 0x4
enc0: flags=0<>
        priority: 0
        groups: enc
        status: active
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33192
        priority: 0
        groups: pflog

=======



Routing table is:
====================

# route show
Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            192.168.0.1        UGSP       4   345002     -     8 re0
default            192.168.1.254      UGSP       4   444011     -     8 rl0
10.0.0/24          link#2             UC         3        0     -     4 em0
10.0.0.150         00:0c:XX:22:XX:f5  UHLc       1    40586     -     4 em0
loopback           localhost          UGRS       0        0 33192     8 lo0
localhost          localhost          UH         2     2154 33192     4 lo0
172.16.1/24        link#3             UC         3        0     -     4 em1
192.168.1.253      00:XX:XX:2a:XX:7d  UHLc       0      212     -     4 lo0
192.168.0/24       link#4             UC         2        0     -     4 re0
192.168.0.1        24:ec:XX:05:XX:4X  UHLc       1        5     -     4 re0
base-address.mcast localhost          URS        0        0 33192     8 lo0

===================

DMZ contains a mail server, which grabs mails from public services (yahoo,...) using pop3s, ntp server, and dns server.
Gateway itself manages with multidomain smtp routing, and incoming mails are forwarded to internal mail server, as well as DNS
===================

Here is the PF.conf setup

================

############################ INTERFACES INTERNES #################################

orange_if  = "em0"
orange_net = "10.0.0.0/24"

green_if  = "em1"
green_net = "172.16.1.0/24"

############################ INTERFACES EXTERNES #################################

nc  = "re0"
free = "rl0"

############################ PASSERELLES EXTERNES #################################

nc_gw = "192.168.0.1"
free_gw = "192.168.1.254"


############################ TABLES ##################################

table <ournets> persist { 10.0.0.0/24, 172.16.1.0/24 }
table <bruteforce> persist
table <ossec_fwtable> persist # ossec_fwtable
table <allowed_out> persist { }
table <firewall> const { self }

############################ PARAMETRES ##############################

set state-policy floating
set block-policy drop
set optimization normal
#set require-order yes
set timeout { interval 10, frag 30 }
set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
set timeout { tcp.closing 900, tcp.finwait 45, tcp.closed 90 }
set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
set timeout { icmp.first 20, icmp.error 10 }
set timeout { other.first 60, other.single 30, other.multiple 60 }
set timeout { adaptive.start 0, adaptive.end 0 }
set limit { states 5000, frags 2000 }
set limit table-entries 500000
set fingerprints "/etc/pf.os"
set skip on lo0

########################## fragment reassemble  ############################
# Normalisation de tous les paquets entrants.
match in all scrub (no-df)
#scrub out all fragment reassemble max-mss 1400

###################################### N  A  T  ##########################################

#  nat outgoing connections on each internet interface
match out on egress inet from !(egress:network) to any nat-to { egress }

###################################### REGLES DENY  #########################################

# block unwanted hosts
block in quick from <bruteforce>
block in quick from <ossec_fwtable>

# block anything by default
block in log
block out log

# ORANGE TO GREEN NOK
block drop in on $green_if from $orange_net to $green_net

###################################### REGLES ALLOW #########################################

# Allow ICMP on external interfaces

pass in quick on $green_if proto icmp from {$green_net, $orange_net} to any nat-to egress keep state


#pass in inet proto tcp from {$green_net, $orange_net} to any port { http, https } divert-to 127.0.0.1 port 3128

#  pass all outgoing packets on internal interface

pass out quick log on $green_if to $green_net
pass out quick log on $orange_if to $orange_net

################################################################################

# Autoriser le trafic sortant et entrant sur le r?seau local.
# ces r?gles cr?eront des entr?es au niveau de la table d'?tat ?tant
# donn? que le mot-cl? "keep state" est automatiquement appliqu?.

# on autorise le LAN et certains services a atteindre le firewall
pass in quick log on $green_if proto tcp from $green_net to ($green_if) port { ssh, 3128 } keep state
pass in quick log on $green_if proto udp from $green_net to ($green_if) port 53

# on autorise certains services en zone DMZ a etre atteints par le LAN

pass in quick log on $green_if proto tcp from $green_net to $orange_net port { smtp, http, https, ssh, 137, 139, 445, 993 } keep state
pass in quick log on $orange_if proto udp from $orange_net to !$green_net port { domain }
pass in quick log on $orange_if proto tcp from $orange_net to !$green_net port {25, 110, 993, 995 }

# on autorise la sortie sur internet

# Allow ICMP on external interfaces
pass in quick on $free proto icmp from <firewall> to any keep state
pass in quick on $nc proto icmp from <firewall> to any keep state


pass in quick log on $green_if proto tcp from $green_net to !$orange_net port { http, https }


pass out log on egress proto udp from { <firewall>, $green_net, $orange_net } to any port { domain, ntp } keep state (if-bound)
pass out log on egress proto tcp from { <firewall>, $green_net } to any port { domain, http, https, ntp } keep state (if-bound)
pass out log on egress proto tcp from $orange_net to any port { domain, smtp, ntp } keep state (if-bound)
pass out log on egress inet proto icmp all icmp-type { echoreq }

########################### REGLES EN ENTREE ##############################
######################## SERVICES EN ENTREE #########################

###################### E/SMTP/IMAPS MOBILE VIA FREE ######################
pass in quick log on $free inet proto tcp from any to $free port = 25 reply-to ($free $free_gw) flags S/SA keep state
#pass in quick log on $free inet proto tcp from any to $free port = 9930 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993

pass in quick log on $free inet proto tcp from XX.84.144.0/22 to $free port = 9930 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $free inet proto tcp from XX.84.144.0/22 to $free port = 9931 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25

pass in quick log on $free inet proto tcp from XX.10.159.0/24 to $free port = 9930 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $free inet proto tcp from XX.10.159.0/24 to $free port = 9931 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25

pass in quick log on $free inet proto tcp from XX.160.0.0/12  to $free port = 9930 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $free inet proto tcp from  XX.8.160.0/12 to $free port = 9931 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25
pass in quick log on $free inet proto tcp from any to $free port = 25 reply-to ($free $free_gw) flags S/SA keep state

################### E/SMTP/IMAPS MOBILE VIA NUMERICABLE ##################
#pass in quick log on $nc inet proto tcp from any to $nc port = 9930 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993

pass in quick log on $nc inet proto tcp from XX.84.144.0/22 to $nc port = 9930 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $nc inet proto tcp from XX.84.144.0/22 to $nc port = 9931 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25

pass in quick log on $nc inet proto tcp from XX.10.159.0/24 to $nc port = 9930 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $nc inet proto tcp from XX.10.159.0/24 to $nc port = 9931 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25

pass in quick log on $nc inet proto tcp from XX.160.0.0/12  to $nc port = 9930 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $nc inet proto tcp from XX.160.0.0/12  to $nc port = 9931 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25

===================

[jggimi]

Hello, and welcome!

Originally Posted by Atlantis

To make things easier, i enabled multipath into sysctl.conf (tried both equal and unequal).

This is for automatic multipath routing. When set to 1 (equal), routes added with -mpath use equal cost multipath routing, and when set to 0 (not-equal), only the first path added with -mpath will be used. It is unclear how you are adding the route definitions, but if -mpath is not used with route(8), and you are using PF rules only, then the OS will not test the sysctl setting. Equal cost multipath routing is described and an example is shown in OpenBSD FAQ 6.14. You may find that example and/or the route testing methods there helpful.

I believe it has something to do with reply-to sent to the wrong interface (?)

I would not know, as I've never used it. It's purpose is to ensure replies are sent to a specific interface, for symmetric routing enforcement, and it can only function on stateful processes.

In parallel, i have a lot of packets losses, for an unknown reason (i log everything for now - debugging purposes)

Your many customized settings are a possible root cause. You have many timeouts set. All of these knobs are very powerful, it is possible you may have shot yourself in the foot with one of them.

[Atlantis]

Hi jggimmi

Thanks for your time !

This is for automatic multipath routing. When set to 1 (equal), routes added with -mpath use equal cost multipath routing, and when set to 0 (not-equal), only the first path added with -mpath will be used. It is unclear how you are adding the route definitions, but if -mpath is not used with route(8), and you are using PF rules only, then the OS will not test the sysctl setting. Equal cost multipath routing is described and an example is shown in OpenBSD FAQ 6.14. You may find that example and/or the route testing methods there helpful.

Actually, i do both: mpath equal routing and PF policy routing (maybe the root cause ??)

Here is re0 hostname conf

inet 192.168.0.253 255.255.255.0 192.168.0.255 description "Connexion Numericable"
dest 192.168.0.1
!route add -mpath default 192.168.0.1

and rl0

inet 192.168.1.253 255.255.255.0 192.168.1.255 description "Connexion Free"
dest 192.168.1.253
!route add -mpath default 192.168.1.254

sysctl.conf contains a line to enable mpath in an equal routing behavior:

net.inet.ip.multipath=1 # 1=Enable IP multipath routing

I have another set of rules without optimizations, etc... and problem remains the same...
I also tried to modify the route priority to lower FREE default route compared to NC one. It works, but still several packet losses, for a reason i miss.

I would be helpful if one among you has already set up such a kind of conf (tried several ones found on the internet, but most use old school ruleset)
A bsd 5.4 ruleset i could adapt to my case would be helpful



Thanks again !

[jggimi]

I've only used multipath routing in a lab setting, and only tested equal cost routing. This was in 2011, for another member here. I tested with 4.9-release, which was after the major syntax change for PF. An example PF is in one of my replies in the thread:

http://daemonforums.org/showthread.php?t=6287

[jggimi]

I should point out that the last post in that thread shows a working reply-to, used as a workaround to a pppoe configuration issue.

[Atlantis]

Thanks so much for this !
I gonna make several tries using your set of rules, just to check
If anyone around here has ideas, i buy it, don't hesitate to post, i will give it a try If i find solutions on my own, gonna post it here for posterity
Thanks all !

[Atlantis]

Hi all, long time and still issues.
Everything works well going outbound, but not the same inbound.
I now use equal mpath, so outgoing traffic is correctly balanced through $nc and alternatively to $free.
BUT issue encountered now is that incoming trafic is only allowed to the first default route listed in the routing table, also used for outgoing trafic meaning that i think that the replys to the incoming requests coming to the second default route are sent to the wrong interface (first default route) and dropped by PF.
My goal is the gateway to be reachable from anywhere to both $free and $nc.
There might be an issue btw pf and routing table, which makes every reply to an incoming connection to be sent to the first default gateway whatever PF reply-to rules are.
I have found something about this kind of issues, and a potential solution, using virtual routing on openBSD. I am not familiar with this new feature. Any help would be welcome, if someone has already implemented such a conf.

Thanks all !

[jggimi]

I am unsure what you have changed, now that you are using mpath. Would you please post your revised configuration, including your pf.conf and your routing table?

With or without new information, I may not be able to provide any further insight, and you should consider posting an informal problem report to the Project's misc@ mailing list.