DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD Security

OpenBSD Security Functionally paranoid!

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 12th January 2014
Atlantis Atlantis is offline
New User
 
Join Date: Jan 2014
Posts: 4
Default OpenBSD-5.4::MPATH MULTIHOMED PF config issue

greetings All !

I encounter problems while setting up a multihomed FW/GW under OpenBSD 5.4
To make things easier, i enabled multipath into sysctl.conf (tried both equal and unequal).
Setup is as follow


Code:
INTERNET ===|(dsl:82.22X.XX.XX) FREE-DSL-BOX (int:192.168.1.254) |===| (rl0)             (em1) |=== LAN
								|	     OPENBSD BOX       | 
INTERNET ===|(fttb:81.xx.xx.xx) NC-FTTB-BOX (int:192.168.0.1) |======| (re0)             (em0) |=== DMZ
=======

Now the question

Setup is working, but not so good. I would need to have all traffic to pass through NC in priority, and , in case of failure, switch to FREE (NC connection bandwidth is 200MBps, Free is 14MBps).
In parallel, i would need the GW to be reachable from FREE as well as NC networks (both providers boxes do NAT traffic as well...)
I am able to proceed when i disable either re0 or rl0, but when both are enabled, only one is working...
I believe it has something to do with reply-to sent to the wrong interface (?)
In parallel, i have a lot of packets losses, for an unknown reason (i log everything for now - debugging purposes)
Any ideas about how to setup this conf correctly and/or to optimize this setup ?
BTW: i tried to setup -mpath using both equal and unequal weights, without success, pb remains the same...

Thanks for your help !

sincerely
___________________
here are the interfaces and setup description:

===
Code:
# ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33192
        priority: 0
        groups: lo
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr XX:XX:18:XX:XX:7d
        description: Connexion Free
        priority: 0
        groups: egress
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 192.168.1.253 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::2e0:18ff:XXXX:XXXX%rl0 prefixlen 64 scopeid 0x1
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr XX:XX:6e:XX:XX:XX
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex,master)
        status: active
        inet 10.0.0.254 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::XXX:6eff:XXXX:XXXX%em0 prefixlen 64 scopeid 0x2
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:30:6e:XX:XX:XX
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet 172.16.1.254 netmask 0xffffff00 broadcast 172.16.1.255
        inet6 fe80::230:XXXX:XXXX:8e81%em1 prefixlen 64 scopeid 0x3
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:1e:XX:XX:XX:XX
        description: Connexion Numericable
        priority: 0
        groups: egress
        media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
        status: active
        inet 192.168.0.253 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 feXX::21e:XXXX:feXX:83XX%re0 prefixlen 64 scopeid 0x4
enc0: flags=0<>
        priority: 0
        groups: enc
        status: active
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33192
        priority: 0
        groups: pflog
=======



Routing table is:
====================

Code:
# route show
Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            192.168.0.1        UGSP       4   345002     -     8 re0
default            192.168.1.254      UGSP       4   444011     -     8 rl0
10.0.0/24          link#2             UC         3        0     -     4 em0
10.0.0.150         00:0c:XX:22:XX:f5  UHLc       1    40586     -     4 em0
loopback           localhost          UGRS       0        0 33192     8 lo0
localhost          localhost          UH         2     2154 33192     4 lo0
172.16.1/24        link#3             UC         3        0     -     4 em1
192.168.1.253      00:XX:XX:2a:XX:7d  UHLc       0      212     -     4 lo0
192.168.0/24       link#4             UC         2        0     -     4 re0
192.168.0.1        24:ec:XX:05:XX:4X  UHLc       1        5     -     4 re0
base-address.mcast localhost          URS        0        0 33192     8 lo0
===================

DMZ contains a mail server, which grabs mails from public services (yahoo,...) using pop3s, ntp server, and dns server.
Gateway itself manages with multidomain smtp routing, and incoming mails are forwarded to internal mail server, as well as DNS
===================

Here is the PF.conf setup

================

Code:
############################ INTERFACES INTERNES #################################

orange_if  = "em0"
orange_net = "10.0.0.0/24"

green_if  = "em1"
green_net = "172.16.1.0/24"

############################ INTERFACES EXTERNES #################################

nc  = "re0"
free = "rl0"

############################ PASSERELLES EXTERNES #################################

nc_gw = "192.168.0.1"
free_gw = "192.168.1.254"


############################ TABLES ##################################

table <ournets> persist { 10.0.0.0/24, 172.16.1.0/24 }
table <bruteforce> persist
table <ossec_fwtable> persist # ossec_fwtable
table <allowed_out> persist { }
table <firewall> const { self }

############################ PARAMETRES ##############################

set state-policy floating
set block-policy drop
set optimization normal
#set require-order yes
set timeout { interval 10, frag 30 }
set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
set timeout { tcp.closing 900, tcp.finwait 45, tcp.closed 90 }
set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
set timeout { icmp.first 20, icmp.error 10 }
set timeout { other.first 60, other.single 30, other.multiple 60 }
set timeout { adaptive.start 0, adaptive.end 0 }
set limit { states 5000, frags 2000 }
set limit table-entries 500000
set fingerprints "/etc/pf.os"
set skip on lo0

########################## fragment reassemble  ############################
# Normalisation de tous les paquets entrants.
match in all scrub (no-df)
#scrub out all fragment reassemble max-mss 1400

###################################### N  A  T  ##########################################

#  nat outgoing connections on each internet interface
match out on egress inet from !(egress:network) to any nat-to { egress }

###################################### REGLES DENY  #########################################

# block unwanted hosts
block in quick from <bruteforce>
block in quick from <ossec_fwtable>

# block anything by default
block in log
block out log

# ORANGE TO GREEN NOK
block drop in on $green_if from $orange_net to $green_net

###################################### REGLES ALLOW #########################################

# Allow ICMP on external interfaces

pass in quick on $green_if proto icmp from {$green_net, $orange_net} to any nat-to egress keep state


#pass in inet proto tcp from {$green_net, $orange_net} to any port { http, https } divert-to 127.0.0.1 port 3128

#  pass all outgoing packets on internal interface

pass out quick log on $green_if to $green_net
pass out quick log on $orange_if to $orange_net

################################################################################

# Autoriser le trafic sortant et entrant sur le r?seau local.
# ces r?gles cr?eront des entr?es au niveau de la table d'?tat ?tant
# donn? que le mot-cl? "keep state" est automatiquement appliqu?.

# on autorise le LAN et certains services a atteindre le firewall
pass in quick log on $green_if proto tcp from $green_net to ($green_if) port { ssh, 3128 } keep state
pass in quick log on $green_if proto udp from $green_net to ($green_if) port 53

# on autorise certains services en zone DMZ a etre atteints par le LAN

pass in quick log on $green_if proto tcp from $green_net to $orange_net port { smtp, http, https, ssh, 137, 139, 445, 993 } keep state
pass in quick log on $orange_if proto udp from $orange_net to !$green_net port { domain }
pass in quick log on $orange_if proto tcp from $orange_net to !$green_net port {25, 110, 993, 995 }

# on autorise la sortie sur internet

# Allow ICMP on external interfaces
pass in quick on $free proto icmp from <firewall> to any keep state
pass in quick on $nc proto icmp from <firewall> to any keep state


pass in quick log on $green_if proto tcp from $green_net to !$orange_net port { http, https }


pass out log on egress proto udp from { <firewall>, $green_net, $orange_net } to any port { domain, ntp } keep state (if-bound)
pass out log on egress proto tcp from { <firewall>, $green_net } to any port { domain, http, https, ntp } keep state (if-bound)
pass out log on egress proto tcp from $orange_net to any port { domain, smtp, ntp } keep state (if-bound)
pass out log on egress inet proto icmp all icmp-type { echoreq }

########################### REGLES EN ENTREE ##############################
######################## SERVICES EN ENTREE #########################

###################### E/SMTP/IMAPS MOBILE VIA FREE ######################
pass in quick log on $free inet proto tcp from any to $free port = 25 reply-to ($free $free_gw) flags S/SA keep state
#pass in quick log on $free inet proto tcp from any to $free port = 9930 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993

pass in quick log on $free inet proto tcp from XX.84.144.0/22 to $free port = 9930 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $free inet proto tcp from XX.84.144.0/22 to $free port = 9931 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25

pass in quick log on $free inet proto tcp from XX.10.159.0/24 to $free port = 9930 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $free inet proto tcp from XX.10.159.0/24 to $free port = 9931 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25

pass in quick log on $free inet proto tcp from XX.160.0.0/12  to $free port = 9930 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $free inet proto tcp from  XX.8.160.0/12 to $free port = 9931 reply-to ($free $free_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25
pass in quick log on $free inet proto tcp from any to $free port = 25 reply-to ($free $free_gw) flags S/SA keep state

################### E/SMTP/IMAPS MOBILE VIA NUMERICABLE ##################
#pass in quick log on $nc inet proto tcp from any to $nc port = 9930 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993

pass in quick log on $nc inet proto tcp from XX.84.144.0/22 to $nc port = 9930 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $nc inet proto tcp from XX.84.144.0/22 to $nc port = 9931 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25

pass in quick log on $nc inet proto tcp from XX.10.159.0/24 to $nc port = 9930 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $nc inet proto tcp from XX.10.159.0/24 to $nc port = 9931 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25

pass in quick log on $nc inet proto tcp from XX.160.0.0/12  to $nc port = 9930 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 993
pass in quick log on $nc inet proto tcp from XX.160.0.0/12  to $nc port = 9931 reply-to ($nc $nc_gw) flags S/SA keep state (if-bound) rdr-to 10.0.0.150 port 25
===================

Last edited by Atlantis; 13th January 2014 at 05:21 PM.
Reply With Quote
  #2   (View Single Post)  
Old 12th January 2014
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,975
Default

Hello, and welcome!
Quote:
Originally Posted by Atlantis View Post
To make things easier, i enabled multipath into sysctl.conf (tried both equal and unequal).
This is for automatic multipath routing. When set to 1 (equal), routes added with -mpath use equal cost multipath routing, and when set to 0 (not-equal), only the first path added with -mpath will be used. It is unclear how you are adding the route definitions, but if -mpath is not used with route(8), and you are using PF rules only, then the OS will not test the sysctl setting. Equal cost multipath routing is described and an example is shown in OpenBSD FAQ 6.14. You may find that example and/or the route testing methods there helpful.
Quote:
I believe it has something to do with reply-to sent to the wrong interface (?)
I would not know, as I've never used it. It's purpose is to ensure replies are sent to a specific interface, for symmetric routing enforcement, and it can only function on stateful processes.
Quote:
In parallel, i have a lot of packets losses, for an unknown reason (i log everything for now - debugging purposes)
Your many customized settings are a possible root cause. You have many timeouts set. All of these knobs are very powerful, it is possible you may have shot yourself in the foot with one of them.
Reply With Quote
  #3   (View Single Post)  
Old 12th January 2014
Atlantis Atlantis is offline
New User
 
Join Date: Jan 2014
Posts: 4
Default

Hi jggimmi

Thanks for your time !

Quote:
This is for automatic multipath routing. When set to 1 (equal), routes added with -mpath use equal cost multipath routing, and when set to 0 (not-equal), only the first path added with -mpath will be used. It is unclear how you are adding the route definitions, but if -mpath is not used with route(8), and you are using PF rules only, then the OS will not test the sysctl setting. Equal cost multipath routing is described and an example is shown in OpenBSD FAQ 6.14. You may find that example and/or the route testing methods there helpful.
Actually, i do both: mpath equal routing and PF policy routing (maybe the root cause ??)

Here is re0 hostname conf

Quote:
inet 192.168.0.253 255.255.255.0 192.168.0.255 description "Connexion Numericable"
dest 192.168.0.1
!route add -mpath default 192.168.0.1
and rl0

Quote:
inet 192.168.1.253 255.255.255.0 192.168.1.255 description "Connexion Free"
dest 192.168.1.253
!route add -mpath default 192.168.1.254
sysctl.conf contains a line to enable mpath in an equal routing behavior:

Quote:
net.inet.ip.multipath=1 # 1=Enable IP multipath routing
I have another set of rules without optimizations, etc... and problem remains the same...
I also tried to modify the route priority to lower FREE default route compared to NC one. It works, but still several packet losses, for a reason i miss.

I would be helpful if one among you has already set up such a kind of conf (tried several ones found on the internet, but most use old school ruleset)
A bsd 5.4 ruleset i could adapt to my case would be helpful



Thanks again !

Last edited by Atlantis; 12th January 2014 at 03:23 PM.
Reply With Quote
  #4   (View Single Post)  
Old 12th January 2014
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,975
Default

I've only used multipath routing in a lab setting, and only tested equal cost routing. This was in 2011, for another member here. I tested with 4.9-release, which was after the major syntax change for PF. An example PF is in one of my replies in the thread:

http://daemonforums.org/showthread.php?t=6287
Reply With Quote
  #5   (View Single Post)  
Old 12th January 2014
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,975
Default

I should point out that the last post in that thread shows a working reply-to, used as a workaround to a pppoe configuration issue.
Reply With Quote
  #6   (View Single Post)  
Old 12th January 2014
Atlantis Atlantis is offline
New User
 
Join Date: Jan 2014
Posts: 4
Default

Thanks so much for this !
I gonna make several tries using your set of rules, just to check
If anyone around here has ideas, i buy it, don't hesitate to post, i will give it a try If i find solutions on my own, gonna post it here for posterity
Thanks all !
Reply With Quote
  #7   (View Single Post)  
Old 15th February 2014
Atlantis Atlantis is offline
New User
 
Join Date: Jan 2014
Posts: 4
Default

Hi all, long time and still issues.
Everything works well going outbound, but not the same inbound.
I now use equal mpath, so outgoing traffic is correctly balanced through $nc and alternatively to $free.
BUT issue encountered now is that incoming trafic is only allowed to the first default route listed in the routing table, also used for outgoing trafic meaning that i think that the replys to the incoming requests coming to the second default route are sent to the wrong interface (first default route) and dropped by PF.
My goal is the gateway to be reachable from anywhere to both $free and $nc.
There might be an issue btw pf and routing table, which makes every reply to an incoming connection to be sent to the first default gateway whatever PF reply-to rules are.
I have found something about this kind of issues, and a potential solution, using virtual routing on openBSD. I am not familiar with this new feature. Any help would be welcome, if someone has already implemented such a conf.

Thanks all !
Reply With Quote
  #8   (View Single Post)  
Old 15th February 2014
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,975
Default

I am unsure what you have changed, now that you are using mpath. Would you please post your revised configuration, including your pf.conf and your routing table?

With or without new information, I may not be able to provide any further insight, and you should consider posting an informal problem report to the Project's misc@ mailing list.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
OpenBSD 5.3 X server config danielcrvg OpenBSD General 18 20th June 2013 01:03 AM
OpenBSD 4.7 issue? rpindy OpenBSD Installation and Upgrading 18 21st May 2010 12:33 AM
Downloading free OpenBSD issue karolina OpenBSD General 1 19th June 2009 10:36 AM
How to Fix Security Issue In OpenBSD 4.1 Stable ? openbsdspirit OpenBSD General 4 21st June 2008 11:33 AM
openVPN 2.1_rc7 (server) on openBSD 4.3 config examples s2scott Guides 2 23rd May 2008 06:16 PM


All times are GMT. The time now is 07:27 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick