DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD Security

FreeBSD Security Securing FreeBSD.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 10th March 2011
MatCauthon MatCauthon is offline
New User
 
Join Date: Mar 2011
Posts: 3
Thanked 0 Times in 0 Posts
Default FreeBSD 7, pf, carp, pfsync

G'day Mate! I' ve discovered just today this useful forum: I'm pleased to join you!
My first post is about a weird problem encountered few days ago...

I've two firewalls configured in parallel (connected with a crossover cable) and I use pfsync+carp to failover. So one firewall (A) handles all traffic as MASTER and, if it dies or if some NIC interface go down, the second firewall (B) takes over automatically.

Well... As usually everything works properly, but since a few days ago "B" takes control and "A" become backup. "A" cannot return to be master until rebooting.
After reboot, "A" is the master for a while, then I've the same problem...

I identified a problem here:
Quote:
fwA# sysctl -a | grep arp
net.inet.ip.same_prefix_carp_only: 0
net.inet.carp.allow: 1
net.inet.carp.preempt: 1
net.inet.carp.log: 1
net.inet.carp.arpbalance: 0
net.inet.carp.suppress_preempt: 1
From man carp:
Quote:
net.inet.carp.suppress_preempt:
A read only value showing the status of preemption suppression.
Preemption can be suppressed if link on an interface is down or when pfsync(4) interface is not synchronized.
Value of 0 means that preemption is not suppressed, since no problems are detected. Every problem increments suppression counter.
Well... All my interfaces are UP... now I don't know how to check if pfsync is synched or not...

Meanwhile, in B node:
Quote:
fwB# sysctl -a | grep arp
net.inet.ip.same_prefix_carp_only: 0
net.inet.carp.allow: 1
net.inet.carp.preempt: 1
net.inet.carp.log: 1
net.inet.carp.arpbalance: 0
net.inet.carp.suppress_preempt: 0
I tried with a tcpdump on the interfaces, but I see just the change of condition (master/backup) with the advskew modification...
This is the only strange thing on DMZ interface... :
Quote:
17:01:32.397429 01:80:c2:00:00:01 (oui Unknown) > 01:80:c2:00:00:01 (oui Unknown), ethertype Unknown (0x8808), length 60:
0x0000: 0001 ffff 0000 0000 0000 0000 0000 0000 ................
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
I just tried to change the NIC, but nothing! "A" continue to lose control in 30/45 minutes...


I read somewhere that the result of "pfctl -ss" must give the same result on both nodes:

Quote:
fwA# pfctl -ss | wc -l
5833
fwB# pfctl -ss | wc -l
5507
Could it be important?


Some additional information:
Quote:
fwA# more /etc/rc.conf
ifconfig_em0="inet a.a.a.12 netmask 255.255.255.0 polling" ### DMZ ###
ifconfig_em1="inet b.b.b.2 netmask 255.255.0.0 polling" ### CROSSOVER ###
ifconfig_em2="inet c.c.c.189 netmask 255.255.255.224 polling" ### ISP1 ###
ifconfig_em3="inet d.d.d.249 netmask 255.255.255.0 polling" ### ISP2 ###
defaultrouter="a.a.a.1"

#Firewall
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"

#Failover
pfsync_enable="YES"
pfsync_syncdev="em1"
cloned_interfaces="carp0 carp1 carp2"
ifconfig_carp0="a.a.a.1/24 vhid 1 pass foo"
ifconfig_carp0_alias0="a.a.a.11/24 vhid 1 pass foo"
ifconfig_carp1="d.d.d.14/24 vhid 2 pass bar"
ifconfig_carp1_alias0="d.d.d.2/24 vhid 2 pass bar"
ifconfig_carp2="c.c.c.188/27 vhid 3 pass jack"
ifconfig_carp2_alias0="c.c.c.165/27 vhid 3 pass jack"
Quote:
fwB# more /etc/rc.conf
ifconfig_ste0="inet a.a.a.13 netmask 255.255.255.0 polling"
ifconfig_ste1="inet b.b.b.3 netmask 255.255.0.0 polling"
ifconfig_em0="inet c.c.c.190 netmask 255.255.255.224 polling"
ifconfig_em1="inet d.d.d.250 netmask 255.255.255.0 polling"
defaultrouter="151.12.49.1"

#Firewall
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"

#Failover
pfsync_enable="YES"
pfsync_syncdev="ste1"
cloned_interfaces="carp0 carp1 carp2"
ifconfig_carp0="a.a.a.1/24 vhid 1 advskew 128 pass foo"
ifconfig_carp0_alias0="a.a.a.11/24 vhid 1 advskew 128 pass foo"
ifconfig_carp1="d.d.d.14/24 vhid 2 advskew 64 pass bar"
ifconfig_carp1_alias0="d.d.d.2/24 vhid 2 advskew 64 pass bar"
ifconfig_carp2="c.c.c.188/27 vhid 3 advskew 100 pass jack"
ifconfig_carp2_alias0="c.c.c.165/27 vhid 3 advskew 100 pass jack"
In each node pf.conf I added:
Quote:
fwA# more pf.conf | grep failover
pass quick on { em1 } proto pfsync # failover
pass on { em0 em2 em3 } proto carp # failover

fwB# more pf.conf | grep failover
pass quick on { ste1 } proto pfsync # failover
pass on { em0 ste0 em1 } proto carp # failover
I hope that someone can give me a solution please, or maybe just an idea, cause I'm getting crazy!!!
Please ask me, if you need further information...

Thank you all!
Reply With Quote
  #2   (View Single Post)  
Old 11th March 2011
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,135
Thanked 182 Times in 149 Posts
Default

If one firewall needs to be able to take over from a failing one, it needs the same states.

Have you tried to increase the logging level? From the carp man page:

Code:
net.inet.carp.log	       Value of 0 disables any logging.  Value of 1
			       enables logging of bad carp packets.  Values
			       above 1 enable logging state changes of carp
			       interfaces.  Default value is 1.
From the FreeBSD 7.3 man page of pfsync:

Code:
BUGS
     Possibility to view state changes using tcpdump(1) has not been ported
     from OpenBSD yet.
Unfortunately for you

You could ask on the OpenBSD misc list, but you will be surely told to drop FreeBSD and try the latest and greatest OpenBSD. The OpenBSD pf devs, usually have no idea which pf version FreeBSD 7.x is using.

Have you seen http://www.mail-archive.com/misc@ope.../msg83651.html ?
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #3   (View Single Post)  
Old 14th March 2011
MatCauthon MatCauthon is offline
New User
 
Join Date: Mar 2011
Posts: 3
Thanked 0 Times in 0 Posts
Default

Ehm.. where carps write logs?!

Thank you J65nko, I'll try to ask on various pf lists too...
Reply With Quote
  #4   (View Single Post)  
Old 15th March 2011
MatCauthon MatCauthon is offline
New User
 
Join Date: Mar 2011
Posts: 3
Thanked 0 Times in 0 Posts
Default

Update...

Its a strange day, today...
Yesterday I changed the nodes states, so now the master is the old backup. And it works fine...
The new backup (the nodes that give me problem) seems to be ok... but...

The last rule of my pf ruleset is for load balancing between external connections ...

Quote:
pass out log on $ext_if1 route-to ($ext_if2 $ext_if2_gw) from $ext_if2 to any
pass out log on $ext_if2 route-to ($ext_if1 $ext_if1_gw) from $ext_if1 to any
Now I'm running out of ideas I checked all the pf rules with pfctl -sa and... Arrrggh!...

Quote:
pass out log on em2 route-to (axe0 213.x.x.254) inet from 212.52.82.27 to any flags S/SA keep state
WHAT!?! the hell and who is 212.52.82.27??? It doesn't belong to my IP addresses!?

After a flush it becomes normal:

Quote:
pass out log on em2 route-to (axe0 213.x.x.254) inet from 213.x.x.249 to any flags S/SA keep state
What do you think about that?!
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
CARP Abbass OpenBSD Security 3 13th April 2011 07:22 PM
pfsync and pf.conf tenderoni OpenBSD Security 1 8th October 2010 07:48 PM
Clustering with CARP revzalot OpenBSD General 10 17th September 2009 04:44 AM
pfsync+carp+wifi firewall redundancy inquiry revzalot OpenBSD Security 1 18th May 2009 03:06 PM
carp configuration ohhcarp OpenBSD General 3 16th April 2009 10:50 PM


All times are GMT. The time now is 09:18 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick