Problem with just one website !?

[Redrobes]

Hi guys. In months past I have been using a small little consumer boxed router but decided a few months back to get something much more secure so went with OpenBSD 4.6 so got a new machine installed, configured etc

Everything has been just great and I am really lovin it. I have PF and Squid running and I can browse and log everything with squids ACL's etc.

Just one small problem that I am hoping that someone can help me with. As one of the leaders on a forum at "forum.cartographersguild.com" just this one web site gives me grief. It often hangs getting the pages in or takes ages getting most of the pages in.

I have unplugged the BSD firewall and put my old box back on and its ok again. Something about OpenBSD and my config for it is not compatible with it. The other guys on that forum are not having issues with it. If I use a web proxy then its ok through the OpenBSD firewall too.

My pf.conf has scrub going in both directions from the ext_if and I have tried the min-ttl and max-mss with varying values but I see no difference. I am not blocking it with squid and using lynx can hang it too.

I have not gone into actual packet content inspection. I think there must be something peculiar about OpenBSD packets that its web host is balking at. I don't have any other friends with OpenBSD other than one which has exactly the same setup machine and install as I and also has the same issues so its not my ISP either.

I can ping it with constant times no problem. They are hosted on secureserver.com if that makes any difference.

If I use Firefox then it hangs or takes ages, with Opera then it shows the count of parts of page to go and it usually gets so far in and then stops. Usually after about a minute the page completes so I think my requests are exiting but maybe caught up somewhere.

Not found a single other web site with this issue.

Thats about everything I know on the problem. What can I try to fix it ?

Thx in advance.

[J65nko]

It could be a "TCP window scaling" issue. See http://www.daemonforums.org/showthread.php?t=2387

[Redrobes]

Thanks, I havent heard of that before.

I ran pfctl -vss and when it hangs it comes up with a SYN:Closed and it is showing the wscale in the range 0 to 2 range.

So I have changed this net.inet.tcp.rfc1323: 1 -> 0 and retried the web site and so far its looking much better. I think you might have hit the solution in one go !

I think I will have to read around this topic more. Maybe OpenBSD should ship with the option at 0 if lots more Linux boxes are using scaling by default now.

So is what this rfc1323=0 is doing is disabling the ability for allowing the packets to use the scaling option and so when the external site responds with all the scaling not set it works ok. I guess my little router box would not have coped and so set it to zero anyway.

I'll try this for a while but have a big thanks

[J65nko]

If only have this problem with your OpenBSD box. And if setting the tfc1323 sysctl to 0 solves it, there is only one conclusion: your pf.conf rules for port 80 don't create state on the first packet of the TCP connection.

[Redrobes]

Yeah, one good thrashing later and its just great now.

[Redrobes]

Originally Posted by J65nko

If only have this problem with your OpenBSD box. And if setting the tfc1323 sysctl to 0 solves it, there is only one conclusion: your pf.conf rules for port 80 don't create state on the first packet of the TCP connection.

Yes that is the case. So do you think that they should and that if I did then the rfc1323 can be turned back on. Almost all of the examples I see dont set the SYN flag in the filter. So do I need a pf rule which is for the $ext_if out setting the state ? I just have "pass out". Should that be pass out flags S/SA or something like that ?

You can tell that I am new to PF.

[J65nko]

Since several years the flags S/SA has become the default.

A simple sample ruleset to protect a OpenBSD host from it's Windows peers in the LAN:

EXT=re0

TCPservices = "{ www https domain }"
UDPservices = "{ domain }"

set skip on lo0

block log all

pass out quick on $EXT inet proto tcp from $EXT to any port $TCPservices
pass out quick on $EXT inet proto udp from $EXT to any port $UDPservices

As you can see neither a keep state nor a flags S/SA used.
A test load of the rules shows:

# pfctl -vvnf block-all.pf  
EXT = "re0"
TCPservices = "{ www https domain }"
UDPservices = "{ domain }"
set skip on { lo0 }
@0 block drop log all
@1 pass out quick on re0 inet proto tcp from 192.168.222.20 to any port = www flags S/SA keep state
@2 pass out quick on re0 inet proto tcp from 192.168.222.20 to any port = https flags S/SA keep state
@3 pass out quick on re0 inet proto tcp from 192.168.222.20 to any port = domain flags S/SA keep state
@4 pass out quick on re0 inet proto udp from 192.168.222.20 to any port = domain keep state

As you can see pf has automatically added all the necessary keep state stuff.

[Redrobes]

Actually the more I read about this the more confused I am getting. I am using either XPx64 or latest Ubuntu to look at this website from my internal NAT side. Maybe both of these are using it ? So its not Vista initiating this TCP packet. Once the packet leaves the XP machine then its requested from my Squid proxy and therefore started from the in side of the firewall to the out. So the filter which needs to have the S/SA bit set is going out not initiated from external coming in and once the connection is set up then PF holds it in the state table. At least thats what I thought. I am using 4.6 so that I thought the flags S/SA bit and the keep state were on by default. Gah !

[Redrobes]

I seem to post out of sync here... sorry.

Ok so we dont need to add the flags S/SA bit which is what I thought was the case. Why then is disabling OpenBSD support for rfc1323 making it work. It seems from what your saying that PF should be able to handle the state because it filters on the first SYN transaction. Is it the case that most web sites do handle it ok and that this particular one does not.

[Redrobes]

My rules:

[J65nko]

Could you please paste in your ruleset in text format so I can edit it an re-post with some improvements.?

[J65nko]

You can use this as a skeleton

EXT=re0
INT=bge0

TCPservices = "{ www https ssh }"
UDPservices = "{ domain ntp }"

set skip on lo0

# --- NAT/RDR section
nat on $EXT from ! $EXT to any -> ($EXT)

# --- Default policy
block log all

# -- EXTERNAL interface out

pass out quick on $EXT inet proto tcp tagged OUT_ok

# -- INTERNAL interface in

pass in quick on $INT inet proto tcp from $INT:network to any port $TCPservices tag OUT_ok
pass in quick on $INT inet proto tcp from $INT:network to any port $UDPservices tag OUT_ok

A test load

EXT = "re0"
INT = "bge0"
TCPservices = "{ www https ssh }"
UDPservices = "{ domain ntp }"
set skip on { lo0 }
@0 nat on re0 inet from ! 192.168.222.20 to any -> (re0) round-robin
@0 block drop log all
@1 pass out quick on re0 inet proto tcp all flags S/SA keep state tagged OUT_ok
@2 pass in quick on bge0 inet proto tcp from 10.0.0.0/8 to any port = www flags S/SA keep state tag OUT_ok
@3 pass in quick on bge0 inet proto tcp from 10.0.0.0/8 to any port = https flags S/SA keep state tag OUT_ok
@4 pass in quick on bge0 inet proto tcp from 10.0.0.0/8 to any port = ssh flags S/SA keep state tag OUT_ok
@5 pass in quick on bge0 inet proto tcp from 10.0.0.0/8 to any port = domain flags S/SA keep state tag OUT_ok
@6 pass in quick on bge0 inet proto tcp from 10.0.0.0/8 to any port = ntp flags S/SA keep state tag OUT_ok

Are you using PPPoE ?

[Redrobes]

Sure

ext_if     = "rl1"
int_if     = "rl0"

services   = "{ 22, 80, 443 }"
icmp_types = "{ echoreq, unreach }"

set block-policy drop
set loginterface $ext_if

table <blocked> persist                # Permanent table of blocked hosts

set skip on lo                         # Allow all loopback

match in on $ext_if all scrub (no-df)
match out on $ext_if all scrub (no-df random-id)

nat on $ext_if from !($ext_if) -> ($ext_if:0)      # NAT

block in log quick from <blocked>      # Block and forget banned
block in log                           # Default: block incoming traffic
block return in log on $ext_if inet proto tcp from any to any port 113

pass out log                           # Allow all outgoing traffic

antispoof quick for { lo, $ext_if, $int_if }

pass in on $ext_if inet proto tcp from any to any port $services

# Turn on when we want externals to PING
# pass in on $ext_if inet proto icmp to ($ext_if) icmp-type $icmp_types

pass in on $int_if                     # Local network traffic OK

# Allow SSH on the external interface, but add the client to blocked
# table if it tries to connect more than X times in Y seconds
pass in on $ext_if inet proto tcp to ($ext_if) port ssh keep state \
      (max-src-conn-rate 2/150, overload <blocked> flush globa

[J65nko]

I like to keep things simple, so I got rid of all those match and antispoof stuff. You can add them in later if you want.

BTW I wonder why you did not allow out any port domain/DNS (53) out

EXT     = "rl1"
INT     = "rl0"

# -- for j65nko only
EXT = re0
INT = bge0

TCPservices   = "{ ssh www https }"
UDPservices   = "{ domain ntp }"
icmp_types = "{ echoreq, unreach }"

set block-policy drop
set loginterface $EXT

table <blocked> persist                # Permanent table of blocked hosts

set skip on lo                         # Allow all loopback

# match in on $EXT all scrub (no-df)
# match out on $EXT all scrub (no-df random-id)

# --- NAT/RDR section

nat on $EXT from !($EXT) -> ($EXT)      # NAT


# --- DEFAULT policy
block log all
block in log quick from <blocked>      # Block and forget banned

#antispoof quick for { lo, $EXT, $int_if }


# ---- OUTGOING TRAFFIC

# -- external interface
pass out quick on $EXT tagged OUT_ok

# --- INCOMING TRAFFIC

# - internal interface
pass in quick on $INT inet proto tcp from $INT:network to any port $TCPservices tag OUT_ok
pass in quick on $INT inet proto tcp from $INT:network to any port $UDPservices tag OUT_ok

# - external interface
# Allow SSH on the external interface, but add the client to blocked
# table if it tries to connect more than X times in Y seconds
pass in quick  on $EXT inet proto tcp to ($EXT) port ssh  \
      (max-src-conn-rate 2/150, overload <blocked> flush global)

# Turn on when we want externals to PING
# pass in quick on $EXT inet proto icmp to ($EXT) icmp-type $icmp_types

[Redrobes]

I'm not using PPPoE.

Thanks for suggesting some rules. Its interesting to see what you have here.

You say I am not allowing any out on 53 but I thought I was allowing all stuff out of the ext interface. I need to serve DNS requests from the inside to external but I am not running a DNS server visible to the outside. I can do DNS to a DNS server like OpenDNS. Again with the ntp. I think I might want to use an ntp server but not be an ntp server. Because UDP is connectionless then maybe I need to open that for UDP ntp port.

With the tag OUT_ok you seem to be suggesting that theres some optimization in setting those packets with a tag and passing them quick with the tag set. I thought that with PF being a stateful filter that as soon as you have a connection made and its in the state block that PF would automatically use those rules for that connection until it was closed. Again not sure what happens with UDP tho. So is this what you had in mind ? Why do you think its better with the tag OUT_ok ?

So with this rfc thing. In post #4 you said that the only reason it could be happening on my system is that I must not have the keep state bit set for the first packet but we know that its applying the flags S/SA and keep state. So I cant figure this out. This window scaling seems to be saying that on the first SYN request it passes 3 bytes saying what kind of scaling we allow and the initiator seems to set the scaling. Then the server is supposed to ack that syn presumably setting what scaling it can accept and from then on your alright and use whatever you both agree on. So my guess is that by turning it off we say no we don't do scaling and the other side says fine by me and its alright. If we say yes we do scaling and they ack with yes so do we but thats from some kind of transparent proxy handling the Syn/Syn Ack and in reality a machine behind it does not then those packets are going to be forwarded to a machine that chokes on them for being the wrong size. Thats the best guess as to what I think is going on in this case. The forum message that your linking to in the other page seems more like the other way around where a new Vista machine says it can do scaling but the firewall cant if running old PF where you need to specify that the its the SYN packet which will drive the state. Presumably that has to be done on the internal interface tho. The PF advice is to do that on all interfaces so thats why its defaulted now. I still dont really get it tho. Just seems to me that with v4.6 it should have worked ok with the scaling extension switched on.

Another thing I am unsure about. You have pass in quick on $INT inet... but there does not seem to be any pass in $EXT except for the SSH. How does any traffic get to the web server ? Or how does non port 80/443 traffic get out ? Say mail or ftp etc ?

It looks to me like your general strategy is to pass selectively on the internal in interface and rely on the stateful / tag mechanism to hold the line open were as mine does it on the external in interface allowing all internal traffic out which opens up the state for the line. Is this a correct assessment ?

[J65nko]

Yes, your assessment is correct.

I stick a label or a tag on a packet, when it enters the firewall box. For traffic initiated by your local LAN, this means the internal interface.

A rule like pass in quick on $INT inet proto tcp from $INT:network to any port $TCPservices tag OUT_ok creates a state. All traffic belonging to this state are accepted, thus also the reply packets who first enter the firewall on the external interface.
This accepting in of reply packets is taken care of by the pass out quick on $EXT tagged OUT_ok rule, which first accepts all packets with the OUT_ok tag, sends them out and because it also creates state allows them in again.

This allows you to create very simple and logical rulesets.

Tags have been in pf since 2003 IIRC but since everybody seems to copy old rulesets covered with cobwebs from the dark and dusty attics from the Internet, hardly anybody uses them. I have been advocating tags on the predecessor of these forums for about 7 years. Especially if you have to redirect traffic, using tags makes it very easy to follow.

I also group the rules by direction and interface. This makes the rules easy to understand .In most of my rule sets the pf rule optimizer has hardly anything to do

Now the question is. does it help in your case? I have no problems with visiting that forum.cartographersguild.com website and I am behind an OBSD firewall too.

[Redrobes]

Ok cool I think I am getting it - at least the general principles.

Originally Posted by J65nko

Now the question is. does it help in your case? I have no problems with visiting that forum.cartographersguild.com website and I am behind an OBSD firewall too.

Killer question. If you have no problem then do you have the rfc1323 extension switched on or off ?

[J65nko]

I never touched it, it is the default,

# sysctl net.inet.tcp.rfc1323
net.inet.tcp.rfc1323=1

http://en.wikipedia.org/wiki/Transmi...Window_scaling

[Redrobes]

Well ill leave it there. Thanks for all the help - ill do more reading and asking about why this is happening when we think it has the state on the first packet. Just marvellous to be able to browse there again and not to have to change over routers !