pf: why is that rule not working?

[ivanatora]

Hello,
The situation is simple: two machines are behind NAT and I'm operating on the NAT box. The NAT is set up correctly - both of the machines are connected to the Internet. I have a few IPs from Internet that are put into a table <data>.
I'm trying to learn PF, but something is not going well. I have a rule that doesn't match. In order to debug things, I've set up a logging on that rule and it really doesn't match at all. Could you explain me why?
Forget about the (probably messed up) ALTQ, now everything I want is to understand why the last rule doesn't match.

### Macros
int_if = "re0"
ext_if = "rl0"
ext_ip = "192.168.1.2"

### Tables
table <network>  { 192.168.0.34, 192.168.0.223 }
table <data> persist file "/root/ip-store.data"

### Normalizations
scrub in all

### Queueing

altq on $int_if hfsc bandwidth 10Mb queue {general, data}
queue general bandwidth 4Mb hfsc (realtime 4Mb upperlimit 4Mb default)   
queue data bandwidth 1Mb hfsc (realtime 128Kb upperlimit 256Kb)

### Translation
nat pass on $ext_if from <network> to any -> $ext_ip

### Filtering

#pass log (all to pflog0) on $ext_if proto icmp # this is working on pflog0 or pflog1, so probability of not working logging devices is zero
pass out log (all to pflog1) on $int_if proto tcp from <data> to <network> #this is not working - nothing is logged to pflog1

First I made myself sure there are some ips into the <data> table - they were there. Then I tried to replace "from <data>" with "from any" - I hought there is no traffic from these hosts, and I tried expanding the rule to apply to the whole traffic. Nothing was logged again.
As you have seen I'm trying to do some ALTQ on the internal interface (for incomming traffic I thing this is the right interface?), and that's why I need that rule to get working. I assume something is totaly wrong in my setup or in my understandings, isn't it?

************************************************** *******************
Things are getting even more confusing!
I changed

pass out log (all to pflog1) on $int_if proto tcp from <data> to <network>

to

pass in log (all to pflog1) on $int_if proto tcp from <network> to <data>

and pflog1 began to log!
Despite the "from <network> to any" I see in tcpdump packets flying in both directions, like:

19:51:36.024411 IP 195.149.248.137.80 > 192.168.0.34.46276:  tcp 1472 [bad hdr length 8 - too short, < 20]
19:51:36.024738 IP 192.168.0.34.46276 > 195.149.248.137.80:  tcp 12 [bad hdr length 8 - too short, < 20]

I thought only the second packet should show up becouse of the "one way" matching only?
And why the opposite direction rule again doesn't match?

pass out log (all to pflog0) on $int_if proto tcp from <data> to <network>

************************************************** *******************
I'd say there is something interesting even more.
I see packets on pflog1, but according to pfctl -s rules, there shouldn't be any packets at all:

# pfctl -v -s rules
scrub in all fragment reassemble
  [ Evaluations: 39611     Packets: 19895     Bytes: 7958775     States: 0     ]
  [ Inserted: uid 0 pid 3338 ]
pass out quick on re0 from any to <network> flags S/SA keep state label "incomming"
  [ Evaluations: 5050      Packets: 8         Bytes: 1747        States: 8     ]
  [ Inserted: uid 0 pid 3338 ]
pass in log (all, to pflog1) on re0 proto tcp from <network> to <data> flags S/SA keep state label "??? in"
  [ Evaluations: 4688      Packets: 0         Bytes: 0           States: 0     ]                                                     <--- packets 0 !
  [ Inserted: uid 0 pid 3338 ]
pass in log (all) on rl0 proto tcp from <data> to <network> flags S/SA keep state label "??? out"
  [ Evaluations: 3186      Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 3338 ]

[jggimi]

The reason you're having trouble is you are confused about NAT. As you probably know already, NAT allows your entire private network to share a single IP address. So, your first failing rule:

pass out log (all to pflog1) on $int_if proto tcp from <data> to <network>

fails because there are no packets from <data> to <network>. There never will be. Think about it. Your <data> users do not route packets to IP addresses on <network>. Not directly. They can't. 192.168/16 and the other RFC1918 addresses are not routable on the internet anyway, since they are designed for private networks only.

With NAT, state table tracking is used to keep sessions open when they are originated from within your local network, destined outward. You don't show any rules for that, but if you had them, that is how outbound-originating traffic is managed. Since pass all is the default (well, it is in the OpenBSD implementation of PF, it probably is in FreeBSD) then these are handled by that default rule. "keep state" has been the default for some time, and I assume that's the case with FreeBSD's PF implementation, as you have mentioned success with outbound traffic.

For inbound traffic that needs to reach a server on the internal network, and if a state does not already exist, your NAT router must know about the service and the server. This is commonly done via port forwarding (the "rdr" rules).

[jggimi]

Let me be clear about packets that do originate from <data> IP addresses. The destination IP address will be the IP address of your external interface. That is why you never see packets being matched for that failing rule.

[J65nko]

If you wonder why something is not being logged, just run tcpdump on the interface the traffic is going through.

If I have to debug a firewall with an external and internal NIC, I usually ssh in to the box from my X workstation. In each ssh session I run tcpdump.

1. On the external interface

   tcpdump -ni re0

2. On the internal interface

   # tcpdump -ni bge0 not port ssh

   Note the 'not port ssh' filter, to prevent pollution of the tcpdump output with my own ssh traffic.
3. On the pflog device

   # tcpdump -eni pflog0

[s2scott]

### Translation
nat pass on $ext_if from <network> to any -> $ext_ip

Per the longer explanation already offered, NAT has precedence (occurs first), and, with the pass modifier in effect, no subsequent pass/block RULE is ever hit and, consequently, no pass/block logging is being triggered.

/S

[s2scott]

You're looking for something along these lines...

#STMT 1
nat on $ext_if inet tagged MYIN2OUT -> ($ext_if:0)
#
# STMT 2
pass in log quick on $int_if inet \
 from ($int_if:network) to any \
 tag MYIN2OUT \
 keep state
#
# STMT 3
pass out log quick on $ext_if inet \
 tagged MYIN2OUT \
 keep state
#

In the order of precedence, your traffic originating on your $int_if:network and destined for the internet is FIRST inspected and passed by SMTM#2, then NAT translation occurs by STMT#1, and is then inspected and passed by STMT#3.

The use of tag/tagged means you -- the pf.conf author -- don't need to know the real IP source/dest translations at the point of each statement to write a clean (and leak-proof) pf.conf.


/S

[ivanatora]

J65nko, thanks for your suggestion, but I already have said that I'm using tcpdump and I see there is traffic going trough.

s2scott, this is an interesting way to do NAT in PF. It is not shown into the tutorials, but it works I don't understand what is the SMT3 rule for? It doesn't get matched at all.
And I don't think I can do more evaluations on these packets, because of the 'quick' keyword. AFAIK, when a packet is matched in that rule it doesn't check the next rules at all.

Question: I've accidently added a rule for queuing, and it was working - the download traffic got shaped (did not expect that). Why is it working in the "from <network> to <data>" rule? Isn't that the upload matching traffic rule? To remind - <data> addresses are public servers from Internet.

This is all of my pf.conf:

### Macros
int_if = "re0"
ext_if = "rl0"
ext_ip = "192.168.1.2"

### Tables
table <network>  { 192.168.0.34, 192.168.0.223 }
table <data> persist file "/root/ip-store.data"

### Normalizations
scrub in all

### Queueing
altq on $int_if hfsc bandwidth 10Mb queue {general, data}
queue general bandwidth 4Mb hfsc (realtime 4Mb upperlimit 4Mb default)   
queue data bandwidth 1Mb hfsc (realtime 128Kb upperlimit 256Kb)

### Translation
nat pass on $ext_if from <network> to any -> $ext_ip

### Filtering
pass in log (all to pflog1) on $int_if proto tcp from <network> to <data> label "??? in" queue data
pass out log (all to pflog0) on $ext_if proto tcp from <data> to <network> label "??? out"

pflog0 still logs nothing, but you have said it is due to 192.168/16 private class network, which is not seen from the Internet. (Btw, I thought PF have states for these connections and it would recognize traffic for the hosts behind NAT)

[J65nko]

How about adding a log directive to your NAT rule?

[s2scott]

We've tried to explain that there's a big difference between

nat ...

-and-

nat pass ...

nat pass on $ext_if from <network> to any -> $ext_ip

...drop the "pass" and watch what happens.

/S

[ivanatora]

Originally Posted by s2scott

...drop the "pass" and watch what happens.
/S

Oh, didn't saw that I will try that later and repost here.
************************************************** ********************
So, you are proposing this way of alternative doing the NAT (I've applied some queues for testing):

### Translation
nat on $ext_if tagged DO_NAT -> $ext_ip

### Filtering
pass in quick on $int_if from <network> to <data> tag DO_NAT keep state queue data label "do_nat_data"
pass in quick on $int_if from <network> to any tag DO_NAT keep state label "do_nat"
pass out quick on $ext_if tagged DO_NAT keep state label "revert_nat?"

This works - I'm seeing packets in these labels:

# pfctl -s label
do_nat_data 450 382 344201 148 6602 234 337599
do_nat 112 226 38383 131 9522 95 28861
revert_nat? 412 608 382584 329 366460 279 16124

I'm still wandering why queueing works for downloads (direction: <data> -> $ext_if -> $int_if -> <network>) while the rule is read in the opposite direction (from <network> to <data>). Can you put some light on that? What rule should I use if I want to limit upload speed?

[s2scott]

Originally Posted by ivanatora

I'm still wandering why (sic) queuing works for downloads (direction: <data> -> $ext_if -> $int_if -> <network>) while the rule is read in the opposite direction (from <network> to <data>). Can you put some light on that? What rule should I use if I want to limit upload speed?

Because...

pass in quick on $int_if \
 from <network> to <data> tag DO_NAT \
 keep state queue \
 data label "do_nat_data"

"keep state" sets up the return path INCLUDING, in your case, return path through your "queue." Your outbound packets establish state entry and the matching reply packets come through the state table entry, with your queue attached, and not the rules table.

/S

[ivanatora]

I'm here again with a bit different approach.
I've read that there is a PF option - "state-policy" that should be "if-bound" if I want to match traffic in different directions. Otherwise the created state would float from one interface to other and I could not shape upload and download differently.
I make one default queue for each interface, and two specialized 'limited' queues, too. I've wrote some rules, but the problem is that traffic never goes assigned to the limited queues. And it is pretty puzzling to me, because I see there are packets matching these labels.
Here again is my pf.conf:

### Macros
int_if = "re0"
ext_if = "rl0"
ext_ip = "192.168.1.2"

### Tables
table <network>  { 192.168.0.34, 192.168.0.223 }
table <tusite>  { 222.111.111.222 }
### Options
set state-policy if-bound
### Normalizations
scrub in all

### Queueing

altq on $int_if hfsc bandwidth 8Mb queue {download, gen_down}
queue gen_down bandwidth 4Mb hfsc (realtime 2Mb upperlimit 2Mb default)
queue download bandwidth 2Mb hfsc (realtime 1Mb upperlimit 1536Kb)

altq on $ext_if hfsc bandwidth 8Mb queue {upload, gen_up}
queue gen_up bandwidth 4Mb hfsc (realtime 2Mb upperlimit 2Mb default)
queue upload bandwidth 1Mb hfsc (realtime 512Kb upperlimit 768Kb)

### Translation
nat on $ext_if from <network> to any -> $ext_ip

### Filtering
pass in log (all to pflog0) quick on $int_if from <network> to <tusite> keep state tag to_data label "to_data" queue upload
pass out log (all to pflog1) quick on $ext_if tagged to_data label "from_data" queue download

Here is some output from pftop, showing all traffic goes assigned to the default queues:

pfTop: Up Queue 1-6/6, View: queue, Cache: 10000                           20:48:19

QUEUE               BW SCH  PR   PKTS  BYTES  DROP_P DROP_B QLEN BORR SUSP P/S  B/S
root_re0         8000K hfsc  0      0      0       0      0    0             0    0
 gen_down        4000K hfsc     63051 10784K      38  34460   42           226 248K
 download        2000K hfsc         0      0       0      0    0             0    0
root_rl0         8000K hfsc  0      0      0       0      0    0             0    0
 gen_up          4000K hfsc    112576   133M       0      0    0           225 171K
 upload          1000K hfsc         0      0       0      0    0             0    0

And here is the most annoying thing - seeing traffic per label in PF - I see there are packets matching these rules:

# pfctl -s label
to_data 205761 9262 8544433 3647 155866 5615 8388567
from_data 205755 9262 8544433 5615 8388567 3647 155866

If so, why there is no traffic in the specialized queues?

[s2scott]

You cannot DIRECTLY rate-limit the traffic you RECEIVE traffic from your ISP. You CAN rate-limit the traffic you SEND (TX out an interface you/pf control).

As such, I think your queues are backwards.

QUEUE assignment(s) may be made (configured) on both in -and- out bound directions/interfaces; however, the queue-affect is in the OUT direction only of a configured interface.

/S

[s2scott]

What is (are) the essential purpose(s)/goal(s) of your queue implementation?

/S

[ivanatora]

The purpose of that queueing is that I have a content web server that is reached via another link at my ISP. The situation is as follows:
1) Our network --> the GW I'm working on
2) the GW I'm working on --> our ISP
3.1) our ISP --> Internet
3.2) our ISP ----> content server at different link

The link to that content server is at very limited bandwidth, that's why I want to make sure noone could 'takeover' all the bandwidth.
So you say that queue download and queue upload should be swapped here?

pass in log (all to pflog0) quick on $int_if from <network> to <tusite> keep state tag to_data label "to_data" queue upload
pass out log (all to pflog1) quick on $ext_if tagged to_data label "from_data" queue download

You are right, I will check that a bit later.