PF w/ ALTQ - Queue errors exceeding bandwidth

[plexter]

Hello all,

I've been trying to get PF ALTQ setup and for the life of me I cannot figure out what the errors below are meaning, or at least what they are asking me to change. I have tried playing around with the values with no luck! I'm hoping someone here can assist with setting this up.

Before I get into the errors I will first outline what I am trying to accomplish with ALTQ.

I have chosen to work with HFSC as it seemed like the best option for VOIP. If this was an incorrect choice please let me know.

My ALTQ is supposed to help reduce network congestion as I am using both time sensitive and non-time sensitive services. However I would have preferred my total bandwidth to be shared by all the queues (borrow w/ CBQ) as I do not really care if one queue takes up most or all of the bandwidth as long as it gives it back when something like VOIP needs it.

VOIP vs bittorent basically. So I am looking to firstly prioritize VOIP over everything so my calls still work while also making sure traffic still flows (even if it is slower) and finally I am trying to optimize the flow of Ack's as I am on an ADSL line.

My ALTQ Section:

altq on $ext_if bandwidth 600Kb hfsc queue { ack, dns, ssh, webftp, bitt, voip, rdpvnc, general }


queue ack priority 7 qlimit 500 hfsc (realtime 20% red ecn)
Note: I am not sure how much I should devote to ACK packets --comments?

queue voip priority 6 qlimit 500 hfsc (realtime 65% red enc)
queue dns priority 5 qlimit 500 hfsc (realtime 5% red ecn)
queue ssh priority 4 qlimit 500 hfsc (realtime 2% red ecn) {ssh_login, ssh_bulk}
queue ssh_login priority 4 qlimit 500
queue ssh_bulk priority 3 qlimit 500
rdpvnc priority 4 qlimit 500 hfsc (realtime 3% red ecn)
webftp priority 4 qlimit 500 hfsc (upperlimit 70% red ecn)
games priority 2 qlimit 500 hfsc (upperlimit 70% red ecn)
bitt     priority 3 qlimit 500 hfsc (upperlimit 80% red enc)
general priority 1 qlimit 500 hfsc (upperlimit 75% red ecn default)

I won't bother writing all of the errors however I basically get a screen full and then some of the following when running PF:

pfctl: linkshare sc exceeds parent's sc 
/etc/pf.conf:34: errors in queue definition
pfctl: the sum of the child bandwidth higher than parent "root_fxp0" 
pfctl: linkshare sc exceeds parent's sc
/etc/pf.conf:35: errors in queue definition
parent ssh not found for ssh_login
/etc/pf.conf:36: errors in queue definition

Line 35 as an example is the start of my SSH queue

....and on and on...

My pass rules (temporary):

pass out quick on $ext_if inet proto udp from any to any port 5060 keep state queue (voip, ack)
pass out quick on $ext_if inet proto udp from any to any port domain keep state queue (dns, ack)
pass out quick on $ext_if inet proto tcp from any to any port ssh modulate state queue (ssh_bulk, ssh_login)
pass out quick on $ext_if inet proto (tcp, udp) from any to any keep state queue (general, ack)

and I have one for each queue...

Also I am unclear as to how I should be placing the ACK queue. Have I done this correctly by adding it as the TOS queue on all queues?

Anyway hope someone can help!

[plexter]

Hello,

While I have been waiting for a reply I decided to setup CBQ. I have run into a more specific error which I think "could" be part of it not the reason my HFSC queuing was not working.

I am getting an error now that says "error on line 30"

Line 30 for me is:

altq on $ext_if cbq bandwidth 600Kb qlimit 500 queue { ack, dns, ssh, webftp, bitt, voip, rdpvnc, general }

I cannot see anything wrong with this!

I have also tried for the sake of argument commenting out mine and typing various examples I have seen. All produce the same error!

I just don't understand! Is ALTQ somehow not supported on my install? ....this isnt possible with a GENERIC kernel right? By default OpenBSD comes with ALTQ support...right?

Actually here is my complete PF.CONF

#Interfaces
ext_if="fxp0"
int_if="fxp1"

#Options
set require-order yes
set block-policy drop
set loginterface $ext_if
set state-policy if-bound
set fingerprints "/etc/pf.os"
set ruleset-optimization none

#Timeout Settings
set optimization aggressive
set timeout { frag 10, tcp.established 3600 }
set timeout { tcp.first 30, tcp.closing 10, tcp.closed 10, tcp.finwait 10 }
set timeout { udp.first 30, udp.single 30, udp.multiple 30 }
set timeout { other.first 30, other.single 30, other.multiple 30 }
set timeout { adaptive.start 5000, adaptive.end 10000 }

#Connection Normalization

scrub on $ext_if all random-id min-ttl 254 max-mss 1452 reassemble tcp fragment reassemble

#Queuing ALTQ
#Upload assumed aprox 80kb or 640Kb queued aprox 94%

altq on $ext_if cbq bandwidth 600Kb qlimit 500 queue { tos, dns, ssh, webftp, bitt, voip, rdpvnc, general }

queue   tos     bandwidth 20% priority 7      qlimit 500 cbq (borrow red ecn) { ack, ssh_login }
                queue   ack             bandwidth 90%   qlimit 500 cbq (borrow red ecn)
                queue   ssh_login       bandwidth 10%   qlimit 500 cbq (borrow red ecn)
queue   voip    bandwidth 50% priority 6      qlimit 500 cbq (borrow red ecn)
queue   dns     bandwidth 5%  priority 5      qlimit 500 cbq (borrow red ecn)
queue   ssh     bandwidth 2%  priority 4      qlimit 500 cbq (borrow red ecn)
queue   webftp  bandwidth 5%  priority 4      qlimit 500 cbq (borrow red ecn)
queue   games   bandwidth 5%  priority 3      qlimit 500 cbq (borrow red ecn)
queue   bitt    bandwidth 10% priority 2      qlimit 500 cbq (borrow red ecn)
queue   general bandwidth 3%  priority 1      qlimit 500 cbq (borrow red ecn default)

pass out quick on $ext_if inet proto udp from any to any port 5060 keep state queue (voip, ack)
pass out quick on $ext_if inet proto udp from any to any port domain keep state queue (dns, ack)
pass out quick on $ext_if inet proto tcp from any to any port ssh modulate state queue (ssh, ssh_login)
pass out quick on $ext_if inet proto tcp from any to any port 3389 modulate state queue (rdpvnc, ack)
pass out quick on $ext_if inet proto udp from any to any port 30340:30341 keep state queue (games, ack)
pass out quick on $ext_if inet proto udp from any to any port 6112 keep state queue (games, ack)
pass out quick on $ext_if inet proto tcp from any to any port 10050 modulate state queue (bitt, ack)
pass out quick on $ext_if inet proto (tcp, udp) from any to any keep state queue (general, ack)

pass in
pass out

Anyway I am hoping someone can help me figure this out.

Thanks for your help.

[plexter]

What has everyone gone on holiday? haha.

I'm really baffled with this. I was hoping that one of the more experienced OpenBSD users would be able to comment as his/her knowledge of OpenBSD inner workings might help. It seems really odd that neither the sample commands from OpenBSD's FAQ nor mine will work and I am starting to think there might be more going on than just my PF syntax.

ANY help would be greatly appreciated.

Updated PF.CONf also keep in mine lines may not be exact and you should look at my comments "line 30 is...." for reference.

#Interfaces
ext_if="fxp0"
int_if="fxp1"
 
#Options
set require-order yes
set block-policy drop
set loginterface $ext_if
set state-policy if-bound
set fingerprints "/etc/pf.os"
set ruleset-optimization none
 
#Timeout Settings
set optimization aggressive
set timeout { frag 10, tcp.established 3600 }
set timeout { tcp.first 30, tcp.closing 10, tcp.closed 10, tcp.finwait 10 }
set timeout { udp.first 30, udp.single 30, udp.multiple 30 }
set timeout { other.first 30, other.single 30, other.multiple 30 }
set timeout { adaptive.start 5000, adaptive.end 10000 }
 
#Connection Normalization
 
scrub on $ext_if all random-id min-ttl 254 max-mss 1452 reassemble tcp fragment reassemble
 
#Queuing ALTQ
#Upload assumed aprox 80kb or 640Kb queued aprox 94%
 
altq on $ext_if cbq bandwidth 600Kb queue { tos, voip, dns, ssh, webftp, bitt, general }
 
queue   tos     bandwidth 20% priority 7      cbq (borrow red) { ack, ssh_login }
                queue   ack             bandwidth 90%   cbq (borrow red)
                queue   ssh_login       bandwidth 10%   cbq (borrow red)
queue   voip    bandwidth 50% priority 6      cbq (borrow red)
queue   dns     bandwidth 7%  priority 5      cbq (borrow red)
queue   ssh     bandwidth 3%  priority 4      cbq (borrow red)
queue   webftp  bandwidth 5%  priority 4      cbq (borrow red)
queue   bitt    bandwidth 10% priority 2      cbq (borrow red)
queue   general bandwidth 5%  priority 1      cbq (borrow red default)
 
pass out quick on $ext_if inet proto udp from any to any port 5060 keep state queue (voip, ack)
pass out quick on $ext_if inet proto udp from any to any port domain keep state queue (dns, ack)
pass out quick on $ext_if inet proto tcp from any to any port ssh modulate state queue (ssh, ssh_login)
pass out quick on $ext_if inet proto tcp from any to any port 10050 modulate state queue (bitt, ack)
pass out quick on $ext_if inet proto { tcp, udp } from any to any keep state queue (general, ack)
 
pass in
pass out

Thanks!

[roundkat]

I had the same need as you.. this is my pf.conf
(that has been modded over the past few years)

Note: I am not a pf expert..
I did a lot of trial and error to get this to work..

There are some Stellar pf folks here and I am surprised that one
of them has not answered..

My goal was the same as yours...
To give VOIP traffic #1 priority over everything..

Granted pf changes over time but this has worked for me for
the past couple of years.

TODO
Update pf :-)

# --------------------
# Variable definitions
# --------------------

  ext_if = "rl0"
  int_if = "em0"
  mail = "192.168.1.110"
  mymailserver = "{sanitized}" 
  smtp2 = "2500"
  email = "{https, imaps, 465 }"
  torrent = "{ 43804, 35834,6346,57462,10517,36791,16883 }"
  httpmail = "{ 81 }"
  apc ="{ 8001 }"
  misc ="{1024, 9001 }"
 
  client = "{ 43804,16883, 5190, 20, 8880, 81, 2082, 110, 5526, ntp, 53, 8000, \
  8080, ftp, whois, 5525, http, https, ssh, imaps, imap, 2200, \
  smtp, 6346,57462, 10517,36791, 113, 465, 45000 ><65000 }"
 
# Vonage Adapter
    voiphost = "{192.168.1.2}"
# Vonage traffic ports (SIP and RTP).
    voipports = "{ 5060, 5061, 10000:20000 }"

# ---------------------
# Generic configuration
# ---------------------

set loginterface $ext_if
set fingerprints "/etc/pf.os"

# http://rfc.net/rfc1918.html
  reserved = " {
  0.0.0.0/8,      10.0.0.0/8,     20.20.20.0/24,  127.0.0.0/8,
  169.254.0.0/16, 172.16.0.0/12,  192.0.2.0/24,   192.168.0.0/16,
  224.0.0.0/3  } "

#Tables
table <cidr> persist file "/etc/tables/cidr"
 table <scanners> persist

# -----------
# scrub rules
# -----------

 scrub in all
#   scrub in on $ext_if all random-id
#  scrub  on $ext_if all reassemble tcp

# -----------
# AltQ rules
# -----------
  altq on $ext_if priq bandwidth 1124Kb queue { std, voip, tcpack }
  queue std priq(red default)
  queue tcpack priority 10 priq(red)
  queue voip priority 15 priq(red)

# ---------
# Nat and Rdr rules
# ---------
## nat-anchor "ftp-proxy/*"
## rdr-anchor "ftp-proxy/*"
## rdr pass on $int_if proto tcp to port ftp -> 127.0.0.1 port 8021

  nat on $ext_if inet from ($int_if:network) to any -> ($ext_if:0)

# Redirect Rules
# note: if this is commented out, no access to ssh remotely
   set skip on {lo0 lo1}

# Spamd Rule
  no rdr on $ext_if proto tcp from <spamd-white> to any port smtp
  rdr pass on $ext_if proto tcp from any to any port smtp \
        -> 127.0.0.1 port spamd

# Redirect Rules for internal Services 
  rdr on $ext_if proto tcp from $mymailserver to $ext_if port $smtp2 -> $mail port smtp
  rdr on $ext_if proto tcp from $mymailserver to $ext_if port imap -> $mail port imap
  rdr on $ext_if proto tcp from any to $ext_if port $email -> $mail
  rdr on $ext_if proto tcp from any to $ext_if port $apc -> 192.168.1.10
  rdr on $ext_if proto tcp from $mymailserver to $ext_if port $ssh2 -> $mail port ssh
  rdr on $ext_if proto tcp from any to $ext_if port $httpmail -> $mail 

# --------
# pf rules
# --------
## anchor "ftp-proxy/*"

# Default policy.
# remove the log-keyword if you ever get ddossed.
  block in log on $ext_if all
# block in on $ext_if all

# ---------------
# SPOOFERS
# ---------------

# Block spoofing.
  block in on $ext_if from $reserved to any
  block out on $ext_if from $reserved to any

# Block the table data 
 block in quick on $ext_if from <cidr> to any

 antispoof for { $int_if, $ext_if }

 block drop in quick on $ext_if from any os { SCO, NMAP }

# ----
# ICMP
# ----
  pass out on $ext_if inet proto icmp all icmp-type 8 code 0 
  pass in  on $ext_if inet proto icmp all icmp-type 8 code 0 

# ---
# Internal network.
# ---
# Note that in order for our special Vonage queue to work, we need to
# tag incoming traffic so we can identify it on its way out.
#
 pass in on $int_if from $int_if:network to any 
 pass in quick on $int_if proto udp from $int_if:network to any \
 port $voipports tag VONAGE_OUT 

# ---
# UDP
# ---
# outgoing UDP
   pass out on $ext_if proto udp all 
# bitorrent UDP
  pass in on $ext_if inet proto udp from any to any port $torrent 

# ---
# TCP
# ---

# misc blocked outgoing ports
  block in quick on $ext_if inet proto { tcp, udp}  from any to any port $misc

# bitorrent
   pass in on $ext_if inet proto tcp from any to any port $torrent 

# ssh and brute force table
   pass in quick on $ext_if proto tcp from !<scanners> to $ext_if port ssh flags S/SA \
   synproxy state (max-src-conn-rate 2/60, overload <scanners> flush)

# Pass rule for Webserver on Mailserver
   pass in on $ext_if inet proto tcp from any to any port $apc 
   pass in on $ext_if inet proto tcp from any to any port ssh 
   pass in on $ext_if inet proto tcp from any to any port http 
   pass in on $ext_if inet proto tcp from any to any port $httpmail 

# Pass rule for Mailserver
   pass in on $ext_if inet proto tcp from any to any port $email 
   pass in on $ext_if inet proto tcp from any to any port smtp 
   pass in on $ext_if inet proto tcp from $mymailserver to any port smtp 
   pass in on $ext_if inet proto tcp from $mymailserver to $mail port imap

# Outgoing TCP

# current pass statement with port restrictions
   pass out quick on $ext_if proto tcp to any port $client queue(std, tcpack)

# Outgoing Tagged Vonage Traffic
  pass out quick on $ext_if tagged VONAGE_OUT queue(voip, tcpack)

[plexter]

I have even tried updating my src (stable) and recompiling but still no luck. I am left to hope that if this is a glitch on OpenBSD 4.3's side that with the release of 4.4 it will be resolved.

Hopefully someone can comment on "our" situation?

[roundkat]

Originally Posted by plexter

I have even tried updating my src (stable) and recompiling but still no luck. I am left to hope that if this is a glitch on OpenBSD 4.3's side that with the release of 4.4 it will be resolved.

Hopefully someone can comment on "our" situation?

P...
What I have posted does work for giving VOIP priority..

I just need to do some house cleaning..

rk

[plexter]

I can try doing a priq instead of CBQ or HFSC but I'm not sure why that would change anything. (be nice if it did though)

Will let you know.
Thanks

[roundkat]

Originally Posted by plexter

I can try doing a priq instead of CBQ or HFSC but I'm not sure why that would change anything. (be nice if it did though)

Will let you know.
Thanks

I noticed you had posted on EE.. will be curious to see
what gheist comes up with..

In the mean time I have re-written my rules based on the
altq example http://openbsd.org/faq/pf/queueing.html#example1

The ruleset is working but something is amiss on the voip side.
The phone rings but answering machine does not pick up..
When I switch back the old ruleset, the answering machine
does pick up..

Running 4.4 -current...
Still trying to find the issue..

rk

[s2scott]

sip, as a protocol, has its quirks. You may need to understand "invite" and "re-invite," as this drives which side establishes the session, and this drives how the pf rule set can instrument queuing.

I don't use vonage, but I do use a voip wholesaler, voicenetwork.ca, with TRIXBOX CE behind my pf.

in a NO re-invite sip configuration, the following works well.

table <VSP> const { sipgw.voicenetwork.ca }
#...
altq on outside priq bandwidth 1024Kb queue {q0,q1,q4,q5vpn,q6voip,q7}
queue q7     priority 7
queue q6voip priority 6
queue q5vpn  priority 5
queue q4     priority 4
queue q1     priority 1 priq(default)
queue q0     priority 0
#
nat on outside inet proto udp tagged OKSIPOUT -> (outside:0) static-port
#
pass in  log quick on inside inet proto udp \
 from (inside:network) to <VSP> tag OKSIPOUT \
 keep state queue(q6voip,q7)
#
pass out log quick on outside inet proto udp \
 tagged OKSIPOUT keep state queue(q6voip,q7)
#

In a YES re-invite sip configuration, then add...

#
rdr on outside inet proto udp \
 from <VSP> to (outside:0) port 5060 \
 tag OKSIPIN -> <inside_sip_endpoint>
#
pass in log quick on outside inet proto udp \
 tagged OKSIPIN keep state queue(q6voip,q7)
#
pass out log quikc on inside inet proto udp \
 tagged OKSIPIN keep state queue(q6voip,q7)
#

The point being with a re-invite enabled, or VSP initiated SIP streams, sip topologies, the udp stream is being initiated by the VSP side; consequently, pf has to rule-catch this inbound stream and attach it to a queue for the reply stream BEFORE pf's default state and queue attaches.

/S

[plexter]

Hi roundkat,

I want to thank you for posting your config. I tried using what you wrote for ALTQ and it seems to work! Then I tried writing mine again little simpler version this time and it works too! I still have NO IDEA why my original script would not work. As far as I can tell they all look the same to me as far as syntax goes. haha!

Oh well it seems to be working so I'm happy.

Thank you very much.

Case closed....for now. :P

[roundkat]

P...
np..
btw..
Scott is really a pf wizard..
his understanding of pf is orders of magnitude over mine...

I had to back off the Example.. I am not accustomed to blocking
inbound and outbound on *both* interfaces...

Oh.. Scott, I found out why the VIOP had issues...
I forgot to let dns out.. just an oversight as I was testing..

I got to the point of all my services working..
i.e.
Mailserver receiving (not sending)
Webserver working
DNS /NTP working..

After much trial and error I could not get the rule order figured it out.
so decided to shelve that ruleset for now.. will come back to it later..

I did do a re-write clean up of my current ruleset to make it easier to read..

Once I sanitize it a little I will post back..

My goal is to end up with an optimized ruleset using what I have..

I have made a few changes.. and just got off the phone with my
mate in the UK while I was downloading an ISO and running bittorent..
I did get some latency but not sure if it had to do with my ruleset..

P..
Most folks will tell you not to "blindly" cut and paste pf rules.. but you have
to start somewhere..
Take a look to see what is actually happening with the ruleset ..

As Jiggimi states http://daemonforums.org/showthread.php?t=1070

Step 0: Make sure all pass and block rules log their actions.

Step 1. Make sure net.inet.ip.forwarding=1.

Step 2. Use "# tcpdump -neti pflog0" to see what rule # is blocking

Step 3. Use "# pfctl -vvsr" to see match the number to the rule"

Lather. Rinse. Repeat.

This is what I am using to see what is going on with my current rules to
try and optimize them..

Scott, thanks for your input..

rk

[s2scott]

Originally Posted by roundkat

P...
np..
btw..
Scott is really a pf wizard..

Scott, thanks for your input..

rk

<blush>, and

You're very welcome.

Hope the "static-port" modifier helps someone, sometime along the way.

/S