Firewall routing

[Magoo]

I have a device rl0 for the wan and a device rl1 from the lan. The lan has no problem leasing clients via dhcpd and the wan has no problem retrieving a lease via dhclient. The problem is that the traffic coming from the lan does not pass on to the wan. I'm assuming I may have a routing problem, but I'm not sure. Any assistance is appreciated, please let me know if you need more information. Here is the critical configuration info:

# netstat -rn

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            XXX.XXX.XXX.1        UGS         0        5    rl0
XXX.XXX.XXX/22       link#1             UC          0        0    rl0
XXX.XXX.XXX.1        00:1a:2f:8b:30:05  UHLW        2        0    rl0   1199
127.0.0.1          127.0.0.1          UH          0        0    lo0
169.254.75/24      link#2             UC          0        0    rl1
169.254.75.254     00:1f:33:cd:a9:59  UHLW        1      358    rl1   1170

----------------
# cat /etc/rc.conf

gateway_enable="YES"
natd_enable="YES"
natd_interface="rl0"
ipnat_enable="YES"
ifconfig_rl1="inet 169.254.75.1 netmask 255.255.255.0"
dhcpd_enable="YES"
dhcpd_ifaces="rl1"
sshd_enable="YES"
ifconfig_rl0="DHCP"
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"

------------------
# cat /etc/ipfw.rules

IPF="ipfw -q add"
ipfw -q -f flush

#loopback
$IPF 10 allow all from any to any via lo0
$IPF 20 deny all from any to 127.0.0.0/8
$IPF 30 deny all from 127.0.0.0/8 to any
$IPF 40 deny tcp from any to any frag

# statefull
$IPF 50 check-state
$IPF 60 allow tcp from any to any established
$IPF 70 allow all from any to any out keep-state
$IPF 80 allow icmp from any to any

# open port ftp (20,21), ssh (22), mail (25)
# http (80), dns (53) etc
$IPF 130 allow tcp from any to any 22 in
$IPF 140 allow tcp from any to any 22 out
$IPF 170 allow udp from any to any 53 in
$IPF 175 allow tcp from any to any 53 in
$IPF 180 allow udp from any to any 53 out
$IPF 185 allow tcp from any to any 53 out
$IPF 200 allow tcp from any to any 80 in
$IPF 210 allow tcp from any to any 80 out

# deny and log everything
$IPF 500 deny log all from any to any

----------------------
# cat /usr/src/sys/i386/conf/MYKERNEL

[truncated]
options         IPFIREWALL
options         IPFIREWALL_DEFAULT_TO_ACCEPT
options         IPFIREWALL_FORWARD
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=10
options         IPDIVERT
[truncated]

[phoenix]

Where are your NAT rules? 169.254.0.0/16 is unroutable, and even if it does make it through your firewall, the next hop router will drop all packets. You never divert any packets to natd, so they don't get NAT'd, so they are unroutable.

[Magoo]

I tried doing the following now:

Firewall rules added to beginning:
/sbin/ipfw add divert natd all from any to any via rl0
/sbin/ipfw add pass all from any to any

Added to rc.conf:
natd_flags="-f /etc/natd.conf"

Added to /etc/natd.conf:
interface rl0
use_sockets yes
same_ports yes
dynamic yes

Still no go. Please let me know if there's something I'm overlooking.

[Magoo]

I tried doing the following now:

Firewall rules added to beginning:
/sbin/ipfw add divert natd all from any to any via rl0
/sbin/ipfw add pass all from any to any

Added to rc.conf:
natd_flags="-f /etc/natd.conf"

Added to /etc/natd.conf:
interface rl0
use_sockets yes
same_ports yes
dynamic yes

Still no go. Please let me know if there's something I'm overlooking.

[Magoo]

I also added

map rl0 169.254.75.0/24 -> 0.0.0.0/32 portmap tcp/udp auto
map rl0 169.254.75.0/24 -> 0.0.0.0/32

to ipnat.conf and then ran ipnat -f /etc/ipnat.conf

Still with no success.

[phoenix]

You don't use ipnat with ipfw. ipnat is part of IPFilter.

The bare minimum rules you need for NAT to work are:

#!/bin/sh

natd -same_ports -use_sockets -dynamic -interface rl0

ipfw add divert natd ip from any to me in recv rl0
ipfw add allow ip from any to <privatesubnet> in recv rl0
ipfw add allow ip from any to <privatesubnet> out xmit rl1

ipfw add allow ip from <privatesubnet> to any in recv rl1
ipfw add divert natd ip from <privatesubnet> to any out xmit rl0
ipfw add allow ip from me to any out xmit rl0

The natd command-line will use the IP of the rl0 interface, and the dynamic keyword will make sure that the process is always current, in case the IP changes.

The keyword natd gets translated to the default natd port of 8668. If you use a different port in the natd command (-port <whatever>), then you put that number into the ipfw divert rule.

The keyword me gets dynamically translated to "any IP that I am currently listening on", so that if the IP of the public interface changes (due to dhclient updates) the rules will continue to work.

The rules above can be simplified, but I find making the rules as exact and specific as possible, with separate rules for traffic coming in (in public, out private NIC) and going out (in private, out public NIC) makes it easier to understand what's going on as the packets go through the firewall.

Oh, and I don't use /etc/rc.conf to enable/configure the firewall rules. I find it a lot easier to write custom scripts that take care of everything natd/ipfw related. You can point the firewall_script option in rc.conf to the custom script, or write a custom rc.d wrapper for it (which is what we do).

[Magoo]

I tried putting the ruleset you listed in ipfw.rules by itself and I still didn't see traffic passing from rl1 to rl0. I also tried those rules in conjunction with the rules I listed in different orders and it still didn't pass through. I replaced <privatesubnet> with 169.254.75.0/24 by the way. Any ideas what I might be doing wrong?

[phoenix]

Can you post the output of the following commnds (use [ code ] tags around it):

ifconfig rl0
ifconfig rl1
netstat -rn
pgrep -lf natd
ipfw show

You can x out the first two or three octets of the IPs if needed, but don't touch the netmasks.

[Magoo]

---
# ifconfig rl0

rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=8<VLAN_MTU>
        inet XXX.XXX.XXX.69 netmask 0xfffffe00 broadcast 255.255.255.255
        ether 00:c0:f0:54:c0:bd
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active

---
# ifconfig rl1

rl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=8<VLAN_MTU>
        inet 169.254.75.1 netmask 0xffffff00 broadcast 169.254.75.255
        ether 00:50:ba:5f:0c:d4
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active

---
# netstat -rn

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            XXX.XXX.XXX.1         UGS         0        4    rl0
XXX.XXX.XXX/23        link#1             UC          0        0    rl0
XXX.XXX.XXX.1         00:1a:2f:8b:30:05  UHLW        2        0    rl0   1190
127.0.0.1          127.0.0.1          UH          0        0    lo0
169.254.75/24      link#2             UC          0        0    rl1
169.254.75.254     00:1f:33:cd:a9:59  UHLW        2      194    rl1   1160

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UHL         lo0
fe80::%lo0/64                     fe80::1%lo0                   U           lo0
fe80::1%lo0                       link#3                        UHL         lo0
ff01:3::/32                       fe80::1%lo0                   UC          lo0
ff02::%lo0/32                     fe80::1%lo0                   UC          lo0

---
# pgrep -lf natd

419 natd -same_ports -use_sockets -dynamic -interface rl0

---
# ipfw show

00010   0     0 allow ip from any to any via lo0
00020   0     0 deny ip from any to 127.0.0.0/8
00030   0     0 deny ip from 127.0.0.0/8 to any
00040   0     0 deny tcp from any to any frag
00050   0     0 check-state
00060 685 69553 allow tcp from any to any established
00070   9   697 allow ip from any to any out keep-state
00080   1    61 allow icmp from any to any
00130   3   180 allow tcp from any to any dst-port 22 in
00140   0     0 allow tcp from any to any dst-port 22 out
00170   0     0 allow udp from any to any dst-port 53 in
00175   0     0 allow tcp from any to any dst-port 53 in
00180   0     0 allow udp from any to any dst-port 53 out
00185   0     0 allow tcp from any to any dst-port 53 out
00200   0     0 allow tcp from any to any dst-port 80 in
00210   0     0 allow tcp from any to any dst-port 80 out
00341   2    80 divert 8668 ip from any to me in recv rl0
00342   0     0 allow ip from any to 169.254.75.0/24 in recv rl0
00343   0     0 allow ip from any to 169.254.75.0/24 out xmit rl1
00344   0     0 allow ip from 169.254.75.0/24 to any in recv rl1
00345   0     0 divert 8668 ip from 169.254.75.0/24 to any out xmit rl0
00346   0     0 allow ip from me to any out xmit rl0
00500  52 19988 deny log logamount 10 ip from any to any
65535   0     0 allow ip from any to any

I wasn't sure what level to put the nat divert rules at, I tried putting them at the beginning and at the end.

[phoenix]

Ah, you're using stateful filtering rules and natd. Good luck with that. The rules needed to make that work are quite complicated. I've never bothered trying, just trying to decipher the examples given in mailing lists makes my head spin.

Try it without the stateful rules.

You're also using link-local auto-configuration IPs (169.254.x.x). Try using a proper private subnet like 192.168.x.x, or 10.x.x.x.

Other than that, the network config looks correct.