Multihome, packets leaving by "wrong" interface

[irukandji]

My setup:

         +--------+                +--------+
         |internet|                |internet|
         +----|---+                +---|----+
              |                        |
              |                        |
    +---------|---------+     +--------|---------+
    |    adj.router     |     |   93.27.123.23   |
    |   (VPN server)    |     |    lan router    |
    |    10.10.10.1     |     |(nat and port fwd)|
    +---------|---------+     |   192.168.1.1    |
              |               +--------+---------+
              |                        |
              |                        |
    +---------|------------------------|----------+
    |         |                        |          |
    |        tun0                     em0         |
    |     10.10.10.77            192.168.1.200    |
    |   (default route)            |       |      |
    |                              |       |      |
    |  +--------------+       +----|---+---|---+  |
    |  | client tools |       |80:HHTPD|21:SSHD|  |
    |  +--------------+       +--------+-------+  |
    |                                             |
    +---------------------------------------------+

I want to keep my srv daemons beeing accessible by static ip (93.27.123.23) while all other communication going out via tun0.

Lan router is having port forwards to daemons on the host. When the openvpn (as client) is running it sets the route for 0.0.0.0 to its gateway and becoase of this (at least i speculate this is the reason), the SYN comes from the internet to em0 but ACK leaves the server via tun0. I believe the pf reply-to should be able to enforce tcp packets leaving on the same interface where the tcp session was established but except from regularly killing my networking i wasnt able to configure it

Can someone please help me, i cant post rules i have written until now as my network is down again and i am on remote location Once i get to the console, i'll also provide netstat -r

[jggimi]

Hello, and welcome!

Every time your tun interface is destroyed and recreated, your applicable PF rules are lost.
This link mentions an option to OpenVPN which may assist.

http://marc.info/?l=openbsd-misc&m=137175773628106&w=2

[irukandji]

Without openvpn running:

Destination        Gateway            Flags    Refs      Use  Netif Expire
default            rtr                UGS         0 16024767    em0
localhost          link#7             UH          0    73063    lo0
192.168.1.0        link#1             U           0   433405    em0
mini               link#1             UHS         0      474    lo0

Internet6:
Destination        Gateway            Flags      Netif Expire
::                 localhost          UGRS        lo0
localhost          link#7             UH          lo0
::ffff:0.0.0.0     localhost          UGRS        lo0
fe80::             localhost          UGRS        lo0
fe80::%em0         link#1             U           em0
fe80::222:4dff:fe8 link#1             UHS         lo0
fe80::%lo0         link#7             U           lo0
fe80::1%lo0        link#7             UHS         lo0
ff01::%em0         fe80::222:4dff:fe8 U           em0
ff01::%lo0         localhost          U           lo0
ff02::             localhost          UGRS        lo0
ff02::%em0         fe80::222:4dff:fe8 U           em0
ff02::%lo0         localhost          U           lo0

With openvpn running:

Destination        Gateway            Flags    Refs      Use  Netif Expire
0.0.0.0/1          10.10.10.77        UGS         0      123   tun0 =>
default            rtr                UGS         0 16028962    em0
10.10.10.1/32      10.10.10.77        UGS         0        0   tun0
10.10.10.77        link#9             UH          0        0   tun0
10.10.10.78        link#9             UHS         0        0    lo0
62.212.85.79/32    rtr                UGS         0      191    em0
localhost          link#7             UH          0    73063    lo0
128.0.0.0/1        10.10.10.77        UGS         0       64   tun0
192.168.1.0        link#1             U           0   433475    em0
mini               link#1             UHS         0      474    lo0

Internet6:
Destination        Gateway            Flags      Netif Expire
::                 localhost          UGRS        lo0
localhost          link#7             UH          lo0
::ffff:0.0.0.0     localhost          UGRS        lo0
fe80::             localhost          UGRS        lo0
fe80::%em0         link#1             U           em0
fe80::222:4dff:fe8 link#1             UHS         lo0
fe80::%lo0         link#7             U           lo0
fe80::1%lo0        link#7             UHS         lo0
fe80::%tun0        link#9             U          tun0
fe80::222:4dff:fe8 link#9             UHS         lo0
ff01::%em0         fe80::222:4dff:fe8 U           em0
ff01::%lo0         localhost          U           lo0
ff01::%tun0        fe80::222:4dff:fe8 U          tun0
ff02::             localhost          UGRS        lo0
ff02::%em0         fe80::222:4dff:fe8 U           em0
ff02::%lo0         localhost          U           lo0
ff02::%tun0        fe80::222:4dff:fe8 U          tun0

Originally Posted by jggimi

Every time your tun interface is destroyed and recreated, your applicable PF rules are lost.

Thank you for this i didnt know that...

[jggimi]

With OpenVPN running, you define "0.0.0.0/1" for tun0, but that isn't a default route and it isn't being used. Your default route (0.0.0.0/0, by the way) is still pointing to "rtr".

[jggimi]

I should point out that I'm neither a FreeBSD nor an OpenVPN user. Any advice I can offer would be either PF-specific or networking generic.

If you'll post your RELENG version, I will at least be able to figure out which revision of PF is being used. There have been many changes over the years that affect both capabilities and syntax.

http://forums.freebsd.org/showthread.php?t=39295

[irukandji]

Originally Posted by jggimi

PF-specific

This is just perfect

pf version is OpenBSD 4.5

Well what i did was define the following rule for pf, looks like it works to some point (the daemons are accessible via 93.27.123.23) but now i cant access them from lan :@

pass in on em0 reply-to (em0 192.168.1.1) to 192.168.1.200 keep state

[jggimi]

Thanks. The routing options (route-to, reply-to, dup-to) syntax changed at 4.7.

Your routing option applies to all incoming traffic on em0 destined for the server (192.168.1.200). Either add another standard pass rule without reply-to after this one for your two TCP services, or add a quick pass rule that doesn't use reply-to for those two TCP services before this rule.

Remember, with PF, unless the "quick" option is used, the last matching rule wins. If you use standard pass rules, try something like:

pass in on em0 reply-to (em0 192.168.1.1) to 192.168.1.200 keep state
pass in on em0 proto tcp from any to 192.168.1.200 port {21 80}

The first rule matches all incoming traffic, the second rule without reply-to matches your two services, and "wins" for that traffic.

You may also need to adjust rules for your FTP data connections, depending on whether passive or active FTP is used and the type of FTP proxy deployed, if any.

----
Edited to add: keep state became the default at 4.1.

[irukandji]

unless the "quick" option is used, the last matching rule wins

jggimi thank you very very very much!!!! This was actually my problem...