home router + firewall behind ISP router

[beiroot]

Hi,
I've been trying to set up an OpenBSD firewall inside my local network behind my ISP's router. I generally know what and how I should do it.

I've read openbsd "Building a router", Peter Hansen's "PF Guide" and gazillion threads. The closest to what I need are those two threads:
http://daemonforums.org/showthread.php?t=7953
http://daemonforums.org/showthread.php?t=9782

My architecture is like that:


INTERNET <-> ISP router (public, dhcp) to 192.168.1.1<-> 192.168.1.254 (OPENBSD two nic) 10.0.0.1 <-> 10.0.0.2 SWITCH (soho router used as a switch) <- (dhcp - e.g. 10.0.0.100) LAN (home computers, linux)

nevermind the dhcp on the inside - it's just temporary

From the openbsd router itself, I can ping every host in my network and the internet.
From the machines in 10.0.0.0 I can ping the 10/8 and 192.168.1.0 network, but I can't reach to the internet.

jggimi said in http://daemonforums.org/showthread.php?t=7953:

Additional routes would only be needed if you had additional routers. You don't.

Let's pretend for a moment that you had a different architecture, one with multiple routers, such as:

{Internet} - [Firewall A] - {DMZ servers} - [Firewall B] - {everything else}

The DMZ server subnet has two routers. Those servers need two routes: a default route through Firewall A to the Internet, and a specific route through Firewall B to that inner subnet. Firewall A also needs two routes -- a default route through the ISP, and a specific route to the inner subnet.

Without adding a specific (sometimes called static) route, those packets would be shipped to the default route, which would be to the wrong router.

and this in http://daemonforums.org/showthread.php?t=9782:

You have two subnets, 192.168.0/24 (outer), and 192.168.1/24 (inner), and packet forwarding enabled. That would be fine, if either 1 or 2 were true, but not both:

Your ISP's router (your defined "cable modem") has a routing table that includes a route to the 192.168.1/24 network through 192.168.0.2, your OpenBSD device.
Your OpenBSD router uses NAT. (If NAT is used, that would be provisioned in your missing pf.conf)

So what am I doing wrong?
1. What (static) routes should I add and where? I can't add any on my ISP's router.
or
2. What NAT rules should I have in my PF or in general, how should my pf.conf look in order just to connect nodes in the network to the internet?
or
3. Should I use vether(4)?

[jggimi]

Originally Posted by beiroot

So what am I doing wrong?

I don't know, you haven't posted what is failing. But I can guess, because you state you cannot add any routes to your ISP gateway.

Here's what I understand from your post. You have two local networks: 192.168.1/24, and 10.0.0/24.

[ISP gateway] - 192.168.1/24 - [your router] - 10.0.0/24 - [your stuff]

Let's assume that's correct.

Now, let's level-set. Forgive me if this is already understood:

• A routing table entry basically consists of two fields: 1) a subnet, and 2) the address of a router on the local network that will forward packets to or towards that subnet.
• The vast majority of end user devices, like those above I named [your stuff], will only need a single entry in their routing tables: The 0.0.0.0/0 subnet, which is the subnet of everywhere, and the address of the single router on the local network. This 0.0.0.0/0 routing table entry is also called the "default route."
• Whenever a packet is destined for a device on the local network, no routing is needed. The end user devices know if it is local or not because their IP address is configured with a netmask (or a "/" CIDR, which is applied as a netmask).
• Whenever a packet is destined for a device OFF the local network, the routing table is consulted. For most end-user devices, there will only be a default route, and a single router. =

With that out of the way, let's assume for the moment that you are NOT using NAT on the local networks.

• All [your stuff] devices need a single default route, pointing to the local router at 10.0.0.1.
• That router at 10.0.0.1 also needs a default route, pointing to your ISP gateway at 192.168.1.1. That way lies the Internet. But the router at 10.0.0.1 should have a second routing table entry, because you defined two NICs. That routing table entry is automatic, and points to all the devices on the 10.0.0/24 network. It happens at network configuration time automatically, because you defined a NIC with that subnet attached to it. Your router has two NICs, and therefore two routes, one of them default, to get to the Internet.
• Any "DMZ" servers you might have on the 192.168.1/24 network would need 2 routes also. A default route to the ISP gateway at 192.168.1.1, and an additional route to the 10.0.0/24 network via your router at 192.168.1.254. You would add this second route manually.
• The ISP gateway router today has two routes it knows about. A default route, provided to it by your ISP, and it knows about the 192.168.1/24 network. That is not enough in this situation.

The ISP gateway device doesn't know ANYTHING about 10.0.0/24, so it can't route packets to that subnet. This is why you would need to add a route to it, so that it knows to route traffic destined for 10.0.0/24 devices through the router sitting at 192.168.1.254, that it can reach directly.

You have stated is it not possible to add a route to the ISP gateway device. I don't believe it. I believe *you* believe it, and I believe the people you might have spoken with at your ISP believe it.

But whether I believe it or not is immaterial. There are two solutions, one if you (and your ISP, perhaps) are misinformed, and one where I lose my faith in TCP/IP hardware vendors.

----

Solution 1 - add a static route to the ISP gateway device. To be used if I'm right, and if you (and perhaps your ISP) are wrong. This is the preferred method.

The two fields needed in that additional route are the subnet: 10.0.0/24, and the router that reaches it: 192.168.1.254.

---

Solution 2 - Add NAT translation to your router at 192.168.1.254/10.0.0.1. This is not preferred, because "double-NAT" introduces complexity, can cause problems for some protocols, and can make diagnosing problems difficult. Therefore, only to be used if I am proven wrong about the device I have named your ISP gateway. When you add NAT to traffic on your 10.0.0/24 network, any traffic that transits the router will be translated, and appear to originate from 192.168.1.254. Because of this, the ISP gateway will not need to have any knowledge of the 10.0.0/24 network.

[beiroot]

jggimi thank you for your as always thorough, clear and exhaustive answer. You are truly ment to be a guru

As far as the merrits are concerned. I have written what is my problem

From the openbsd router itself, I can ping every host in my network and the internet.
From the machines in 10.0.0.0 I can ping the 10.0.0.0 and 192.168.1.0 network, but I can't reach to the internet.

but you deduced it correctly anyway. Again chapeaus bas

Unfortunately, it's the problem I thought it was - either add route to the isp gateway or double nat.

The problem with my ISP router is that it's a typical consumer product (S@GEM) given by the ISP. It has the web-interface set up by ISP with hardly any settings. And I know there's a linux inside I could control if was given access - but probably that's not gonna happen. I'm gonna call the ISP and talk about either me or them adding a route, but people on the phone mostly aren't technical.

I could of course change the ISP gateway to something more commercial: D-link, tp-link etc., but that would require additional money I don't want to invest right now. I know it's not much, but still...

What would be the pf rules for a double nat on the router?
Again, my problem is:

From the openbsd router itself, I can ping every host in my network and the internet.
From the machines in 10.0.0.0 I can ping the 10.0.0.0 and 192.168.1.0 network, but I can't reach to the internet.

What's funny (probably only to me) is that from 10.0.0.0 network I can ping the whole 192.168.1.0 network, but not the ISP gateway (192.168.1.1). It's not the firewall because I can ping it from other machines inside 192.168.1.0.
Another funny thing is that from 10.0.0.0 network, when I ping external domains (e.g. daemonforums.org) the domain gets resolved to IP, but the ping is dead. What the truck? Is this because my ISP router is also a DNS server to my local network?

[jggimi]

Originally Posted by beiroot

What would be the pf rules for a double nat on the router?

Review the Network Address Translation section of the PF User's Guide. Your rule would likely be similar to:

match out on <your.external.NIC> from 10.0.0.0/24 to any nat-to 192.168.1.254

What's funny (probably only to me) is that from 10.0.0.0 network I can ping the whole 192.168.1.0 network, but not the ISP gateway (192.168.1.1).

Of course. That router does not know anything about the 10.0.0/24 network, it *can not* reply to any 10.0.0/24 address.

Another funny thing is that from 10.0.0.0 network, when I ping external domains (e.g. daemonforums.org) the domain gets resolved to IP, but the ping is dead. What the truck? Is this because my ISP router is also a DNS server to my local network?

It's because that router doesn't know how to send packets to 10.0.0/24. Packets such as these would normally go back out the default route ... towards the Internet. But these addresses are RFC1918 addresses, so they should never be routed on the Internet. Either the ISP gateway or the upstream router will drop the packets.

[beiroot]

Ok, I had this rule but instead of any i had sis0:network, so it didn't cross my ISP router. stupid mistake...but! for my excuse ...I try not to eat from the bowl of copypasta and understand the whole thing so I write pf rules on my own

Of course. That router does not know anything about the 10.0.0/24 network, it *can not* reply to any 10.0.0/24 address.

but I can ping e.g. 192.168.1.17 and not 192.168.1.1 (ISP router). I'll check it again because I feel again it's a stupid mistake.

It's because that router doesn't know how to send packets to 10.0.0/24. Packets such as these would normally go back out the default route ... towards the Internet. But these addresses are RFC1918 addresses, so they should never be routed on the Internet. Either the ISP gateway or the upstream router will drop the packets.

But how does the computer inside 10/24 know how to resolve google.com to correct ip address? and he knows... Is it because I set the DNS to 192.168.1.1 which has the dns server (set by isp) and resolves the address?

[jggimi]

Originally Posted by beiroot

But how does the computer inside 10/24 know how to resolve google.com to correct ip address? and he knows... Is it because I set the DNS to 192.168.1.1 which has the dns server (set by isp) and resolves the address?

Based on my understanding of your network, which may be incomplete or incorrect:

If a device on the 10.0.0.0/24 network -- OTHER than your router -- is directed to use the nameserver running in the ISP gateway device at 192.168.1.1, it will not receive any domain resolution responses from the device.

Let us pretend it is a workstation at address 10.0.0.5.

• Domain resolution request packets will travel from 10.0.0.5 through the router at 10.0.0.1 and continue to the ISP gateway and its nameserver at 192.168.1.1.
• Responses will never be sent back. The ISP gateway does not know how to route packets to the 10.0.0.5 device. If it stupidly tries to use its default route -- up towards the Internet -- the next router at your ISP will drop the packet. Normally, it should drop the packet and not try to send a packet to any RFC 1918 address via the Internet.

The router can receive resolution packets from the ISP gateway, because the ISP gateway knows how to reach it at 192.168.1.254.

[Head_on_a_Stick]

Originally Posted by beiroot

The problem with my ISP router is that it's a typical consumer product (S@GEM) given by the ISP. It has the web-interface set up by ISP with hardly any settings. And I know there's a linux inside I could control if was given access

Have you tried telnet(1)?

My ISP-supplied router doesn't have a user interface toggle for everything but the telnet(1) interface does

[jggimi]

A "S@GEM" router may be a device from http://www.sagemcom.com. Their broadband gateway devices seem to be capable of adding static routes. http://setuprouter.com/router/sagem/...1416-large.htm

Most importantly, their web portal offers support.

http://www.sagemcom.com/contact/

[beiroot]

Guys,
Unfortunately it's not a typical consumer Sagemcom router. It's called Sagemcom Orange Funbox (how lame is that?) and it's 100% branded for Orange (inside and outside).

btw... before I typed S@gem, but they have something called Sagemcom F@st - that's where the @ came from. It's a ~different type of adsl modems. The Orange Funbox is probably based on something from this series.
Sometimes wires in my brain do funny things

jggimi, I know, every cheap router nowadays has an option to add static routes and I would do it if I had one. But this one is strongly branded and the normal router options are just not there. I guess I just need to invest some cash and switch to something normal.

Are there any obsd-friendly hardware with built-in adsl modems? I don't mean like a normal PC, but rather something like alix, apu, soerkis etc. None of the mentioned have - I've checked it.

[jggimi]

To my understanding, there are ADSL modems with PCI and miniPCI connectors. But I doubt that any are supported by OpenBSD.

The most common practice is to configure an external ADSL modem in "bridge" mode, so that it operates as a pass-through device rather than as a router.

[shep]

When my ISP supplied modem died an early death, I replaced it with a simple, cheap Netis ADSL2 modem and flashed my old Linksys wrt54gV8 router with DD-wrt. When OpenBSD-current began to provide wireless "N" support, I bought a Trendnet TEW-732BR ($11USD refurbished and still available) and flashed it with OpenWRT.

My ISP uses PPPoE and would either sell you a modem/router for about $100 or rent the same for $10/month. The modem came pre-configured, and from a security standpoint was set up for remote management - hopefully by the ISP only.

I was able to find/obtain my ISP settings along with the username/password for PPPoE. I bought a cheap $20 modem and had two options for setup. The first was is described by @Jggimi above - put the modem in bridge mode and enter the PPPoE settings in the router. Both the modem and router are on the same network. The second option, was to enter the PPPoE settings in the modem and use the routing capabilities of the modem to create a subnet. Both worked for me and I settled on the subnet option.

I'm happy with my setup and, now several month old, it has already paid for itself.

You have a myriad of options but many of them hinge on the protocol and settings that your ISP uses.

From the standpoint of an OpenBSD router, one of the things I considered were wired, OpenBSD setups that you linked in the first post and then subneting a cheap wireless access point to the router.

There is also a nascent FreeBSD router project and NetBSD has early/incomplete support for some embedded MIPS devices.

[shep]

I did find a website, with screenshots, that describes connecting a third party modem/router combination to Orange here. This example suggests that the VCI=0, VPI=38 with a PPPoA protocol. The Netis BS4201, that I used, had these options and I suspect that most third party modems do also.
Note that the VCI, VPI and encapsulation settings are Country and ISP specific.
Netgear guide w/ table of ISP settings by provider and Country.

Edit: The linked article notes that the MTU needs to be 1492 but gives the wrong rationale. The maximal packet size for PPPoE is 1492.

Originally Posted by beiroot

Ok, I try not to eat from the bowl of copypasta and understand the whole thing so I write pf rules on my own

I love the book of pf I think it's great


I am currently using openbsd 6.0 with pf as my router/firewall. I originally had it hooked up to an adsl router, and it was NOT configured to use bridge mode. I have since switched to a cable modem, and it is also NOT configured to bridge mode. In fact, I made no changes at all to any of the config files when I changed form adsl to cable. I just plugged my openbsd router to the new modem with the network cable and everything just worked. I am using an apu2c4 with a wle200nx. Everything works fine.

I'll be more than happy to post any of my config files here so you can see how I have things set up.

[beiroot]

Thanks for your replies, but what you suggest is buying something extra and I promised myself I will not spend a single $ to get everything working. I'm trying to work with what I have.

So far I'm double nat-ing what jggimi confirmed, can be problematic. My next experiment is external adsl router on usb - Orange gave them way before the current Sagems. Anyway, this can be fun

amphibious - i'd more than happy to see your configs.
Btw, I'm digging into "the book of pf", so expect more questions

Originally Posted by beiroot

amphibious - i'd more than happy to see your configs.
Btw, I'm digging into "the book of pf", so expect more questions

sure thing.

Give me till tomorrow, and I will start posting them up for you. Please tell me which files you are interested in specific.

[beiroot]

Originally Posted by amphibious

Give me till tomorrow, and I will start posting them up for you. Please tell me which files you are interested in specific.

Could you present your pf.conf for a start?

Originally Posted by amphibious

I originally had it hooked up to an adsl router, and it was NOT configured to use bridge mode. I have since switched to a cable modem, and it is also NOT configured to bridge mode.

You used a subnet and a double nat?

Originally Posted by jggimi

The most common practice is to configure an external ADSL modem in "bridge" mode, so that it operates as a pass-through device rather than as a router.

by external you mean like a USB modem or a lan modem?

Originally Posted by shep

The first was is described by @Jggimi above - put the modem in bridge mode and enter the PPPoE settings in the router. Both the modem and router are on the same network.

jggimi & shep - could you provide more details on the ADSL modem/router bridge with OBSD firewall/router? I don't think I get the difference between that and simple:

ADSL Modem+router [192.168.1.1] <-> [192.168.1.254] OBSD router+firewall

... and later on routing the packets to different subnets. But that requires a double nat as we discussed earlier.

Originally Posted by shep

The second option, was to enter the PPPoE settings in the modem and use the routing capabilities of the modem to create a subnet.

What you mean here is to create a static route from the modem to the internal network? Am I correct?

[shep]

What you mean here is to create a static route from the modem to the internal network? Am I correct?

.
Essentially yes. You can Daisy-Chain routers. On my setup I told my ADSL Modem+router(192.168.1.1) to offer a very narrow range of DHCP addresses on its single RJ-45 nic: 192.168.2.1 to 192.168.2.1. I then set my SOHO wireless router to use a Static IP address of 192.168.2.1. My work station and network printer have static IP's 192.168.2.2 and 192.168.2.3 respectively. The SOHO hands out a limited number of DHCP addresses to my wireless devices starting at 192.168.2.5 and ending at 192.168.2.12.

I'm commenting without knowing how much flexibility your branded "Orange" modem router provides. If it is really limited, there are 3rd party ADSL modems that are reported to work with your ISP. I understand wanting to use the hardware you have, but if it is too restrictive, there are low cost options. I purchased a new ADSL modem for $20 USD + $7 Shipping that has a very low current draw 0.5millamps at 5 Volts. My ISP rents modem/routers for $10month, Coming up on 5 months I'm money ahead.

Edit: Clarified that my modem has a single nic as described by jggimi's example that follows.

[jggimi]

Originally Posted by beiroot

...by external you mean like a USB modem or a lan modem?

To be clear, I was thinking of a modem which has an Ethernet connection to serve the local network. Keep this in mind for my two examples below.

jggimi & shep - could you provide more details on the ADSL modem/router bridge ...

When the gateway device is operated as a router, it will provide the local network with a subnet from RFC 1918, such as addresses in the 192.168/16 or 10/8 address blocks. These are private network addresses that are never routed over the Internet, so NAT must be used.

Let us pretend that the your Internet IP address is 203.0.133.17. Let us also pretend that the gateway device is acting as a NAT router, and provides both DHCPd services and a local RFC 1918 network: 192.168.1/24. None of your devices in the local network are "on" the Internet directly, all of their traffic in and out to the Internet share the single 203.0.133.17 address, which is actually assigned to the gateway device.

Now, let's switch that gateway device from being a router to being a bridge.

The ISP's Internet connection is now directly available. In bridge mode, we would use a single Ethernet connection between the gateway device and a local router -- such as OpenBSD. The IP address on that externally facing NIC of the local router would be 203.0.113.17. Because, in bridge mode, the connection to the Internet does not go through NAT, it is the actual Internet address served by the ISP. The gateway device is no longer a router making any decisions -- it is merely passing packets along between the ISP's media and the local network media.

sorry it took a while to reply. I have been away.

here is my pf.conf file

nt_if="{ athn0 em1 em2 }"

set block-policy drop
set loginterface egress
set skip on lo0

match in all scrub (no-df random-id max-mss 1440)
match out on egress inet from !(egress:network) to any nat-to (egress:0)

block all
pass out quick inet
pass in on $int_if inet
pass in quick on egress inet proto tcp from any to any port 22
pass in on egress inet proto tcp from any to (egress) port 22
pass in on egress inet proto tcp from any to (egress) port 80
pass in on egress inet proto tcp from any to (egress) port 443

I have several ports open for services.

Mine is not configured as nat. It's cofigured as a router/firewall. I don't have another server. my apu2 is just hooked up to my cable modem. Take a look at the following link I found. It can be very helpful in setting things up,

https://github.com/gonzalo-/apu2_openbsd

let me know what other files you may want to see.

[beiroot]

Originally Posted by shep

.
Essentially yes. You can Daisy-Chain routers. On my setup I told my ADSL Modem+router(192.168.1.1) to offer a very narrow range of DHCP addresses on its single RJ-45 nic: 192.168.2.1 to 192.168.2.1. I then set my SOHO wireless router to use a Static IP address of 192.168.2.1.

Wait! What? How does that even work? Two devices have the same IP address, one static, one dhcp? You forced them to bridge? Can you do that? But, they still have private addresses, so I guess your ADSL+router is the main router+firewall in your castle?

jggimi, thank you again for a clear explanation, you Sir, are a hero!

amphibious, thanks for the config and link, I'll analyze it and give feedback as soon as I can.

-----
I decided to play a little with my network architecture, so instead of the Orange adsl+router I'll try old usb adsl modem - just to configure a usb network device and test bridge mode. Wonder how that'll work out.

In the long run it seems that a little investment is inevitable, so what do you think would be the best shot? ADSL modem with bridging capability?