|
OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below. |
|
Thread Tools | Display Modes |
|
|||
[Solved] Cannot access from internet on 2nd ISP
Hi,
I have a OpenBSD 4.9 with 3 NICs. For testing purposes pf is disabled. I can connect to services (eg. ssh) from internet via 1st ISP (shh 78.w.x.y, ping works) but i cant connect via 2nd ISP (ssh 178.w.x.y, ping dosent work). I would like to reach my server from two ISP at the same time. Here is my config: 1st NIC (gateway for lan): Code:
fxp0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:1b:21:22:f3:82 priority: 0 media: Ethernet autoselect (100baseTX full-duplex) status: active inet6 fe80::21b:21ff:fe22:f382%fxp0 prefixlen 64 scopeid 0x3 inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255 Gateway: 87.w.x.y Code:
pppoe0: flags=8951<UP,POINTOPOINT,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1492 priority: 0 dev: bge0 state: session sid: 0x1504 PADI retries: 1 PADR retries: 0 time: 21:47:47 sppp: phase network authproto pap authname "xxxxxxxxxxxx@y.pl" groups: pppoe egress status: active inet6 fe80::21b:21ff:feb5:5899%pppoe0 -> prefixlen 64 scopeid 0x6 inet 78.w.x.y --> 87.w.x.y netmask 0xffffffff Gateway: 178.w.x.254 Code:
em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:1b:xx:xx:xx:xx description: 2/2 priority: 0 media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause) status: active inet6 fe80::21b:21ff:feb5:5899%em0 prefixlen 64 scopeid 0x1 inet 178.w.x.y netmask 0xffffff00 broadcast 178.w.x.255 Code:
Routing tables Internet: Destination Gateway Flags Refs Use Mtu Prio Iface default 87.w.x.y UGS 6 1010372 - 8 pppoe0 87.w.x.y 78.w.x.y UH 1 0 - 4 pppoe0 127/8 127.0.0.1 UGRS 0 0 33200 8 lo0 127.0.0.1 127.0.0.1 UH 0 511 33200 4 lo0 178.w.x/24 link#1 UC 8 0 - 4 em0 178.w.x.6 00:25:9c:xx:xx:xx UHLc 0 0 - 4 em0 178.w.x.34 00:25:9c:xx:xx:xx UHLc 0 0 - 4 em0 178.w.x.64 00:25:9c:xx:xx:xx UHLc 0 0 - 4 em0 178.w.x.65 00:25:9c:xx:xx:xx UHLc 0 0 - 4 em0 178.w.x.116 00:25:9c:xx:xx:xx UHLc 0 0 - 4 em0 178.w.x.139 68:7f:74:xx:xx:xx UHLc 0 0 - 4 em0 178.w.x.140 68:7f:74:xx:xx:xx UHLc 0 0 - 4 em0 178.w.x.254 00:1b:21:xx:xx:xx UHLc 0 0 - 4 em0 192.168.1/24 link#3 UC 51 0 - 4 fxp0 224/4 127.0.0.1 URS 0 0 33200 8 lo0 Code:
em0 at pci3 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: apic 2 int 18 (irq 3), address 00:1b:xx:xx:xx:xx fxp0 at pci6 dev 0 function 0 "Intel 8255x" rev 0x0c, i82550: apic 2 int 21 (irq 11), address 00:1b:xx:xx:xx:xx inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4 bge0 at pci5 dev 0 function 0 "Broadcom BCM5722" rev 0x00, BCM5755 C0 (0xa200): apic 2 int 17 (irq 10), address 00:22:xx:xx:xx:xx brgphy0 at bge0 phy 1: BCM5722 10/100/1000baseT PHY, rev. 0 Code:
sysctl net.inet.ip.forwarding=1 Last edited by n4p1; 30th September 2011 at 11:56 PM. Reason: solved |
|
|||
No, that's not entirely right.. sshd can indeed be listening on both interfaces and clients should be able to connect over both, check your configuration.
You can use pf's route-to to set up outbound loadbalancing for clients, in a round-robin fashion. http://www.openbsd.org/faq/pf/pools.html#outgoing |
|
||||
The current routing configuration is the problem, as far as I can tell. Your subnet attached to em0 (178.x.x.x/255) does not have any routes defined.
Example: An inbound packet from 1.2.3.4 to 178.x.x.x will get a response from your default route and its IP address, 78.x.x.x. That return packet will be dropped by 1.2.3.4, as it was unsolicited. You want to define multipath routing. FAQ 6.14 may be a good place to start. |
|
|||
Thanks for helping me.
I was trying to set multipath route with two default gateways. But then I cant reach ssh from both interfaces. When i try to connect, I always get connection to only one, the second one was unreachable. I was trying also route-to statement in pf.conf without success. And traffic from lan goes always from both interfaces. So the next solution was only route-to with one default gateway (I want all traffic go via pppoe0 ($ext_if1) and pass in (on $ext_if2) only 3 services via em0 - one rdr-to rule, ssh and vpn). It will be best choice for my needs. After reading a lot of faqs and manuals i try to use tags in pf.conf: Code:
ext_if1="pppoe0" ext_gw1="87.x.y.z" ext_if2="em0" ext_gw2="178.x.y.z" int_if="fxp0" table <net_access> { 192.168.1.0/24, !192.168.1.10 } # SCRUBBING SECTION # match on $ext_if1 scrub (max-mss 1440) # priority # UPLOAD altq on $ext_if1 priq bandwidth 500Kb queue {up_std, up_prio} queue up_prio priority 7 queue up_std priority 1 priq(default) # NAT match out on $ext_if1 from 192.168.1.0/24 to any nat-to ($ext_if1) #match out on $ext_if2 from 192.168.1.0/24 to any nat-to ($ext_if2) # Default policy block in log all set block-policy drop # loopback set skip on lo # WWW from LAN pass in log on $int_if proto tcp from <net_access> to any port 80 # RDP redirect - Windows 2008 pass in on $ext_if1 proto tcp from <rdp_direct_access> to any port 9131 rdr-to 192.168.1.50 port 3389 tag IF1 pass in on $ext_if2 proto tcp from <rdp_direct_access> to any port 9131 rdr-to 192.168.1.50 port 3389 tag IF2 # ssh pass in on $int_if proto tcp from ($int_if:network) to any port 22 ####### SSH via LAN pass in on $ext_if1 proto tcp from any to any port 22 queue up_prio tag IF1 ####### SSH via $ext_if1 pass in on $ext_if2 proto tcp from any to any port 22 tag IF2 ####### SSH via $ext_if2 # openvpn pass in on $ext_if1 proto tcp from any to any port 367 tag IF1 ####### OpenVPN via $ext_if1 pass in on $ext_if2 proto tcp from any to any port 367 tag IF2 ####### OpenVPN via $ext_if2 # route-to # pass out log on { $ext_if1, $ext_if2, $int_if } from any to {!192.168.1.0/24, !10.8.0.0/24 } route-to ($ext_if1 $ext_gw1) pass out log on { $ext_if1, $ext_if2 } route-to ($ext_if1 $ext_gw1) tagged IF1 pass out log on { $ext_if1, $ext_if2 } route-to ($ext_if2 $ext_gw2) tagged IF2 But it works like that: 1. $ext_if1 port 9131 - connection is ok 2. $ext_if2 port 9131 - cant connect 3. port 22 is reached from lan and $ext_if1 but not from $ext_if2. It is so frustrating and i don't have idea what do do next. Could anyone point me right direction? Last edited by n4p1; 15th September 2011 at 08:41 AM. |
|
||||
Quote:
|
|
||||
While we await more information from you, I may be able to find time this weekend to run some tests. I've got a topology in mind, which tests a local "server" with external users. If the test were reversed; where the local system is the "user", it would be nearly the same; this topology just includes port forwarding along with NAT.
Please let me know if you would be interested in this type of problem recreation / resolution, before I invest the time and effort: --- Five systems: An "internet user", two "ISPs", a "router", and a "server". Four networks: an "Internet", between each "ISP" and the "router", and between the "router" and the "server." Three tests: static provisioning, DHCP provisioning, and a NATted "server". |
|
||||||
Quote:
Quote:
Quote:
hostname.em0 !/sbin/route add -mpath default 178.x.y.z hostname.pppoe0 !/sbin/route add -mpath default 87.x.y.z After that (working from my home) I can only ssh via em0, pppoe0 was unreachable. Although in my pf.conf i had: pass in on em0 proto tcp from any to any port 22 pass in on pppoe0 proto tcp from any to any port 22 That was weird. Quote:
Quote:
But when I have only one default route to my if1 and Im trying ssh from outside via if2 I see incoming connection in tcpdump on that interface but nothing happen. Btw. when mpath was enabled I can connect to outside services from OpenBSD box without problem (ex. www, ping etc). Some packets goes via em0 and some via pppoe0. Ex. when I connect to my home ssh box it was always from em0. Quote:
|
|
||||
Quote:
Code:
[client] --> SYN packet --> [IF2] [client] <-- SYN-ACK packet <-- [IF2] [client] --> ACK packet --> [IF2] Code:
[client] --> SYN packet --> [IF2] [client] <-- SYN-ACK packet <-- [IF1] [client] ??? |
|
||||
Quote:
|
|
||||
I had no difficulty setting up the lab this evening, here, and running a set of tests. I used 4.9-release systems.
My "router" had the following configuration changes. The changes to sysctl.conf are shown as a patch against the 4.9-release code. I added the following files: hostname.em0 (connection to ISP #1) Code:
inet 10.0.1.4/24 !route add -mpath default 10.0.1.1 Code:
inet 10.0.2.4/24 !route add -mpath default 10.0.2.2 Code:
inet 10.0.3.4/24 Code:
Index: sysctl.conf =================================================================== RCS file: /cvs/src/etc/sysctl.conf,v retrieving revision 1.49 diff -u -r1.49 sysctl.conf --- sysctl.conf 16 Feb 2011 10:37:45 -0000 1.49 +++ sysctl.conf 16 Sep 2011 23:03:45 -0000 @@ -4,9 +4,9 @@ # boot time. See sysctl(3) and sysctl(8) for more information on # the many available variables. # -#net.inet.ip.forwarding=1 # 1=Permit forwarding (routing) of IPv4 packets +net.inet.ip.forwarding=1 # 1=Permit forwarding (routing) of IPv4 packets #net.inet.ip.mforwarding=1 # 1=Permit forwarding (routing) of IPv4 multicast packets -#net.inet.ip.multipath=1 # 1=Enable IP multipath routing +net.inet.ip.multipath=1 # 1=Enable IP multipath routing #net.inet.icmp.rediraccept=1 # 1=Accept ICMP redirects #net.inet6.icmp6.rediraccept=0 # 0=Don't accept IPv6 ICMP redirects #net.inet6.ip6.forwarding=1 # 1=Permit forwarding (routing) of IPv6 packets Code:
# w 7:15PM up 21 mins, 4 users, load averages: 0.28, 0.21, 0.11 USER TTY FROM LOGIN@ IDLE WHAT root 00 - 6:55PM 0 w root p0 10.0.2.2 6:57PM 16 -ksh root p1 10.0.0.3 6:58PM 15 -ksh root p2 10.0.1.1 7:00PM 0 -ksh |
|
||||
NAT testing is complete.
I was able to both initiate connections outbound, and port forward to the inbound "server" with the following pf.conf. The first line NATs all outbound traffic from the internal network according to it's appropriate trunk, however it gets routed. The second line used port forwarding to expose a service, in this case sshd(8), from the internal server. Code:
match out from em2:network to any nat-to {em0,em1} match in proto tcp from any to any port 2222 rdr-to 10.0.3.5 port 22 pass log all I discovered an error I'd made while setting up the lab environment. I'd neglected to add routes between the ISPs "customer" networks (10.0.1, 10.0.2) using the "internet" network (10.0.0). I discovered this by using tcpdump(8). If you are unable to recreate the same success I've had, please consider using tcpdump and watching traffic flow (or not flow) across your NICs. |
|
||||
I just re-read your post.
Quote:
Code:
# route -n show -inet Routing tables Internet: Destination Gateway Flags Refs Use Mtu Prio Iface default 10.0.1.1 UGSP 0 0 - 8 em0 default 10.0.2.2 UGSP 0 0 - 8 em1 10.0.1/24 link#1 UC 1 0 - 4 em0 10.0.1.1 link#1 UHLc 1 0 - 4 em0 10.0.2/24 link#2 UC 1 0 - 4 em1 10.0.2.2 link#2 UHLc 1 0 - 4 em1 10.0.3/24 link#3 UC 0 0 - 4 em2 127/8 127.0.0.1 UGRS 0 0 33200 8 lo0 127.0.0.1 127.0.0.1 UH 1 0 33200 4 lo0 224/4 127.0.0.1 URS 0 0 33200 8 lo0 # |
|
|||
I check and do everything what you say and this dosent work....
pf disabled: pfctl -d mpath enabled (in sysctl.conf) and route added via hostname.if. OpenBSD rebooted. One more time: pfctl -d. Then: Code:
[15:02:43][root@xxx:~]# route -n show -inet Routing tables Internet: Destination Gateway Flags Refs Use Mtu Prio Iface default 178.x.y.z UGSP 3 1263 - 8 em0 default 87.x.y.z UGSP 1 212 - 8 pppoe0 10.8.0/24 10.8.0.2 UGS 0 0 - 8 tun0 10.8.0.2 10.8.0.1 UH 1 0 - 4 tun0 87.105.104.1 78.w.x.y UH 0 0 - 4 pppoe0 127/8 127.0.0.1 UGRS 0 0 33200 8 lo0 127.0.0.1 127.0.0.1 UH 1 7 33200 4 lo0 178.x.y/24 link#1 UC 1 0 - 4 em0 178.x.y.z 00:1b:21:0b:45:6c UHLc 1 0 - 4 em0 192.168.1/24 link#3 UC 5 0 - 4 fxp0 192.168.1.186 00:26:9e:78:2b:55 UHLc 1 3 - 4 fxp0 192.168.1.217 00:10:a7:22:ee:c1 UHLc 0 1018 - 4 fxp0 192.168.1.234 00:26:18:ef:86:47 UHLc 0 76 - 4 fxp0 192.168.1.248 00:24:7e:dd:e0:c8 UHLc 1 471 - 4 fxp0 192.168.1.255 link#3 UHLc 1 50 - 4 fxp0 224/4 127.0.0.1 URS 0 0 33200 8 lo0 [15:03:02][root@xxx:~]# Code:
Connection to ssh (pppoe0) from internet: [15:07:20][root@xxx:~]# tcpdump -i pppoe0 port 50022 tcpdump: listening on pppoe0, link-type PPP_ETHER 15:07:37.081892 79.x.y.z.1112 > 78.w.x.y.50022: S 0:0(0) win 64240 <mss 1354,nop,nop,sackOK> (DF) 15:07:40.009122 79.x.y.z.1112 > 78.w.x.y.50022: S 0:0(0) win 64240 <mss 1354,nop,nop,sackOK> (DF) 15:07:45.070270 79.x.y.z.1112 > 78.w.x.y.50022: R 1:1(0) win 0 15:07:45.872714 79.x.y.z.1112 > 78.w.x.y.50022: S 0:0(0) win 64240 <mss 1354,nop,nop,sackOK> (DF) [15:06:32][root@xxx:~]# tcpdump -i em0 port 50022 tcpdump: listening on em0, link-type EN10MB 15:07:37.081928 78.w.x.y.50022 > 79.x.y.z.1112: S 4021557824:4021557824(0) ack 1 win 16384 <mss 1452,nop,nop,sackOK> (DF) 15:07:40.009147 78.w.x.y.50022 > 79.x.y.z.1112: S 4021557824:4021557824(0) ack 1 win 16384 <mss 1452,nop,nop,sackOK> (DF) 15:07:40.081000 78.w.x.y.50022 > 79.x.y.z.1112: S 4021557824:4021557824(0) ack 1 win 16384 <mss 1452,nop,nop,sackOK> (DF) 15:07:45.872741 78.w.x.y.50022 > 79.x.y.z.1112: S 397263719:397263719(0) ack 1 win 16384 <mss 1452,nop,nop,sackOK> (DF) 15:07:48.869635 78.w.x.y.50022 > 79.x.y.z.1112: S 397263719:397263719(0) ack 1 win 16384 <mss 1452,nop,nop,sackOK> (DF) 15:07:54.882299 78.w.x.y.50022 > 79.x.y.z.1112: S 397263719:397263719(0) ack 1 win 16384 <mss 1452,nop,nop,sackOK> (DF) 15:08:06.907544 78.w.x.y.50022 > 79.x.y.z.1112: S 397263719:397263719(0) ack 1 win 16384 <mss 1452,nop,nop,sackOK> (DF) Code:
Connection to ssh (em0) from internet: [15:08:52][root@xxx:~]# tcpdump -i pppoe0 port 50022 tcpdump: listening on pppoe0, link-type PPP_ETHER [15:08:52][root@xxx:~]# tcpdump -i em0 port 50022 tcpdump: listening on em0, link-type EN10MB 15:09:02.576896 79.x.y.z.1113 > 178.w.x.y.50022: S 651286537:651286537(0) win 64240 <mss 1354,nop,nop,sackOK> (DF) 15:09:02.576950 178.w.x.y.50022 > 79.x.y.z.1113: S 1761386290:1761386290(0) ack 651286538 win 16384 <mss 1460,nop,nop,sackOK> (DF) 15:09:02.901824 79.x.y.z.1113 > 178.w.x.y.50022: . ack 1 win 64240 (DF) 15:09:02.914818 178.w.x.y.50022 > 79.x.y.z.1113: P 1:22(21) ack 1 win 17602 (DF) 15:09:04.966413 79.x.y.z.1113 > 178.w.x.y.50022: P 1:29(28) ack 22 win 64219 (DF) 15:09:04.968072 178.w.x.y.50022 > 79.x.y.z.1113: P 22:878(856) ack 29 win 17602 (DF) 15:09:04.982650 79.x.y.z.1113 > 178.w.x.y.50022: P 29:541(512) ack 22 win 64219 (DF) 15:09:05.005865 79.x.y.z.1113 > 178.w.x.y.50022: P 541:669(128) ack 22 win 64219 (DF) 15:09:05.005896 178.w.x.y.50022 > 79.x.y.z.1113: . ack 669 win 17474 (DF) 15:09:05.120993 79.x.y.z.1113 > 178.w.x.y.50022: P 669:685(16) ack 878 win 63363 (DF) 15:09:05.128536 178.w.x.y.50022 > 79.x.y.z.1113: P 878:1414(536) ack 685 win 17602 (DF) 15:09:05.462415 79.x.y.z.1113 > 178.w.x.y.50022: . ack 1414 win 64240 (DF) 15:09:05.472540 79.x.y.z.1113 > 178.w.x.y.50022: P 685:1197(512) ack 1414 win 64240 (DF) 15:09:05.472734 79.x.y.z.1113 > 178.w.x.y.50022: P 1197:1213(16) ack 1414 win 64240 (DF) 15:09:05.472764 178.w.x.y.50022 > 79.x.y.z.1113: . ack 1213 win 17586 (DF) 15:09:05.571596 178.w.x.y.50022 > 79.x.y.z.1113: P 1414:2518(1104) ack 1213 win 17602 (DF) 15:09:05.854983 79.x.y.z.1113 > 178.w.x.y.50022: . ack 2518 win 63136 (DF) 15:09:05.932382 79.x.y.z.1113 > 178.w.x.y.50022: P 1213:1229(16) ack 2518 win 63136 (DF) 15:09:05.932562 79.x.y.z.1113 > 178.w.x.y.50022: P 1229:1281(52) ack 2518 win 63136 (DF) 15:09:05.932595 178.w.x.y.50022 > 79.x.y.z.1113: . ack 1281 win 17550 (DF) 15:09:05.932724 178.w.x.y.50022 > 79.x.y.z.1113: P 2518:2570(52) ack 1281 win 17602 (DF) 15:09:06.251843 79.x.y.z.1113 > 178.w.x.y.50022: . ack 2570 win 63084 (DF) 79.x.y.z - My home ip 178.w.x.y - OpenBSD em0 78.w.x.y - OpenBSD pppoe0 Also there is no mygate file: Code:
[15:16:19][root@zgkim:~]# ls /etc/mygate ls: /etc/mygate: No such file or directory |
|
||||
Thank you. This has got to be an issue with pppoe, then, because it is a virtual interface.
A quick look through the misc@ archives found http://marc.info/?l=openbsd-misc&m=126902993416220&w=2 |
|
|||
So it’s a bug or it’s a feature?
|
|
||||
I happened across a linked article mentioned this week at the OpenBSD Journal, about using virtual routing domains -- and the article suggested the possibility of using them to connect with multiple ISPs, though it did not show a PF ruleset that might be applied in the solution.
This might be a way to circumvent your apparent pppoe restriction. The article page provides a contact link for the author, as well as a comments section. http://www.packetmischief.ca/2011/09...routing-table/ Last edited by jggimi; 28th September 2011 at 02:54 PM. Reason: Author link at the top, comments section at the bottom. |
|
||||
I seem to remember having a problem with a PPPOE adsl connection and it didn't work properly until I adjusted the MTU... because there was an MTU size problem a few packets actually would make it through if they were small enough, but most everything would be fragmented and then (for whatever reason) dropped.
If I get access to that host again in the next week or so I'll get the working configuration for it and my notes.
__________________
Network Firefighter |
|
||||
Quote:
I have been trying to setup a similar config on a SPARC ULTRA1 but things are not working 100%. I am able to ping and do name resolution on the server. I can ssh from the lan interface(laptop with LinuxMint) to the server. I'm not able to resolve domain names from same but I can ping yahoos IPs. BTW, I am not using ppoe interface I would greatly appreciate you help. Thank you, |
|
||||
The $ indicates a normal user shell. Normal users can display sysctl values, and that command displays only.
# indicates root shell commands. To set a sysctl temporarily, use the command from a root shell, with an = and the value to be set, per the sysctl(8) man page. To set a value permanently edit /etc/sysctl.conf and reboot. |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
4.8 -> 4.9 and internet access stops | thefronny | OpenBSD Security | 4 | 14th August 2011 11:47 AM |
PF cannot access Internet from internal network | gpatrick | OpenBSD Security | 3 | 29th August 2010 10:59 PM |
could not look up internet address for .lan | idefix | FreeBSD General | 2 | 31st January 2009 02:22 PM |
Internet Access Problem OpenBSD 4.3 | alcy | OpenBSD General | 3 | 19th September 2008 06:00 PM |
Internet access within jail | Weaseal | FreeBSD General | 5 | 26th June 2008 02:45 PM |