SSH times out not showing password prompt

[typo3guy]

At our shop we have an OpenBSD firewall with two interfaces. One facing the internet through an ADSL connection and one facing the internal lan. For a month or so we have been using a new fiber connection with fast internet.
But the fiber line has a wifi modem/router iinstead of a regular modem and it has no port forwarding to play with. So we are nat-ing through the router by connecting the external interface on the openbsd firewall to one of the router ports. And it works, but it prevents remote login to the firewall.

So I built a new firewall with OpenBSD 7.4 in a lab environment with two external interfaces and one internal. Each external interface is configured as dhcp clients and each has a connection to an Internet Service Provider (ISP). Interface re0 has a 100Mbs fiber connection, while dc0 has an ADSL 10Mbs connection. The two connections are intended to work independently and not as failover or load balancing setup. The purpose of the ADSL connection is to provide a way to remote login to the firewall and hence the internal lan via ssh. The route inbound through re0 via ssh is "blocked" by the router/modem.

                            ISP 1                 ISP 2
                              |                     |
                        fiber_if (re0) |   | adsl_if (dc0)
                                       |   | 
                                      OpenBSD  
                                      firewall          
                                         |
                                    int_if (fxp0)
                                         |
                                    Internal lan

This is the pf.conf:

# Macros
fiber_if = "re0"         # External interface for fiber
int_if = "fxp0"          # Internal interface
adsl_if = "dc0"          # External interface for adsl 
lan = "192.168.5.0/24"   # Internal network 

# Procedures applied to packets
set optimization normal                         # time for connection rupture
set loginterface $fiber_if                      # stats for extenal interfaces
set block-policy return                         # packets knocking on closed ports are returned
match in all scrub (no-df random-id)            # scrubbing
set skip on lo0                                 # no filtering on loopback

# NAT rules
pass out log quick on $fiber_if from $lan to any nat-to ($fiber_if)

# Filter rules
block in log
pass out quick inet

# Pass rules for external interface with adsl
pass in quick log on $adsl_if proto tcp from any to ($adsl_if) port ssh

# Pass rules for external interface with fast internet
pass in log on $fiber_if proto tcp from any to ($fiber_if) port {22}

# Pass rules for internal interface
pass in on $int_if from $lan
pass out on $int_if to $lan

When connecting to dc0 via ssh from any random address on the internet, tcpdump shows a successful pass in on dc0. On the ssh client, however, the password prompt is missing and it will finally time out.

This is the output for tcpdump -neti pflog0:
tcpdump: WARNING: snaplen raised from 116 to 160
tcpdump: listening on pflog0, link-type PFLOG
rule 2/(match) block in on dc0: 190.203.71.128.52438 > 190.38.244.155.1433: S 1733463736:1733463736(0) win 8192 <mss 1460,nop,nop,sackOK> (DF)
rule 2/(match) block in on re0: 192.168.1.1 > 224.0.0.1: igmp query [tos 0xc0] [ttl 1]
rule 4/(match) pass in on dc0: 45.77.119.194.26094 > 190.38.244.155.22: S 1542149394:1542149394(0) win 64240 <mss 1460,sackOK,timestamp 2295899547 0,nop,wscale 7> (DF)
rule 2/(match) block in on dc0: 35.203.211.7.53202 > 190.38.244.155.45277: S 1871576406:1871576406(0) win 65535 <mss 1460>
rule 2/(match) block in on dc0: 185.224.128.55.56878 > 190.38.244.155.81: S 1079003273:1079003273(0) win 65535

This is the output for ssh -vvv user@host:

OpenSSH_7.9p1 Debian-10+deb10u4, OpenSSL 1.1.1n  15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolve_canonicalize: hostname 190.38.244.155 is address
debug2: ssh_connect_direct
debug1: Connecting to 190.38.244.155 [190.38.244.155] port 22.
debug1: connect to address 190.38.244.155 port 22: Connection timed out
ssh: connect to host 190.38.244.155 port 22: Connection timed out

This is the routing table. Output for route show -inet:

Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            192.168.1.1        UGSP       2     9037     -     8 re0  
default            190.38.224.1       UGSP       0        0     -     8 dc0  
224/4              localhost          URS        0        0 32768     8 lo0  
127/8              localhost          UGRS       0        0 32768     8 lo0  
localhost          localhost          UHhl       1        7 32768     1 lo0  
190.38.224/19      190.38.244.155     UCn        2        0     -     4 dc0  
190.38.224.1       48:2c:d0:6e:1c:6f  UHLch      1      413     -     3 dc0  
190.38.238.141     48:2c:d0:6e:1c:6f  UHLc       0        4     -     3 dc0  
190.38.244.155     00:08:a1:13:e6:34  UHLl       0     2715     -     1 dc0  
190.38.255.255     190.38.244.155     UHb        0        0     -     1 dc0  
192.168.1/24       192.168.1.155      UCn        2     5678     -     4 re0  
192.168.1.1        c0:56:27:d4:ea:46  UHLch      1     3258     -     3 re0  
192.168.1.151      f8:4d:89:77:e4:d4  UHLc       3     2048     -     3 re0  
192.168.1.155      00:06:4f:51:98:b7  UHLl       0      502     -     1 re0  
192.168.1.255      192.168.1.155      UHb        0        0     -     1 re0  
192.168.5/24       192.168.5.50       UCn        1        0     -     4 fxp0 
192.168.5.50       00:04:23:11:7d:ab  UHLl       0       32     -     1 fxp0 
192.168.5.100      00:1e:90:6e:04:c5  UHLc       0       62     -     3 fxp0 
192.168.5.255      192.168.5.50       UHb        0        0     -     1 fxp0

This is what I think is happening. The ssh connection is passed in on the ADSL line (dc0) as shown in rule 4/(match) in the tcpdump, but the reply with the password prompt returns through the first default gateway (re0, the fiber line) and does never reach the ssh client which times out.
Should this be so, how can I correct it?

Please bear with me if you find errors, I'm not a trained network guy. I'm a mechanical engineer that by accident is doing network administration. And please don't assume any knowledge on my side, and explain as plain as possible. Thanks in advance.

[jggimi]

Hello, and welcome!

I think your guess that the return traffic is being mis-routed is a good one. PF has the ability to assist with routing decisions, with options such as route-to or reply-to. From the pf.conf(5) man page, it looks like reply-to might be helpful:

     reply-to  The reply-to option is similar to route-to, but routes packets
               that pass in the opposite direction (replies) to the specified
               address.  Opposite direction is only defined in the context of
               a state entry, and reply-to is useful only in rules that create
               state.  It can be used on systems with multiple paths to the
               internet to ensure that replies to an incoming network
               connection to a particular address are sent using the path
               associated with that address (symmetric routing enforcement).

[J65nko]

I also think it is the return traffic.
Are you able to see and manually add routes to the routing table of the ADSL modem?

I am afraid that is the issue. In my old ADSL setup I had to add a route for the return traffic on the ADSL modem itself.

[jmccue]

See OpenBSD @misc.

Someone was having a similar issue logging in to another system from a MAC. I also ran into to this, I ended up regenerating my keys to use ed25519 instead of RSA. My RSA Keys were created a long time ago.

That corrected the issue for me.

HTH

[typo3guy]

Thanks @jggimi for your reply.
I tried this:

# Pass rules for external interface with adsl
pass in quick log on $adsl_if proto tcp from any to ($adsl_if) port ssh reply-to $adsl_if

It didn't help, status remains as described in the original post.

[jggimi]

Time to inspect traffic with tcpdump(8).

[J65nko]

Your netstat routing table only shows the fibernet gateway is being used (references: 9037).
he default route to the ADSL modem not, number of references is 0:

Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            192.168.1.1        UGSP       2     9037     -     8 re0  
default            190.38.224.1       UGSP       0        0     -     8 dc0

What is the output of ifconfig?
On which interface(s) does ssh LISTEN on?
Remember that in the default configuration ssh listens on all interfaces (0.0.0.0).

#ListenAddress 0.0.0.0
ListenAddress 192.168.222.10

# netstat -an -f inet
netstat -an -f inet           
[snip]
Active Internet connections (including servers)
Proto   Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp          0     44  192.168.222.10.22      192.168.222.241.14416  ESTABLISHED
tcp          0      0  127.0.0.1.8021         *.*                    LISTEN
tcp          0      0  127.0.0.1.25           *.*                    LISTEN
tcp          0      0  192.168.222.10.22      *.*                    LISTEN
[snip]

[typo3guy]

Motivated by the fact that upon rebooting the two default gateways, more often than not, would trade places and that this would render the pf rules useless, mostly the NAT rule, I decided to try virtualization.
At first, I put each interface (both external and the internal) in its own rdomain with its rtable. As a result and with the appropriate pf rules, I could make the ssh remote login on each external interface.
After that, I thought that I could simplify the setup using only two rdomains. I isolated the adsl interface on rdomain 1, leaving all other interfaces on rdomain 0.

cortafuego# cat /etc/hostname.ste0                                                                                   
rdomain 1
inet autoconf

I had to ditch dc0 for ste0. The NIC went bad.

@J65nko ssh is listening on the internal address

#ListenAddress 0.0.0.0
#ListenAddress ::
ListenAddress 192.168.5.50

cortafuego# netstat -an -f inet        
Active Internet connections (including servers)
Proto   Recv-Q Send-Q  Local Address          Foreign Address        IP-Proto
ip           0      0  *.*                    *.*                    1
Active Internet connections (including servers)
Proto   Recv-Q Send-Q  Local Address          Foreign Address        TCP-State
tcp          0     36  192.168.5.50.22        71.6.134.230.52702     CLOSING
tcp          0     44  192.168.5.50.22        192.168.1.151.60379    ESTABLISHED
tcp          0      0  192.168.5.50.22        201.210.101.239.52072  FIN_WAIT_2
tcp          0      0  192.168.5.50.22        192.168.1.151.61541    ESTABLISHED
tcp          0      0  192.168.5.50.22        192.168.1.151.60189    ESTABLISHED
tcp          0      0  127.0.0.1.25           *.*                    LISTEN
tcp          0      0  192.168.5.50.22        *.*                    LISTEN
Active Internet connections (including servers)
Proto   Recv-Q Send-Q  Local Address          Foreign Address       
udp          0      0  192.168.1.109.18895    200.25.3.17.123       
udp          0      0  192.168.1.109.30134    162.159.200.123.123   
udp          0      0  192.168.1.109.18203    162.159.200.123.123   
udp          0      0  *.*                    *.*                   
udp          0      0  192.168.1.109.68       *.*                   
udp          0      0  *.*                    *.*

The PF rules are:

# Macros
fiber_if = "re0"         # External interface for fiber
int_if = "fxp0"          # Internal interface
adsl_if = "ste0"         # External interface for adsl
lan = "192.168.5.0/24"   # Internal network

table <unwanted> persist
block in quick from <unwanted>

# Procedures applied to packets
set optimization normal                         # time for connection rupture
set loginterface $fiber_if                      # stats for extenal interfaces
set block-policy return                         # packets knocking on closed ports are returned
match in all scrub (no-df random-id)            # scrubbing
set skip on lo0                                 # no filtering on loopback

# NAT rules
pass out quick on $fiber_if from $lan to any nat-to ($fiber_if)

# Filter rules
block in log
pass out quick inet

# Pass rules for external interface with adsl
pass in log on $adsl_if proto tcp rdr-to $int_if port ssh flags S/SA keep state (max-src-conn 4, max-src-conn-rate 2/3, overload <unwanted> flush global) rtable 0

# Pass rules for external interface with fast internet
pass in log on $fiber_if proto tcp rdr-to $int_if port ssh rtable 0

# Pass rules for internal interface
pass in log on $int_if from $lan
pass out log on $int_if to $lan

The rules allow me to ssh remote login on any of the three interfaces.
Only thing I don’t like is the landslide of matches I see with tcpdump on ste0.

cortafuego# tcpdump -neti pflog0 action pass 
tcpdump: WARNING: snaplen raised from 116 to 160
tcpdump: listening on pflog0, link-type PFLOG
rule 5/(match) pass in on ste0: 94.102.61.54.47597 > 201.210.18.130.8187: S 415123947:415123947(0) win 65535 <mss 1460>
rule 5/(match) pass in on ste0: 162.216.149.46.57734 > 201.210.18.130.22000: S 647654564:647654564(0) win 65320 <mss 1420,sackOK,timestamp 1075685110 0,nop,wscale 7> (DF)
rule 5/(match) pass in on ste0: 94.102.61.54.37962 > 201.210.18.130.8187: S 557742556:557742556(0) win 64240 <mss 1460,sackOK,timestamp 1061052991 0,nop,wscale 7> (DF)
rule 5/(match) pass in on ste0: 124.239.177.205.52352 > 201.210.18.130.2375: S 2370852556:2370852556(0) win 29200 <mss 1460,sackOK,timestamp 1303396152 0,nop,wscale 7> (DF)
rule 5/(match) pass in on ste0: 162.243.140.49.50019 > 201.210.18.130.81: S 2673828012:2673828012(0) win 65535
rule 5/(match) pass in on ste0: 35.203.210.183.53304 > 201.210.18.130.46439: S 1369554523:1369554523(0) win 65535 <mss 1460>
rule 5/(match) pass in on ste0: 198.235.24.73.55265 > 201.210.18.130.5916: S 2091996449:2091996449(0) win 1024 <mss 1460>

cortafuego# tail -f /var/log/authlog 
Feb  1 17:18:39 cortafuego sshd[35560]: Connection closed by 103.138.109.69 port 57062
Feb  1 17:18:43 cortafuego sshd[33587]: Connection closed by 103.138.109.69 port 58555
Feb  1 17:18:50 cortafuego sshd[66744]: banner exchange: Connection from 35.203.210.16 port 62128: invalid format
Feb  1 17:18:51 cortafuego sshd[82084]: banner exchange: Connection from 35.203.210.16 port 62136: invalid format
Feb  1 17:18:55 cortafuego sshd[68438]: Connection closed by 103.138.109.69 port 62285
Feb  1 17:18:57 cortafuego sshd[85916]: Connection closed by 162.216.149.161 port 52004
Feb  1 17:19:14 cortafuego sshd[14237]: banner exchange: Connection from 71.6.134.230 port 40538: invalid format
Feb  1 17:19:27 cortafuego sshd[20454]: banner exchange: Connection from 162.216.149.214 port 65328: invalid format
Feb  1 17:19:27 cortafuego sshd[52354]: banner exchange: Connection from 162.216.149.214 port 65338: invalid format
Feb  1 17:19:28 cortafuego sshd[19595]: banner exchange: Connection from 162.216.149.99 port 60982: invalid format
Feb  1 17:19:45 cortafuego sshd[56833]: banner exchange: Connection from 198.199.104.98 port 45574: invalid format
Feb  1 17:19:49 cortafuego sshd[28265]: banner exchange: Connection from 35.203.210.194 port 58766: invalid format

I tried an alternate ssh port, but it didn't reduce the number of attempts significantly. Maybe I didn't do it right. I will look into that again.

[J65nko]

With tcpdump you can filter additionally on port 22. That will exclude all non ssh traffic.

Now that you can connect, ssh -vvv should show the reason for the multiple retries.

[typo3guy]

Hello @J76nko. I might not expressed myself correctly, but there are no multiple retries. This is arbitrary, random traffic coming from the internet to the open port 22. I would like to reduce or filter it further, but I will need you to elaborate a bit on how this should be done. Furthermore, what should I be looking for in ssh -vvv to achieve this? Thanks.

[J65nko]

To stop those probes on port 22, you can configure ssh to listen on a different port. You have more than 65000 ports to choose from.

Changing the standard port, does not enhance security. But by selecting a rather high port you will eliminate those probes from polluting your log files. These probes are trying to guess the password. By using pre-shared public keys, you will render them useless.

You also can add certificate authentication. See https://en.m.wikibooks.org/wiki/Open...Authentication

[jggimi]

I pass inbound SSH traffic by selected OS fingerprint. That effectively eliminates the useless log entries.

[frcc]

typo3guy
In addition a simple solution which will reduce the number of probing attacks:
You can filter out foreign countries I.P. address's, by adding a line or so to your pf.conf file. When administrating several severs for my business years ago, that was one of the first things i did as most probing actions were from countries in the pacific rim and elsewhere overseas. These probes were several hundred per day and were eliminated immediately by pf.conf. A simple search of the web will provide you with these address's, when obtained, and placed in a text file they can be referenced in pf.conf. This is not a complete security solution as security is most effective in layers. In addition to what has been stated above, this will work if your system does not need exposure in other countries. (i.e.) a static business ip. or for simple personal use which is exposed to the Web.

Posted back in 2017
Table is much faster than variable.
I block all foreign ip cidr's in pf.conf using table which is large and instantaneous.
i/we use
"block in quick log (all) on fxp0 from !<usip> to any label "foreign"
where <usip> is a .csv file of U.S. based cidr's

Syntax a bit dated i am sure...but the concept is solid..
It cuts down the amount of traffic greatly if you r not interested in a non-us
based cidr.
Edit/Delete Message

[typo3guy]

@J65nko, @jggimi, @frcc Thank you all for your valuable input. I changed the default ssh port, allowed a user in sshd, rate-limited connections and send offenders to a tar-pit-table and limited connections based on country zones. The amount of attempts to brute force the ssh port were substantially reduced.