Have you looked at the pflog with tcpdump? are ssh packets getting blocked? is sshd listening on the private address? you could confirm this using
nc(1) or by using
fstat(8)/
netstat(8).
If DNS is not properly configured on your network that can cause a noticeable delay connecting to SSH.. a solution to this is setting
UseDNS to no in
sshd_config(5), I believe this may be the case as your pf rules don't indicate you pass in remote ssh connections.
So, at this point all I can give is conjecture.. please let us know how it works out.