You ended up in the bruteforce table.
What you should have is
Code:
...
block all
pass quick from <sshadmins>
block quick from <bruteforce>
...
This will allow the sshadmins in even if they end up in the bruteforce table.
Something else you should look into is expiretable (/usr/ports/security/expiretable) to clean out entries that are older than a specified amount of time. That way, if you do get banned you only have to wait so long before you can get in again. I have expiretable cleaning out entries after 30 minutes.