Non-standard SSH ports have always been a poor security measure and more of a pain in dealing with client configuration than they're worth. Public key auth FTW! PasswordAuthentication No is always one of the first edits I make to sshd_config.