If you use public key encryption, ignore it.
If you use a weak password, worry.
If it annoys you and you want cleaner logs (though not truly more secure server) switch the port number.
As for allow/disallow you can do that from within sshd.config as well
AllowUsers username
or even more restrictive
AllowUsers
username@IP.Number