It is best practice to use two factors for authentication. Most commonly, this is something you have (key) and something you know (password or passphase). The ssh keys can be used in combination with the latter.
Passwords are weak because they are short. See
http://xkcd.com/936/