Just a hunch, but you may need to add your login name to /etc/ssh/sshd_config:
Code:
AllowUsers me you
#allows clients to log in only as the local users me or you
That is, if you still have the "Connection refused" problem. That's what I needed to do, at least (on FreeBSD 6).