sshd logging - can we get the ssh command?

[Mantazz]

My FreeBSD server at home is periodically subjected to distributed hack attempts (which inevitable fail for various reasons). It is not unusual to see these involve over 200 unique IP addresses in a single day. I find these attempts to be little more than annoying, and the distributed nature seems to make it rather meaningless to report them or do much of anything else proactive or reactive for them.

However I have been wondering how my poor little server at home ever came to be subjected to this to begin with. I host only my own web pages, and thy are so insignificant that the main page on said server isn't even indexed by google.

Of course my server could be accessed over ssh via two different methods of calling by address - either by name or by numeric address. The name is rather obscure (via dyndns.org) so the odds of someone guessing it at random are rather small. I suspect it is more likely that someone did a scan on port 22 over a great range of IP addresses and found mine to be open.

Is there any way to confirm this? I would like sshd, if possible, to tell me who accesses my server via the command

ssh myserver.mydomain.youcantguessthis.org

as opposed to

ssh 123.234.231.132

Does the ssh daemon know the difference? Is there any way for it to know the difference and log it somewhere? I don't even care what password is provided as the distributed hacks have so far always provided only invalid usernames or usernames that are not allowed to log in via ssh anyways.

[anomie]

Originally Posted by Mantazz

My FreeBSD server at home is periodically subjected to distributed hack attempts (which inevitable fail for various reasons). It is not unusual to see these involve over 200 unique IP addresses in a single day.

I've experienced the same on several 'net-facing servers.

Originally Posted by Mantazz

I suspect it is more likely that someone did a scan on port 22 over a great range of IP addresses and found mine to be open.

That's highly likely.

Originally Posted by Mantazz

Is there any way to confirm this? I would like sshd, if possible, to tell me who accesses my server via the command

ssh myserver.mydomain.youcantguessthis.org

as opposed to

ssh 123.234.231.132

AFAIK, this is not possible. Layer 3/4 communication knows nothing about a hostname, and the only reason the sshd daemon (eventually) knows the client hostname is because it does a reverse lookup.

[BSDfan666]

This is just an Internet nuisance, as long as your own setup is secure.. these attempts will remain simply log noise, which your HTTP server likely gets a fair share of.

Public key authentication only, use an alternative port if you want.. etc.

[Mantazz]

Originally Posted by anomie

AFAIK, this is not possible. Layer 3/4 communication knows nothing about a hostname, and the only reason the sshd daemon (eventually) knows the client hostname is because it does a reverse lookup.

Just to make sure that I am understanding your reply correctly - I am trying to figure out if these bots are accessing my system by its name or by its IP address. I already have the IP address of the system trying to get in, sshd grabs that automatically as we all know.

I rather suspected that this information wasn't retained or passed anywhere, that sshd knows only where the system on the other end is coming from and not how it issued the ssh command. And of course I rather doubt I could talk the bot-masters into installing a new ssh client on their zombies so that I could have that information

[anomie]

Originally Posted by Mantazz

I rather suspected that this information wasn't retained or passed anywhere, that sshd knows only where the system on the other end is coming from and not how it issued the ssh command.

Right -- sshd does not know whether the client connected to its host by hostname or IP.

[jggimi]

I think I understand where your confusion comes from. Domain names are part of URLs, which is sent from the browser to the webserver. But URLs are not part of an SSH connection. And there are plenty of other Internet applications that do not use URL/URIs.

Here's how DNS works, in general, for Internet applications. If URLs are used, they're passed in the first data packet after a TCP connection is established. That doesn't happen for ssh:

1. The ssh client program (any of them .... OpenSSH's ssh(1), putty, or a commerical ssh client) is provided a domain name by the user.
2. The client program makes a resolution request of the client's OS. When the OS is *BSD, the resolver is generally configured by /etc/resolv.conf. See the gethostbyname(3) man page for details.
3. If the resolution requires DNS, the resolver makes a DNS request to the primary DNS server. This is a relatively quick UDP "conversation" between the client's OS and the DNS server. The client OS submits the query with the supplied domain name, and the DNS server replies with the IP address.
4. The client application makes a TCP connection to the resolved IP address, using the port supplied by the user or the client application (port 22 is the default "listening" port for sshd servers).

So your sshd server never sees the domain name.

[phoenix]

Computers only know about IP addresses, and only connect via IP addresses. Period.

Humans know about hostnames and prefer to use symbolic hostnames. Hence, DNS was born, to allow us simple humans to use nice names for servers. All the underlying communication, though, happens using just IPs.

Some protocols, like HTTP, allow the remote hostname to be included, to allow for things like virtualhosts (multiple unique hostnames all pointing to a single IP). SSH is not one of these protocols.

[windependence]

Welcome to the world of the internet. Like has been stated, all this is just "noise". They have no target specifically, they are just looking for weak security to exploit. It does no good to try to block these as there are so many of them and a good number of these are spoofing perfectly good IP addresses so in some cases you would actually be doing yourself harm by blocking legitimate traffic.

The best thing you can do if you want to take the load off your server is to set up a good firewall not on your server but separately so that CPU cycles are not being used to deny access. I run pfsense boxes in front of all my servers and it's a wonderful setup. Even my Windoze admins can use the easy web GUI and I can still access the box via CLI if I want to.

Don't lose any sleep over these "attacks" as they aren't directed at you per se, and as long as they aren't getting in, you'll be fine.

-Tim

[Mantazz]

Originally Posted by windependence

Welcome to the world of the internet. Like has been stated, all this is just "noise". They have no target specifically, they are just looking for weak security to exploit.

I never really figured they had any particular target, I was just curious as to how they found my system. I figure they could bump into my system one of two ways:

Crawling through the web, using some sort of bot system to check every host they can find for open ssh ports to try. I would expect if this was the case that the attempts would come by "ssh myobscurehostname.youwontguessthis.org".

Or, crawling the internet, looking for open ssh ports on any system they can get a response from. I would expect if this was the case that the attempts would come by "ssh 123.45.67.89".

Being as my web server is likely in the bottom .001% of the internet in terms of popularity (maybe 2 or 3 unique hits per day) I figure the second is more likely. I also figure that the bot-masters are likely smart enough to know that ssh and httpd are not necessarily employed together in all cases.

Though ultimately this is just a question for my own sake. I don't expect that it would in any way help to resolve the situation.

It does no good to try to block these as there are so many of them and a good number of these are spoofing perfectly good IP addresses so in some cases you would actually be doing yourself harm by blocking legitimate traffic.

I haven't bothered doing anything in particular to respond to these distrubuted attacks. I also view them essentially as noise. I monitor them and mine the data occasionally, but I don't do anything proactive or reactive for them beyond the obvious (no remote root login, very short list of allowed users with strong passwords, etc...)

The best thing you can do if you want to take the load off your server is to set up a good firewall not on your server but separately so that CPU cycles are not being used to deny access. I run pfsense boxes in front of all my servers and it's a wonderful setup. Even my Windoze admins can use the easy web GUI and I can still access the box via CLI if I want to.

CPU cycles are another thing I'm not particularly concerned about on this system. As I said this webserver is just a hobby at home, serving very few pages of very little importance. Right now the only response to the distributed attempts is to let sshd reject them, and log the date, time, username attempted, and IP address.

Don't lose any sleep over these "attacks" as they aren't directed at you per se, and as long as they aren't getting in, you'll be fine.

-Tim

Not to be arrogant, but unless they change their strategy they will never get in. So I certainly don't stay up at night worrying about this. If they developed a strategy to find valid usernames then I might be more concerned. But even when they did an A-Z "phone book" style attack they did not once hit on a valid user name. And their earlier dictionary attempts on root of course failed for reasons described above.

So to reiterate, I'm not worried right now. I'm just curious as to how they found my system to begin with. I know that of course people have scanned the internet for open ports for years (a win2k box I had with cygwin sshd was once found in less than a half hour), so that is what I figure the most likely way that my server was found.

[phoenix]

Originally Posted by Mantazz

Crawling through the web, using some sort of bot system to check every host they can find for open ssh ports to try. I would expect if this was the case that the attempts would come by "ssh myobscurehostname.youwontguessthis.org".

This is rarely, if ever, done.

Or, crawling the internet, looking for open ssh ports on any system they can get a response from. I would expect if this was the case that the attempts would come by "ssh 123.45.67.89".

This is usually what they do. To see how easy it is, try the following on your network (use the correct subnet, of course): $ nmap -p 22 192.168.0.0/24

That will search every IP from 192.168.0.0 through 192.168.0.255 for hosts with port 22 open. Takes maybe 3 seconds, most of which is just displaying the info. The actual scan takes milliseconds.

Now expand that out to use big blocks of IPs, and wrap it in a script that connects to open SSH ports, and you see what the script kiddies are doing.

It has nothing to do with hostnames and everything to do with IPs.

[jggimi]

An analogy: a thief wanders through the neighborhood, trying every front door on every house, hoping that one of the doors will be unlocked. No special tools or skills are required.

[anomie]

And the thief only sees: 812 Main St., 814 Main St., etc. He does not see Mr. Jones's house, Ms. Sanchez's house, etc. (the "resolved" addresses).

[Mantazz]

Originally Posted by jggimi

An analogy: a thief wanders through the neighborhood, trying every front door on every house, hoping that one of the doors will be unlocked. No special tools or skills are required.

Though FWIW, I would say that the distributed hacks that I and so many others have been subjected to are more like a small gang of thieves working together house-by-house, each thief trying one entry (say door, window, garage, etc) multiple times before moving to the next house.

I suspect others have reported similar things to this before, but my logs looked like this not too long ago:

Apr 12 18:30:05 nfsbox sshd[79901]: error: PAM: authentication error for illegal user amora from server.eshops.lt
Apr 12 18:31:09 nfsbox sshd[79904]: error: PAM: authentication error for illegal user amora from a1-grsph1-006.tosa.pl
Apr 12 18:33:13 nfsbox sshd[79922]: error: PAM: authentication error for illegal user amorina from 195.66.185.185
Apr 12 18:34:14 nfsbox sshd[79925]: error: PAM: authentication error for illegal user amorina from s112.silver.fastwebserver.de
Apr 12 18:35:30 nfsbox sshd[79931]: error: PAM: authentication error for illegal user amorina from 221.130.177.154
Apr 12 18:36:18 nfsbox sshd[79934]: error: PAM: authentication error for illegal user amorina from 85.17.184.11
Apr 12 18:37:25 nfsbox sshd[79937]: error: PAM: authentication error for illegal user amorina from 190.5.228.134
Apr 12 18:38:32 nfsbox sshd[79940]: error: PAM: authentication error for illegal user amory from berryx.homedns.org
Apr 12 18:39:27 nfsbox sshd[79943]: error: PAM: authentication error for illegal user amory from 208.89.208.193

[mwatkins]

Originally Posted by Mantazz

Or, crawling the internet, looking for open ssh ports on any system they can get a response from. I would expect if this was the case that the attempts would come by "ssh 123.45.67.89".

I can't add anything new to this except to reiterate that this is normal behaviour and to suggest you consider implementing "pf" - info abounds including the "sticky thread" in this subforum on brute force ssh attacks.

Originally Posted by Mantazz

(a win2k box I had with cygwin sshd was once found in less than a half hour), so that is what I figure the most likely way that my server was found.

Earlier today I brought up a new virtual server running FreeBSD; within several hours auth.log reports the first attempt at breaking in, this machine from Columbia.

On another machine that has been up for some time my "ssh-offenders" table auto-populated from the pf firewall has several dozen entries already over the last week - I clear it out one in a while.

Bottom line: use public-keys for access; disable PermitRootLogin in /etc/ssh/sshd_config, and consider using PF. The latter is a good exercise even if it don't feel blocking the offenders is necessary, because you'll have gained some knowledge and will have a working packet filter config running and be able to extend it when something more serious than random brute force ssh attempts shows up.

Oh yeah... "don't worry, be happy".

There are other security fish to fry.

[Mantazz]

I can't add anything new to this except to reiterate that this is normal behaviour

I apologize if I was not clear when I said earlier that I know others see the same activity on their boxes. I don't for a moment suspect that there is anything particularly interesting about my box; indeed even I consider it rather dull, just another FreeBSD box up 24/7 with a cable modem connection. Not running anything particularly sexy; a pretty basic installation with Apache2 and a semi-functional mysql / PHP install.

There are other security fish to fry.

Which is why I am not investing a great deal of time and energy into this situation.

Really, I am interested in it more from an "informatics" standpoint:

Where are the systems that are being used for this?
How many times does a single system try in a given time frame?
How do the attempts per unit time vary over time?
How much deviation is there between different names or systems?
How does the logic change over time (regarding attempts / name, attempts / address, attempts / unit time)?
What factors are common between the systems who make the most attempts?

And of course
How did my system end up on the list of targets for these (which we have already addressed here)?

But these questions are all for my own interest and nothing else. I don't expect that I would in any way be able to stop the botnets by answering those questions. I know that there are more pressing issues in the world in general and even in the worlds of FreeBSD or openssh security.

I'm a scientist. I find data to be interesting. When my own research (which is not related to this in the least) has data-less days I try to find other topics where there is data that I find interesting.

[J65nko]

Some of your questions are answered in http://www.aptitudetechnology.com/wh...rute_Force.pdf and http://www.securityfocus.com/infocus/1876

[mwatkins]

From the second article J65nko cited:

This equates to roughly one attack and approximately 300 login attempts per day, on average. Some attackers are very serious about performing attacks, executing hundreds of login attempts in a session.

That describes "low average to average" activity these days in my experience, although except for one box which sits outside my firewall, I'm blissfully unaware of the ssh scanners now. Having more readable auth.log's is a nice side benefit of automatically blocking them at the firewall.

[ephemera]

Originally Posted by J65nko

http://www.aptitudetechnology.com/wh...rute_Force.pdf

source of scans and source of logins is particularly interesting.