Suggestions for Web Traffic Logging?

[Bruco]

What I'm looking to do is log http traffic. Pretty simple stuff. I don't actually need blocking capabilities, just logging. This will be in a test environment (meaning my house) and later I might see about implementing it at a small branch at work.

At home I have a little Buffalo router running DD-WRT, and I think I can mirror traffic to a port on the router connected to a NIC on my FreeBSD box in promiscuous mode. Now, obviously from there I could run Wireshark with some filters on it and all that, but I want data that's pretty to look at with DNS resolved and all that if possible!

Alternatively, the DD-WRT software will use syslog to send data to the FreeBSD box (thought I don't QUITE have it working yet) and if there was an app that would peel out just what I want from that, it would work to. And I'm not looking for much - source IP, destination IP (resolved if possible), port 80, 8080, 443, etc.

A full dump of network traffic would be fine too, if I have a way to filter it and can easily see just what I want. I suppose I'm looking for sort of an open source Websense - but like I said, I don't need blocking capabilities.

Any suggestions?

Thanks.

[anomie]

Check out awstats and visitor. (Both will do the trick, but also probably provide way more info than you need.)

---

Oops, that is visitors (plural).

• www/visitors
• www/awstats

[TerryP]

I've used awstats often through cpanel, it's good enough if all you want to find out is where your bandwidth surges are generally going (e.g. x,y,z ip's via http through php scripts, peaks at a,b,c hours, of the day, etc)

For more then that, I usually poke about log files with Perl.

[Bruco]

Thanks for the suggestions - but do both of those do what I think they do, which is organize and present log files? I'm looking to actually log outgoing requests from my network, potentially with packet-sniffing or analyzing syslog logs, and have the information presented in a fairly easy-to-read format.

Maybe these apps do that, at first glance I didn't think so. I'll keep reading.

[anomie]

I completely misunderstood. I thought you were looking for an apache log analyzer.

You want to capture and analyze outbound network traffic to http (tcp 80), right?

[J65nko]

You also could use the Squid proxy/web cache. From http://en.wikipedia.org/wiki/Squid_(software) :

Because it is also a proxy (i.e. it behaves like a client on behalf of the real client), it can provide some anonymity and security. However, it also can introduce significant privacy concerns as it can log a lot of data including URLs requested, the exact date and time, the name and version of the requester's web browser and operating system, and the referer.

[Bruco]

Originally Posted by anomie

I completely misunderstood. I thought you were looking for an apache log analyzer.

You want to capture and analyze outbound network traffic to http (tcp 80), right?

Correct, sir.

I'll check out Squid - I haven't read much, but is it possible to mirror traffic to it without routing ALL traffic through it? I'll read further...

Thanks again.

[chavez243]

Originally Posted by Bruco

Correct, sir.

I'll check out Squid - I haven't read much, but is it possible to mirror traffic to it without routing ALL traffic through it? I'll read further...

Thanks again.

if you don't want squid for it actual proxy capabilities, you might be better off doing this another way. What is it you are trying to accomplish?

You could use a passive tap or a mirrorred switch port to dump outbound traffic to a sniffer (tcpdump,wireshark etc.) and look at the port 80 traffic that way. It's hard to advise you without know what your end goal is.

[Bruco]

Well, you've nailed it - that IS my goal. What (if any) are the ramifications of running Wireshark continuously? I know I could put filters on it to only see port 80 traffic, but it's still processing all those packets. Is that going to be too memory-intensive? And how "readable" are the results to someone that isn't as technical?

I did try out Squid last night on a fresh FreeBSD 7.0 install in a virtual machine. I set it up and then used it as the proxy server for my workstation. It worked really well. I also installed SARG to do log file analysis, and it basically gives me exactly what I want - easy to read HTML reports that reflect what sites were visited from what IP.

So, no, obviously I don't need a proxy - and doing things this way necessitates adding read-only proxy server settings for all browsers on all workstations so users can't change them (which I can do, not that big a deal).

But am I going to get the nice reporting options, with resolved IPs, organization by workstation IP or user ID, etc, by using a packet sniffer? I suppose if there are log analyzers for Wireshark that can do the same thing as SARG does for Squid logs, and if I can run Wireshark continuously without issue, it could work.

[DutchDaemon]

You can run Squid as a transparent proxy using a redirection (using pf, ipf, ipfw, whichever you like).

Wireshark has quite a few statistical report generators built right in (well, the X11 Wireshark, not the command-line one, of course), including a three-part HTTP analyzer. But it doesn't give what you want combined in one neat window, no.

[chavez243]

you can use a PAC (proxy auto-config) file to control which users get assigned to the proxy (if you use static IPs).

squid/SARG might be the way to go - running a proxy is a great security enhancement and gives you total control over users web behaviour.

Just be aware of the privacy ramifications of generating verbose reports based on squid logs. I use calamaris with squid but I keep the results to myself and just silently make ACL adjustments when I see any red flags.

[Bruco]

Originally Posted by DutchDaemon

You can run Squid as a transparent proxy using a redirection (using pf, ipf, ipfw, whichever you like).

I imagine you mean by making a FreeBSD box the gateway for the clients I could do that? My router I'm testing with runs iptables, I imagine I could do it with that.

Yeah, I think Squid+SARG is gonna be perfect for what I need - I mean, I don't need a proxy, obviously, but I don't think it's going to hurt. Thanks for the warning on privacy ramifications, too!

[DutchDaemon]

Squid doesn't have to be (on) a gateway, as long as all traffic passes through it So it could be on a bridge between LAN and gateway (and of course redirection can be done using iptables just as well).

[J65nko]

Why don't you just use squid? Especially since it has all the reporting you want.

Originally Wireshark was Etherreal, which a couple of ways ago, was kicked out of the OpenBSD ports systems because of insecure programming. Are you sure you want to run this?

[Bruco]

Ah, yes, it could be a bridge, couldn't it?

Yep, I plan to go with squid, along with sarg. It'll do what I need. Thanks, everybody!

[DutchDaemon]

If you're using Squid/SARG you might as well add Calamaris to the mix. It's pretty straightforward and minimal, but it'll give you a nice overview on one page. Just run it right before or right after SARG, and then move/zip your logfile of the day/week/month.

[hydra]

I can really recommend Squid with SARG, I've been using it for half a year now, works great.