DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD Ports and Packages

FreeBSD Ports and Packages Installation and upgrading of ports and packages on FreeBSD.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 15th September 2008
Bruco Bruco is offline
Fdisk Soldier
 
Join Date: May 2008
Location: Kalamazoo, MI, USA
Posts: 61
Thanked 1 Time in 1 Post
Default Suggestions for Web Traffic Logging?

What I'm looking to do is log http traffic. Pretty simple stuff. I don't actually need blocking capabilities, just logging. This will be in a test environment (meaning my house) and later I might see about implementing it at a small branch at work.

At home I have a little Buffalo router running DD-WRT, and I think I can mirror traffic to a port on the router connected to a NIC on my FreeBSD box in promiscuous mode. Now, obviously from there I could run Wireshark with some filters on it and all that, but I want data that's pretty to look at with DNS resolved and all that if possible!

Alternatively, the DD-WRT software will use syslog to send data to the FreeBSD box (thought I don't QUITE have it working yet) and if there was an app that would peel out just what I want from that, it would work to. And I'm not looking for much - source IP, destination IP (resolved if possible), port 80, 8080, 443, etc.

A full dump of network traffic would be fine too, if I have a way to filter it and can easily see just what I want. I suppose I'm looking for sort of an open source Websense - but like I said, I don't need blocking capabilities.

Any suggestions?

Thanks.
Reply With Quote
  #2   (View Single Post)  
Old 15th September 2008
anomie's Avatar
anomie anomie is offline
Local
 
Join Date: Apr 2008
Location: Texas
Posts: 446
Thanked 69 Times in 46 Posts
Default

Check out awstats and visitor. (Both will do the trick, but also probably provide way more info than you need.)

---

Oops, that is visitors (plural).
__________________
Kill your t.v.

Last edited by anomie; 15th September 2008 at 09:52 PM.
Reply With Quote
  #3   (View Single Post)  
Old 15th September 2008
TerryP's Avatar
TerryP TerryP is offline
Arp Constable
 
Join Date: May 2008
Location: USofA
Posts: 1,547
Thanked 112 Times in 104 Posts
Default

I've used awstats often through cpanel, it's good enough if all you want to find out is where your bandwidth surges are generally going (e.g. x,y,z ip's via http through php scripts, peaks at a,b,c hours, of the day, etc)

For more then that, I usually poke about log files with Perl.
__________________
My Journal

Thou shalt check the array bounds of all strings (indeed, all arrays), for surely where thou typest ``foo'' someone someday shall type ``supercalifragilisticexpialidocious''.
Reply With Quote
  #4   (View Single Post)  
Old 15th September 2008
Bruco Bruco is offline
Fdisk Soldier
 
Join Date: May 2008
Location: Kalamazoo, MI, USA
Posts: 61
Thanked 1 Time in 1 Post
Default

Thanks for the suggestions - but do both of those do what I think they do, which is organize and present log files? I'm looking to actually log outgoing requests from my network, potentially with packet-sniffing or analyzing syslog logs, and have the information presented in a fairly easy-to-read format.

Maybe these apps do that, at first glance I didn't think so. I'll keep reading.
Reply With Quote
  #5   (View Single Post)  
Old 15th September 2008
anomie's Avatar
anomie anomie is offline
Local
 
Join Date: Apr 2008
Location: Texas
Posts: 446
Thanked 69 Times in 46 Posts
Default

I completely misunderstood. I thought you were looking for an apache log analyzer.

You want to capture and analyze outbound network traffic to http (tcp 80), right?
__________________
Kill your t.v.
Reply With Quote
  #6   (View Single Post)  
Old 15th September 2008
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,148
Thanked 182 Times in 149 Posts
Default

You also could use the Squid proxy/web cache. From http://en.wikipedia.org/wiki/Squid_(software) :
Quote:
Because it is also a proxy (i.e. it behaves like a client on behalf of the real client), it can provide some anonymity and security. However, it also can introduce significant privacy concerns as it can log a lot of data including URLs requested, the exact date and time, the name and version of the requester's web browser and operating system, and the referer.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #7   (View Single Post)  
Old 16th September 2008
Bruco Bruco is offline
Fdisk Soldier
 
Join Date: May 2008
Location: Kalamazoo, MI, USA
Posts: 61
Thanked 1 Time in 1 Post
Default

Quote:
Originally Posted by anomie View Post
I completely misunderstood. I thought you were looking for an apache log analyzer.

You want to capture and analyze outbound network traffic to http (tcp 80), right?
Correct, sir.

I'll check out Squid - I haven't read much, but is it possible to mirror traffic to it without routing ALL traffic through it? I'll read further...

Thanks again.
Reply With Quote
  #8   (View Single Post)  
Old 17th September 2008
chavez243 chavez243 is offline
Fdisk Soldier
 
Join Date: May 2008
Location: Leamington, ON
Posts: 50
Thanked 2 Times in 2 Posts
Default

Quote:
Originally Posted by Bruco View Post
Correct, sir.

I'll check out Squid - I haven't read much, but is it possible to mirror traffic to it without routing ALL traffic through it? I'll read further...

Thanks again.
if you don't want squid for it actual proxy capabilities, you might be better off doing this another way. What is it you are trying to accomplish?

You could use a passive tap or a mirrorred switch port to dump outbound traffic to a sniffer (tcpdump,wireshark etc.) and look at the port 80 traffic that way. It's hard to advise you without know what your end goal is.
Reply With Quote
  #9   (View Single Post)  
Old 17th September 2008
Bruco Bruco is offline
Fdisk Soldier
 
Join Date: May 2008
Location: Kalamazoo, MI, USA
Posts: 61
Thanked 1 Time in 1 Post
Default

Well, you've nailed it - that IS my goal. What (if any) are the ramifications of running Wireshark continuously? I know I could put filters on it to only see port 80 traffic, but it's still processing all those packets. Is that going to be too memory-intensive? And how "readable" are the results to someone that isn't as technical?

I did try out Squid last night on a fresh FreeBSD 7.0 install in a virtual machine. I set it up and then used it as the proxy server for my workstation. It worked really well. I also installed SARG to do log file analysis, and it basically gives me exactly what I want - easy to read HTML reports that reflect what sites were visited from what IP.

So, no, obviously I don't need a proxy - and doing things this way necessitates adding read-only proxy server settings for all browsers on all workstations so users can't change them (which I can do, not that big a deal).

But am I going to get the nice reporting options, with resolved IPs, organization by workstation IP or user ID, etc, by using a packet sniffer? I suppose if there are log analyzers for Wireshark that can do the same thing as SARG does for Squid logs, and if I can run Wireshark continuously without issue, it could work.
Reply With Quote
Old 17th September 2008
DutchDaemon's Avatar
DutchDaemon DutchDaemon is offline
Real Name: Ben
Spam Refugee
 
Join Date: Jul 2008
Location: Rotterdam, The Netherlands
Posts: 337
Thanked 32 Times in 30 Posts
Default

You can run Squid as a transparent proxy using a redirection (using pf, ipf, ipfw, whichever you like).

Wireshark has quite a few statistical report generators built right in (well, the X11 Wireshark, not the command-line one, of course), including a three-part HTTP analyzer. But it doesn't give what you want combined in one neat window, no.
Reply With Quote
Old 17th September 2008
chavez243 chavez243 is offline
Fdisk Soldier
 
Join Date: May 2008
Location: Leamington, ON
Posts: 50
Thanked 2 Times in 2 Posts
Default

you can use a PAC (proxy auto-config) file to control which users get assigned to the proxy (if you use static IPs).

squid/SARG might be the way to go - running a proxy is a great security enhancement and gives you total control over users web behaviour.

Just be aware of the privacy ramifications of generating verbose reports based on squid logs. I use calamaris with squid but I keep the results to myself and just silently make ACL adjustments when I see any red flags.
Reply With Quote
Old 17th September 2008
Bruco Bruco is offline
Fdisk Soldier
 
Join Date: May 2008
Location: Kalamazoo, MI, USA
Posts: 61
Thanked 1 Time in 1 Post
Default

Quote:
Originally Posted by DutchDaemon View Post
You can run Squid as a transparent proxy using a redirection (using pf, ipf, ipfw, whichever you like).
I imagine you mean by making a FreeBSD box the gateway for the clients I could do that? My router I'm testing with runs iptables, I imagine I could do it with that.

Yeah, I think Squid+SARG is gonna be perfect for what I need - I mean, I don't need a proxy, obviously, but I don't think it's going to hurt. Thanks for the warning on privacy ramifications, too!
Reply With Quote
Old 17th September 2008
DutchDaemon's Avatar
DutchDaemon DutchDaemon is offline
Real Name: Ben
Spam Refugee
 
Join Date: Jul 2008
Location: Rotterdam, The Netherlands
Posts: 337
Thanked 32 Times in 30 Posts
Default

Squid doesn't have to be (on) a gateway, as long as all traffic passes through it So it could be on a bridge between LAN and gateway (and of course redirection can be done using iptables just as well).
Reply With Quote
Old 17th September 2008
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,148
Thanked 182 Times in 149 Posts
Default

Why don't you just use squid? Especially since it has all the reporting you want.

Originally Wireshark was Etherreal, which a couple of ways ago, was kicked out of the OpenBSD ports systems because of insecure programming. Are you sure you want to run this?
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
Old 18th September 2008
Bruco Bruco is offline
Fdisk Soldier
 
Join Date: May 2008
Location: Kalamazoo, MI, USA
Posts: 61
Thanked 1 Time in 1 Post
Default

Ah, yes, it could be a bridge, couldn't it?

Yep, I plan to go with squid, along with sarg. It'll do what I need. Thanks, everybody!
Reply With Quote
Old 18th September 2008
DutchDaemon's Avatar
DutchDaemon DutchDaemon is offline
Real Name: Ben
Spam Refugee
 
Join Date: Jul 2008
Location: Rotterdam, The Netherlands
Posts: 337
Thanked 32 Times in 30 Posts
Default

If you're using Squid/SARG you might as well add Calamaris to the mix. It's pretty straightforward and minimal, but it'll give you a nice overview on one page. Just run it right before or right after SARG, and then move/zip your logfile of the day/week/month.
Reply With Quote
Old 18th September 2008
hydra's Avatar
hydra hydra is offline
Port Guard
 
Join Date: May 2008
Location: Slovakia (Europe)
Posts: 41
Thanked 0 Times in 0 Posts
Default

I can really recommend Squid with SARG, I've been using it for half a year now, works great.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
See what process is generating DNS traffic? Bruco FreeBSD General 3 2nd July 2009 05:57 PM
sshd logging - can we get the ssh command? Mantazz FreeBSD Security 17 23rd May 2009 08:34 AM
pflog not logging. bsdnewbie999 OpenBSD General 9 13th March 2009 11:19 PM
PF Blocking VPN Traffic plexter OpenBSD Security 6 23rd January 2009 05:25 PM
Dynamic Traffic Shaping LordZ OpenBSD Security 6 19th January 2009 04:30 PM


All times are GMT. The time now is 06:26 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick