DaemonForums  

Go Back   DaemonForums > Miscellaneous > Programming

Programming C, bash, Python, Perl, PHP, Java, you name it.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 9th February 2014
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default Simplifying pflog file parsing

Last week I decided it was about time to write a Perl script for parsing /var/log/pflog files.

If you use logging in the pf.conf ruleset, the logged packets are written to this file in a modified tcpdump PCAP format.

You can read this file with something like # tcpdump -tttt -nr /var/log/pflog | less or save it in a text file with # tcpdump -tttt -nr /var/log/pflog > ./pflog.txt.

The output looks like:
Code:
#   date(1)    time(2)       ty  source(4)      destination(5)    header info (6)
# 2014-01-31 16:14:30.938665 IP 80.25.124.114 > 1.2.22.222: ICMP echo request, id 0, seq 0, length 64
# 2014-01-31 16:16:35.262293 IP 180.188.194.9.55459 > 1.2.22.222.25: Flags [S], seq 1118489574, win 14600, options [mss 1460,sackOK,TS[|tcp]>
To parse this with Perl I came up with the following subroutine using a long regular expression:
Code:
        if ( /^                         # at beginning of line
            (\d{4}-\d{2}-\d{2})         # capture date yyyy-mm-dd
            \s+                         # one or more spaces
            (\d{2}:\d{2}:\d{2}\.\d+)    # hh:mm:sec.fractional
            \s+
            (\w+)                       # traffic type i.e. IP
            \s+
            (\d+\.\d+.\d+\.\d+          # source IP
                (?:                     # non-capture grouping start
                \.\d+                   # a '.' and one of more digits as port number
                )?                      # end of optional port number
            )                           # end capture
            \s+
            >
            \s+
            (\d+\.\d+.\d+\.\d+          # destination IP address
                (?:                     # non-capture grouping start
                \.\d+                   # optional port number
                )?                      # end of optional port number
            )                           # end capture
            :\s+
            (.+)                        # packet header info
            $                           # up to end of line
            /x                          # allow comments and whitespace

        )
After two or three tries it worked. Although it looks complicated, it actually is quite straightforward.
The only tricky thing was to use the non-capturing grouping (?: ........ ) to match an optional port number.
tcpdump prefixes the port number with a period and appends it to the IP address. In 180.188.194.9.55459 > 1.2.22.222.25 the source IP is 180.188.194.9 using source port 55459, while the destination IP is 1.2.22.222 with port 25.

After a few days, after a walk with the dog, I found a much simpler method. Well actually two.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 9th February 2014 at 06:33 PM.
Reply With Quote
  #2   (View Single Post)  
Old 9th February 2014
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default Simplification nr 1

Code:
#   date(1)    time(2)       ty  source(4)      destination(5)    header info (6)
# 2014-01-31 16:14:38.108331 IP 80.34.103.251 > 1.2.22.222: ICMP echo request [snip]
                                                                                                      ^
With the split function I separate the long line on the colon followed by the space into two parts.
A second [b]split] on a space or blank then separates the first part into the fields that I am interested in.

Code:
sub read_simple {
    my ($leading, $info);
    my ($date, $time, $type, $source, $direction, $dest);
    while (<DATA>) {
        ($leading, $info) = split(/: /);
        print "Leading: ", $leading, "\n";
        print "Info   : ", $info,    "\n";
        ($date,  $time, $type, $source, $direction, $dest) = split( / /, $leading);
        show_raw($date,  $time, $type, $source, $dest, $info);
        export( $date,  $time, $type, $source, $dest, $info);
    }
}
The following output demonstrates this two stages approach:
  1. Stage 1
    Code:
    Leading: 2014-01-31 16:14:30.938665 IP 80.25.124.114 > 1.2.22.222
    Info   : ICMP echo request, id 0, seq 0, length 64
  2. Stage 2
    Code:
    Date             : 2014-01-31
    Time             : 16:14:30.938665
    Type             : IP
    Source IP        : 80.25.124.114
    Destination IP   : 1.2.22.222
    Info             : ICMP echo request, id 0, seq 0, length 64
    
    =============================
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 9th February 2014 at 05:58 PM.
Reply With Quote
  #3   (View Single Post)  
Old 9th February 2014
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default Simplification nr 2

According the documention of the split function you can limit the number of fields that will be separated or split. From http://perldoc.perl.org/functions/split.html :
Quote:
split /PATTERN/,EXPR,LIMIT

[snip]

If LIMIT is specified and positive, it represents the maximum number of fields into which the EXPR may be split; in other words, LIMIT is one greater than the maximum number of times EXPR may be split.
Looking at a sample line:
Code:
#   date(1)    time(2)       ty  source(4)      destination(5)    header info (6)
# 2014-01-31 16:14:30.938665 IP 80.25.124.114 > 1.2.22.222: ICMP echo request, id 0, seq 0, length 64
      1            2         3       4        5       6      7
So in this case we would need a limit of 7. By throwing away field nr 6, the '>', we thus have a one stage split approach:

Code:
sub read_simple {
    my ($leading, $info);
    my ($date, $time, $type, $source, $direction, $dest);
    while (<DATA>) {
        ($date,  $time, $type, $source, $direction, $dest, $info) = split( / /, $_, 7);
        show_raw($date,  $time, $type, $source, $dest, $info);
        #export( $date,  $time, $type, $source, $dest, $info);
    }
}
A complete version:
Code:
#!/usr/bin/perl

use warnings;
use strict;

my $NULL = '\N'; # MySQL null value

#   date(1)    time(2)       ty  source(4)      destination(5)    header info (6)
# 2014-01-31 16:14:30.938665 IP 80.25.124.114 > 1.2.22.222: ICMP echo request, id 0, seq 0, length 64
# 2014-01-31 16:16:35.262293 IP 180.188.194.9.55459 > 1.2.22.222.25: Flags [S], seq 1118489574, win 14600, options [mss 1460,sackOK,TS[|tcp]>

sub show_raw {
    my ($date,  $time, $type, $source, $dest, $info) = @_;
    print STDERR <<END;

Date             : $date
Time             : $time
Type             : $type
Source IP        : $source
Destination IP   : $dest
Info             : $info
=============================
END
}


sub read_simple {
    my ($leading, $info);
    my ($date, $time, $type, $source, $direction, $dest);
    while (<DATA>) {
        ($date,  $time, $type, $source, $direction, $dest, $info) = split( / /, $_, 7);
        show_raw($date,  $time, $type, $source, $dest, $info);
        #export( $date,  $time, $type, $source,        $info);
    }
}

read_simple();

__END__
2014-01-31 16:14:30.938665 IP 80.25.124.114 > 1.2.22.222: ICMP echo request, id 0, seq 0, length 64
2014-01-31 16:16:35.262293 IP 180.188.194.9.55459 > 1.2.22.222.25: Flags [S], seq 1118489574, win 14600, options [mss 1460,sackOK,TS[|tcp]>
2014-01-31 16:16:38.260924 IP 180.188.194.9.55459 > 1.2.22.222.25: Flags [S], seq 1118489574, win 14600, options [mss 1460,sackOK,TS[|tcp]>
The output:
Code:
$ simpler-short.pl 2>&1| less

Date             : 2014-01-31
Time             : 16:14:30.938665
Type             : IP
Source IP        : 80.25.124.114
Destination IP   : 1.2.22.222:
Info             : ICMP echo request, id 0, seq 0, length 64

=============================

Date             : 2014-01-31
Time             : 16:16:35.262293
Type             : IP
Source IP        : 180.188.194.9.55459
Destination IP   : 1.2.22.222.25:
Info             : Flags [S], seq 1118489574, win 14600, options [mss 1460,sackOK,TS[|tcp]>

=============================
Conclusion: a regular expression approach to pflog file parsing is not needed at all. At least not for what I was trying to accomplish.
Why did I think I had to go that way?
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 9th February 2014 at 06:35 PM.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Keeping your /var/log/pflog file clean and managable J65nko Guides 3 8th February 2014 11:07 PM
Log Parsing Not Working plexter OpenBSD Packages and Ports 9 16th July 2011 03:37 PM
Parsing emails with 'awk' and 'perl' J65nko Guides 1 24th February 2011 03:34 AM
Why PFLOG can't LOG anything????? chamnanpol FreeBSD General 1 18th June 2008 07:09 PM


All times are GMT. The time now is 01:01 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick