DaemonForums  

Go Back   DaemonForums > Miscellaneous > Guides

Guides All Guides and HOWTO's.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 29th December 2009
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,125
Default Why tcpdump sometimes drops packets, mangles DNS and shows bad checksums

From http://archive.netbsd.se/?ml=openbsd...-12&m=12145822
Quote:
BPF is implemented as a ring buffer if it overflows it will drop packets.
That's why tcpdump is printing the statistics at the end:
4 packets received by filter
0 packets dropped by kernel

If you tcpdump with a snapsize of 2000 as shown above you will run out of
the default bpf bufsize very quickly since the default is 32k and I guess
you cranked up your tcp buffers to much bigger numbers so that bpf has no
chance to queue the incomming packets, call userland and be done with them
before the 32k buffer overflows. You may want to look into sysctl
net.bpf.bufsize.
For more info about BPF, the Berkely Packet Filter, which is used by tcpdump, see http://en.wikipedia.org/wiki/Berkeley_Packet_Filter
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #2   (View Single Post)  
Old 29th December 2009
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is offline
Real Name: Martin
Tcpdump Spy
 
Join Date: Apr 2008
Location: Netherlands
Posts: 2,243
Default

As a related addition, if you run tcpdump on FreeBSD with the default settings (Possibly also other BSD's, can't recall running into this with OpenBSD though) you will likely run into messages such as:
Code:
tcp 12 [bad hdr length 8 - too short, < 20]
Which can be solved by increasing the snaplen with -s, 256 should be more than enough (Default being 68):
Code:
tcpdump -s 256 -i pflog0
The tcpdump(1) manpage gives an explanation of this option and also explains how it relates to j65nko's post:
Code:
       -s     Snarf  snaplen  bytes  of  data from each packet rather than the
              default of 68 (with SunOSâs NIT, the minimum  is  actually  96).
              68  bytes is adequate for IP, ICMP, TCP and UDP but may truncate
              protocol information from  name  server  and  NFS  packets  (see
              below).   Packets  truncated  because  of a limited snapshot are
              indicated in the output with [|proto], where  proto  is  the
              name of the protocol level at which the truncation has occurred.
              Note that taking larger snapshots both increases the  amount  of
              time it takes to process packets and, effectively, decreases the
              amount of packet buffering.  This may cause packets to be  lost.
              You  should  limit snaplen to the smallest number that will capâ
              ture the protocol information  youâre  interested  in.   Setting
              snaplen  to 0 means use the required length to catch whole packâ
              ets.
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
  #3   (View Single Post)  
Old 30th December 2009
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,125
Default

SNAP LENGTH for DNS requests and answers

If you are debugging DNS requests issued by BIND you need to capture the complete DNS packet of 512 bytes, thus you need -s512

Recently I noticed that a snap length of 512 is not good enough any more.

It used to be that if a name server could not cram an answer in 512 bytes, it would send what fitted in, and indicate the incompleteness of the answer, by setting the truncation bit, a special flag, reserved for this purpose.
Upon noticing this truncation flag, the DNS client then would repeat the DNS query, but this time not using UPD but TCP so the server could send the complete answer over TCP.

This fall back mechanism to TCP is one of the reasons you need to allow port 53 traffic for both UDP and TCP. The second reason: DNS zone transfers use TCP too.

Using TCP is not without costs. Just setting up a TCP connection already takes 3 packets, the TCP handshake, then a packet for the repeated request, another one for the answer, then a few packets for tearing the TCP connection down. It takes a lot of time compared with sending one UDP packet for the request, and the second for the answer.

With IPv6 and DNSSEC mechanisms the articifial 512 byte limit became too restrictive. Nowadays there is an DNS extension, which allows a DNS client to say, "Hey I can receive 4096 bytes long answer over UDP"
See http://en.wikipedia.org/wiki/EDNS

Conclusion: if you want to capture DNS traffic, you really need a snap length of 4096. On some tcpdump versions, the FreeBSD version for example, you can have tcpdump figure out the snap length by using -s0.

Troubles caused by absence of -n flag

To assist in making sense of the it's output, tcpdump by default resolves the host addresses and port numbers to names.
For finding the port names it simply consults "/etc/services". A snippet from that file:
Code:
ssh             22/tcp
ssh             22/udp
telnet          23/tcp
smtp            25/tcp          mail
[snip]
domain          53/tcp          nameserver      # name-domain server
domain          53/udp          nameserver
For converting the hostnames, tcpdump directs reverse DNS lookups to the nameserver(s) specified in the "/etc/resolv.conf" file.
If you happen to debug a nameserver problem and thus are watching port 53 traffic, the tcpdump reverse lookups also show up in your log of dump file. To prevent these extra lookups from interfering you can use the "-n" flag:
Code:
     -n        Do not convert addresses (host addresses, port numbers, etc.)
               to names
Although still issuing extra reverse name lookups, and only mentioned here for completeness sake:
Code:
     -N        Do not print domain name qualification of host names.  For ex-
               ample, if you specify this flag then tcpdump will print ``nic''
               instead of ``nic.ddn.mil''.
Illustrating the two ways of display:
Code:
# tcpdump header output using '-n' option
20:33:44.739528 198.41.0.4.53 > 192.168.222.244.45428

# default tcpdump header output 
20:33:44.739528 a.root-servers.net.domain > vintrax.utp.xnet.45428
Circumventing the extra reverse DNS lookups at capture time

In tracing a DNS issue I thought to be be smart. We all do sometimes, isn't it? I instructed tcpdump to write the packets to file using the "-n" flag.
Code:
tcpdump -ni re0 -s4096 -w file
To analyze output I would use
Code:
tcpdump  -r file
Without any danger of messing up the data written to file, tcpdump could resolve the host addresses to names at analyzing time, not during capture time. That appeared to work nicely.

Code:
# tcpdump -vv -r dns.tcpdump

20:33:44.739528 a.root-servers.net.domain > vintrax.utp.xnet.45428:
[udp sum ok] 7429- q: A? ns1.isc-sns.net. 0/13/16 ns: net. NS
G.GTLD-SERVERS.net., net. NS D.GTLD-SERVERS.net., net. NS
J.GTLD-SERVERS.net., net. NS B.GTLD-SERVERS.net., net. NS
E.GTLD-SERVERS.net., net. NS I.GTLD-SERVERS.net., net. NS
M.GTLD-SERVERS.net., net. NS A.GTLD-SERVERS.net., net. NS
L.GTLD-SERVERS.net., net. NS F.GTLD-SERVERS.net., net. NS
C.GTLD-SERVERS.net., net. NS H.GTLD-SERVERS.net., net. NS
K.GTLD-SERVERS.net. ar:

A.GTLD-SERVERS.net. A a.gtld-servers.net,
A.GTLD-SERVERS.net. AAAA a.gtld-servers.net,
B.GTLD-SERVERS.net. A b.gtld-servers.net,
B.GTLD-SERVERS.net. AAAA b.gtld-servers.net,
C.GTLD-SERVERS.net. A c.gtld-servers.net,
D.GTLD-SERVERS.net. A d.gtld-servers.net,
E.GTLD-SERVERS.net. A e.gtld-servers.net,
F.GTLD-SERVERS.net. A f.gtld-servers.net,
G.GTLD-SERVERS.net. A g.gtld-servers.net,
H.GTLD-SERVERS.net. A h.gtld-servers.net,
I.GTLD-SERVERS.net. A i.gtld-servers.net,
J.GTLD-SERVERS.net. A j.gtld-servers.net,
K.GTLD-SERVERS.net. A k.gtld-servers.net,
L.GTLD-SERVERS.net. A l.gtld-servers.net,
M.GTLD-SERVERS.net. A m.gtld-servers.net,
. OPT UDPsize=4096 (529) (DF) (ttl 244, id 35057, len 557)
Nice isn't it? We see her an answer from 'a.root-servers.net' with source port domain directed to host 'vintrax.utp.xnet' with destination port 45428.

But aren't A records supposed to be IP addresses like this?
Code:
i.gtld-servers.net.     166755  IN      A       192.43.172.30
Yes, they are, but tcpdump in trying to be overly helpful also resolved the IP addresses from the A records in the packet payload.

To see the difference, the same file now decoded with "-n":
Code:
# tcpdump -vvn -r forums-dns.tcpdump

20:33:44.739528 198.41.0.4.53 > 192.168.222.244.45428: [udp sum ok]
7429- q: A? ns1.isc-sns.net. 0/13/16 ns: net. NS G.GTLD-SERVERS.net.,
net. NS D.GTLD-SERVERS.net., net. NS J.GTLD-SERVERS.net., net. NS
B.GTLD-SERVERS.net., net. NS E.GTLD-SERVERS.net., net. NS
I.GTLD-SERVERS.net., net. NS M.GTLD-SERVERS.net., net. NS
A.GTLD-SERVERS.net., net. NS L.GTLD-SERVERS.net., net. NS
F.GTLD-SERVERS.net., net. NS C.GTLD-SERVERS.net., net. NS
H.GTLD-SERVERS.net., net. NS K.GTLD-SERVERS.net. ar:

A.GTLD-SERVERS.net.  A 192.5.6.30,
A.GTLD-SERVERS.net. AAAA 2001:503:a83e::2:30,
B.GTLD-SERVERS.net. A 192.33.14.30,
B.GTLD-SERVERS.net. AAAA 2001:503:231d::2:30,
C.GTLD-SERVERS.net. A 192.26.92.30,
D.GTLD-SERVERS.net. A 192.31.80.30,
E.GTLD-SERVERS.net. A 192.12.94.30,
F.GTLD-SERVERS.net. A 192.35.51.30,
G.GTLD-SERVERS.net. A 192.42.93.30,
H.GTLD-SERVERS.net. A 192.54.112.30,
I.GTLD-SERVERS.net. A 192.43.172.30,
J.GTLD-SERVERS.net. A 192.48.79.30,
K.GTLD-SERVERS.net. A 192.52.178.30,
L.GTLD-SERVERS.net. A 192.41.162.30,
M.GTLD-SERVERS.net. A 192.55.83.30,
. OPT UDPsize=4096 (529) (DF) (ttl 244, id 35057, len 557)
Here we see the answer from 'a.root-servers.net' in it's full glory.

Anyway, happy tcpdump'ing
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #4   (View Single Post)  
Old 3rd January 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,125
Default Tcpdump checksum errors on outgoing packets

Tcpdump checksum errors on outgoing packets

To "fall with the door into the house" (literal translation of a Dutch saying):

Code:
root@hercules[~]tcpdump -vvi re0 -s4096  
tcpdump: listening on re0, link-type EN10MB

07:08:11.939997 hercules.utp.xnet.26487 > mx1.xillhosting.nl.ntp:
[bad udp cksum e4d9!] v4 client strat 0 poll 0 prec 0 dist 0.000000
disp 0.000000 ref (unspec)@0.000000000 orig 0.000000000 rec
-0.000000000 xmt +1710637651.099857769 [tos 0x10] (ttl 64, id 4635,
len 76, bad cksum 38! differs by 1fb2)
You don't encounter bad checksums in NTP packets everyday, but sometimes you do.

Code:
07:08:11.959364 mx1.xillhosting.nl.ntp > hercules.utp.xnet.26487:
[udp sum ok] v4 server strat 2 poll 0 prec -21 dist 0.004638 disp
0.016815 ref chime5.surfnet.nl@3471486924.672977864 orig
1710637651.099857769 rec +1760850040.850035130 xmt +1760850040.850053966
(DF) (ttl 58, id 0, len 76)
Even more stranger, mx1.xillhosting.nl just answers. Aren't packets with erroneous checksums to be silently discarded?

Code:
07:08:12.690406 hercules.utp.xnet.47141 > parmenides.utp.xnet.domain:
[bad udp cksum b20!] 37272+ PTR? 218.91.245.77.in-addr.arpa. (44)
(ttl 64, id 3821, len 72, bad cksum 34! differs by 2e14)
Here again, tcpdump detects a checksum problem on an outgoing packet. Does 'hercules' have a bad day with mathematics today ?

Code:
07:08:13.201840 parmenides.utp.xnet.domain > hercules.utp.xnet.47141:
[udp sum ok] 37272 q: PTR? 218.91.245.77.in-addr.arpa. 1/0/0
218.91.245.77.in-addr.arpa. PTR mx1.xillhosting.nl. (76) (ttl 64,
id 35978, len 104)
The 'parmenides' nameserver seems to have not any problem with 'hercules' request, it just answers.

A diagnostic attempt with netstat -in on hercules does not show any error.
Code:
# netstat -in
Name    Mtu   Network     Address              Ipkts Ierrs    Opkts Oerrs Colls
lo0     33168 <Link>                               0     0        0     0     0
lo0     33168 127/8       127.0.0.1                0     0        0     0     0
lo0     33168 ::1/128     ::1                      0     0        0     0     0
lo0     33168 fe80::%lo0/ fe80::1%lo0              0     0        0     0     0
bge0*   1500  <Link>      00:10:18:00:9f:fd        0     0        0     0     0
re0     1500  <Link>      00:19:db:47:b0:4c     1607     0     1435     0     0
re0     1500  192.168.222 192.168.222.20        1607     0     1435     0     0
re0     1500  fe80::%re0/ fe80::219:dbff:fe     1607     0     1435     0     0
enc0*   1536  <Link>                               0     0        0     0     0
Even netstat -ss reports no checksum related errors, but mentions that the network chip does checksum processing in hardware.

Code:
# netstat -ss
ip:
        1600 total packets received
        1564 packets for this host
        36 packets not forwardable
        1426 packets sent from this host
        1600 input datagrams checksum-processed by hardware
        1426 output datagrams checksum-processed by hardware
        36 multicast packets which we don't join
icmp:
        Input packet histogram:
                echo reply: 3
igmp:
ipencap:
tcp:
        1085 packets sent
                162 data packets (107560 bytes)
                550 ack-only packets (621 delayed)
                213 window update packets
                160 control packets
                1085 packets hardware-checksummed
        1225 packets received
                241 acks (for 107639 bytes)
                74 duplicate acks
                954 packets (1130771 bytes) received in-sequence
                3 completely duplicate packets (0 bytes)
                25 out-of-order packets (10015 bytes)
                6 window update packets
                1225 packets hardware-checksummed
        80 connection requests
        80 connections established (including accepts)
        84 connections closed (including 0 drops)
        321 segments updated rtt (of 322 attempts)
        812 correct data packet header predictions
        5 SACK options sent
udp:
        336 datagrams received
        336 input packets hardware-checksummed
        338 output packets hardware-checksummed
        336 delivered
        338 datagrams output
esp:
ah:
etherip:
ipcomp:
carp:
pfsync:
ip6:
        7 packets sent from this host
        Mbuf statistics:
icmp6:
        Output packet histogram:
                multicast listener report: 6
                neighbor solicitation: 1
        Histogram of error messages to be generated:
pim6:
rip6:
From the OpenBSD re man page
Quote:
All re NICs support IPv4 transmit/receive IP/TCP/UDP checksum offload and VLAN tag insertion, and use a descriptor-based DMA mechanism.
If with tcpdump, you experience similar checkum errors on outgoing packets with a hardware checksum processing NIC, don't worry. It is very unlikely that you have to replace that card or worse, register for a crash course network stack debugging.

From http://lists.freebsd.org/pipermail/f...st/004704.html
Quote:
Originally Posted by Robert Watson
...if you're sniffing outgoing packets
and the network interface is calculating the checksum on send, BPF will
see a version of the packet before the checksum is calculated. If tcpdump
later attempts to verify the checksum, it still won't be calculated in the
copy it sees, and will whine.
Makes sense, isn't it? The packets to be sent out, are presented to the network interface with dummy values in the checksum field.
Before transmission over the wire, the NIC calculates the actual checksum and replaces the dummy checksum field with the correct one

Trust but verify

To verify this I used to browser on 'hercules' to connect to the Apache webserver on 'vintrax'.
On both client and server, tcpdump captured the port 80 traffic.

Before showing some captured packets, we will first, check with netstat if it has detected checksum problems.

After having receved some HTTP requests over tcp, the server 'vintrax' does not report a single checksum related issue:

Code:
# uname -a
OpenBSD vintrax.utp.xnet 4.6 GENERIC#56 i386
# netstat -ss 
ip:
        625 total packets received
        613 packets for this host
        12 packets not forwardable
        521 packets sent from this host
        12 multicast packets which we don't join
icmp:
igmp:
ipencap:
tcp:
        447 packets sent
                438 data packets (52122 bytes)
                7 ack-only packets (245 delayed)
                2 control packets
        536 packets received
                339 acks (for 52031 bytes)
                2 duplicate acks
                245 packets (15339 bytes) received in-sequence
                2 out-of-order packets (0 bytes)
        3 connection accepts
        3 connections established (including accepts)
        4 connections closed (including 0 drops)
        339 segments updated rtt (of 334 attempts)
        133 correct ACK header predictions
        175 correct data packet header predictions
        7 PCB cache misses
        3 SYN cache entries added
                3 completed
The more detailed netstat -s output, which does not surpress counters with a value of zero:

Code:
# netstat -s  
ip:
        659 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
[snip]
        0 input datagrams checksum-processed by hardware
        0 output datagrams checksum-processed by hardware
[snip]
tcp:
        553 packets sent
                544 data packets (60170 bytes)
                0 data packets (0 bytes) retransmitted
[snip]
                2 control packets
                0 packets hardware-checksummed
        594 packets received
                394 acks (for 60063 bytes)
[snip]
                0 window update packets
                0 packets received after close
                0 discarded for bad checksums
[snip]
                0 packets hardware-checksummed
So 'vintrax' did not have send any badly checksummed packets to '/dev/null'.


The TCP connection initiated by 'hercules'

Code:
root@hercules[~]tcpdump -vvi re0 -s4096 'port 80' 
tcpdump: listening on re0, link-type EN10MB

08:00:31.959157 hercules.utp.xnet.6746 > vintrax.utp.xnet.www: S
[bad tcp cksum 6b68!] 803558741:803558741(0) win 65535 <mss
1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 3524320952 0>
(DF) (ttl 64, id 28883, len 64, bad cksum 14! differs by 8b76)
The first packet of the three way TCP handshake, with tcpdump complaining about the bad checksum.


Code:
08:00:31.959306 vintrax.utp.xnet.www > hercules.utp.xnet.6746: S
[tcp sum ok] 1399484825:1399484825(0) ack 803558742 win 16384 <mss
1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 1629307413
3524320952> (DF) (ttl 64, id 45620, len 64)
The second packet of the handshake received from 'vintrax'.


Code:
08:00:31.959331 hercules.utp.xnet.6746 > vintrax.utp.xnet.www: .
[bad tcp cksum fb40!] 1:1(0) ack 1 win 65535 <nop,nop,timestamp
3524320952 1629307413> (DF) (ttl 64, id 11921, len 52, bad cksum
14! differs by cdc4)
The third and last handshake packet, sent by 'hercules' again with checksum whining from tcpdump.

Capture of the same three packets on the 'vintrax' web server

Code:
# tcpdump -vvi fxp0 'port 80'
tcpdump: listening on fxp0, link-type EN10MB

08:00:31.917555 hercules.utp.xnet.6746 > vintrax.utp.xnet.www: S
[tcp sum ok] 803558741:803558741(0) win 65535 <mss
1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 3524320952 0>
(DF) (ttl 64, id 28883, len 64)
While tcpdump on 'hercules' reported a checksum problem on this first packet,
'vintrax' finds not any fault.


Code:
08:00:31.917616 vintrax.utp.xnet.www > hercules.utp.xnet.6746: S
[tcp sum ok] 1399484825:1399484825(0) ack 803558742 win 16384 <mss
1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 1629307413
3524320952> (DF) (ttl 64, id 45620, len 64)
The reply and second packet of the TCP connection setup.


Code:
08:00:31.917732 hercules.utp.xnet.6746 > vintrax.utp.xnet.www: .
[tcp sum ok] 1:1(0) ack 1 win 65535 <nop,nop,timestamp 3524320952
1629307413> (DF) (ttl 64, id 11921, len 52)
In the concluding packet, no checksum problem found by 'vintrax' either.

Although a sample of two packets is not really much, it looks like we can accept Robert Watson's explanation.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 3rd January 2010 at 04:12 PM. Reason: typo
Reply With Quote
  #5   (View Single Post)  
Old 15th January 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,125
Default

A recent post by an FreeBSD developer about the checksum errors reported by tcpdump: http://lists.freebsd.org/pipermail/f...ry/054134.html
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
FreeBSD FreeNAS drops FreeBSD in favor of Debian vermaden News 34 22nd December 2009 11:00 PM
Adobe drops most flash player licensing fees drhowarddrfine Off-Topic 7 18th October 2008 04:43 PM
i would like to know about tcpdump chamnanpol FreeBSD General 8 17th September 2008 11:00 AM
Redirecting ESP packets ales OpenBSD Security 2 15th June 2008 09:13 PM
IPF: Packets Out Of Window bram85 FreeBSD Security 9 2nd June 2008 04:09 PM


All times are GMT. The time now is 04:57 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick