View Single Post
  #4   (View Single Post)  
Old 3rd January 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default Tcpdump checksum errors on outgoing packets

Tcpdump checksum errors on outgoing packets

To "fall with the door into the house" (literal translation of a Dutch saying):

Code:
root@hercules[~]tcpdump -vvi re0 -s4096  
tcpdump: listening on re0, link-type EN10MB

07:08:11.939997 hercules.utp.xnet.26487 > mx1.xillhosting.nl.ntp:
[bad udp cksum e4d9!] v4 client strat 0 poll 0 prec 0 dist 0.000000
disp 0.000000 ref (unspec)@0.000000000 orig 0.000000000 rec
-0.000000000 xmt +1710637651.099857769 [tos 0x10] (ttl 64, id 4635,
len 76, bad cksum 38! differs by 1fb2)
You don't encounter bad checksums in NTP packets everyday, but sometimes you do.

Code:
07:08:11.959364 mx1.xillhosting.nl.ntp > hercules.utp.xnet.26487:
[udp sum ok] v4 server strat 2 poll 0 prec -21 dist 0.004638 disp
0.016815 ref chime5.surfnet.nl@3471486924.672977864 orig
1710637651.099857769 rec +1760850040.850035130 xmt +1760850040.850053966
(DF) (ttl 58, id 0, len 76)
Even more stranger, mx1.xillhosting.nl just answers. Aren't packets with erroneous checksums to be silently discarded?

Code:
07:08:12.690406 hercules.utp.xnet.47141 > parmenides.utp.xnet.domain:
[bad udp cksum b20!] 37272+ PTR? 218.91.245.77.in-addr.arpa. (44)
(ttl 64, id 3821, len 72, bad cksum 34! differs by 2e14)
Here again, tcpdump detects a checksum problem on an outgoing packet. Does 'hercules' have a bad day with mathematics today ?

Code:
07:08:13.201840 parmenides.utp.xnet.domain > hercules.utp.xnet.47141:
[udp sum ok] 37272 q: PTR? 218.91.245.77.in-addr.arpa. 1/0/0
218.91.245.77.in-addr.arpa. PTR mx1.xillhosting.nl. (76) (ttl 64,
id 35978, len 104)
The 'parmenides' nameserver seems to have not any problem with 'hercules' request, it just answers.

A diagnostic attempt with netstat -in on hercules does not show any error.
Code:
# netstat -in
Name    Mtu   Network     Address              Ipkts Ierrs    Opkts Oerrs Colls
lo0     33168 <Link>                               0     0        0     0     0
lo0     33168 127/8       127.0.0.1                0     0        0     0     0
lo0     33168 ::1/128     ::1                      0     0        0     0     0
lo0     33168 fe80::%lo0/ fe80::1%lo0              0     0        0     0     0
bge0*   1500  <Link>      00:10:18:00:9f:fd        0     0        0     0     0
re0     1500  <Link>      00:19:db:47:b0:4c     1607     0     1435     0     0
re0     1500  192.168.222 192.168.222.20        1607     0     1435     0     0
re0     1500  fe80::%re0/ fe80::219:dbff:fe     1607     0     1435     0     0
enc0*   1536  <Link>                               0     0        0     0     0
Even netstat -ss reports no checksum related errors, but mentions that the network chip does checksum processing in hardware.

Code:
# netstat -ss
ip:
        1600 total packets received
        1564 packets for this host
        36 packets not forwardable
        1426 packets sent from this host
        1600 input datagrams checksum-processed by hardware
        1426 output datagrams checksum-processed by hardware
        36 multicast packets which we don't join
icmp:
        Input packet histogram:
                echo reply: 3
igmp:
ipencap:
tcp:
        1085 packets sent
                162 data packets (107560 bytes)
                550 ack-only packets (621 delayed)
                213 window update packets
                160 control packets
                1085 packets hardware-checksummed
        1225 packets received
                241 acks (for 107639 bytes)
                74 duplicate acks
                954 packets (1130771 bytes) received in-sequence
                3 completely duplicate packets (0 bytes)
                25 out-of-order packets (10015 bytes)
                6 window update packets
                1225 packets hardware-checksummed
        80 connection requests
        80 connections established (including accepts)
        84 connections closed (including 0 drops)
        321 segments updated rtt (of 322 attempts)
        812 correct data packet header predictions
        5 SACK options sent
udp:
        336 datagrams received
        336 input packets hardware-checksummed
        338 output packets hardware-checksummed
        336 delivered
        338 datagrams output
esp:
ah:
etherip:
ipcomp:
carp:
pfsync:
ip6:
        7 packets sent from this host
        Mbuf statistics:
icmp6:
        Output packet histogram:
                multicast listener report: 6
                neighbor solicitation: 1
        Histogram of error messages to be generated:
pim6:
rip6:
From the OpenBSD re man page
Quote:
All re NICs support IPv4 transmit/receive IP/TCP/UDP checksum offload and VLAN tag insertion, and use a descriptor-based DMA mechanism.
If with tcpdump, you experience similar checkum errors on outgoing packets with a hardware checksum processing NIC, don't worry. It is very unlikely that you have to replace that card or worse, register for a crash course network stack debugging.

From http://lists.freebsd.org/pipermail/f...st/004704.html
Quote:
Originally Posted by Robert Watson
...if you're sniffing outgoing packets
and the network interface is calculating the checksum on send, BPF will
see a version of the packet before the checksum is calculated. If tcpdump
later attempts to verify the checksum, it still won't be calculated in the
copy it sees, and will whine.
Makes sense, isn't it? The packets to be sent out, are presented to the network interface with dummy values in the checksum field.
Before transmission over the wire, the NIC calculates the actual checksum and replaces the dummy checksum field with the correct one

Trust but verify

To verify this I used to browser on 'hercules' to connect to the Apache webserver on 'vintrax'.
On both client and server, tcpdump captured the port 80 traffic.

Before showing some captured packets, we will first, check with netstat if it has detected checksum problems.

After having receved some HTTP requests over tcp, the server 'vintrax' does not report a single checksum related issue:

Code:
# uname -a
OpenBSD vintrax.utp.xnet 4.6 GENERIC#56 i386
# netstat -ss 
ip:
        625 total packets received
        613 packets for this host
        12 packets not forwardable
        521 packets sent from this host
        12 multicast packets which we don't join
icmp:
igmp:
ipencap:
tcp:
        447 packets sent
                438 data packets (52122 bytes)
                7 ack-only packets (245 delayed)
                2 control packets
        536 packets received
                339 acks (for 52031 bytes)
                2 duplicate acks
                245 packets (15339 bytes) received in-sequence
                2 out-of-order packets (0 bytes)
        3 connection accepts
        3 connections established (including accepts)
        4 connections closed (including 0 drops)
        339 segments updated rtt (of 334 attempts)
        133 correct ACK header predictions
        175 correct data packet header predictions
        7 PCB cache misses
        3 SYN cache entries added
                3 completed
The more detailed netstat -s output, which does not surpress counters with a value of zero:

Code:
# netstat -s  
ip:
        659 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
[snip]
        0 input datagrams checksum-processed by hardware
        0 output datagrams checksum-processed by hardware
[snip]
tcp:
        553 packets sent
                544 data packets (60170 bytes)
                0 data packets (0 bytes) retransmitted
[snip]
                2 control packets
                0 packets hardware-checksummed
        594 packets received
                394 acks (for 60063 bytes)
[snip]
                0 window update packets
                0 packets received after close
                0 discarded for bad checksums
[snip]
                0 packets hardware-checksummed
So 'vintrax' did not have send any badly checksummed packets to '/dev/null'.


The TCP connection initiated by 'hercules'

Code:
root@hercules[~]tcpdump -vvi re0 -s4096 'port 80' 
tcpdump: listening on re0, link-type EN10MB

08:00:31.959157 hercules.utp.xnet.6746 > vintrax.utp.xnet.www: S
[bad tcp cksum 6b68!] 803558741:803558741(0) win 65535 <mss
1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 3524320952 0>
(DF) (ttl 64, id 28883, len 64, bad cksum 14! differs by 8b76)
The first packet of the three way TCP handshake, with tcpdump complaining about the bad checksum.


Code:
08:00:31.959306 vintrax.utp.xnet.www > hercules.utp.xnet.6746: S
[tcp sum ok] 1399484825:1399484825(0) ack 803558742 win 16384 <mss
1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 1629307413
3524320952> (DF) (ttl 64, id 45620, len 64)
The second packet of the handshake received from 'vintrax'.


Code:
08:00:31.959331 hercules.utp.xnet.6746 > vintrax.utp.xnet.www: .
[bad tcp cksum fb40!] 1:1(0) ack 1 win 65535 <nop,nop,timestamp
3524320952 1629307413> (DF) (ttl 64, id 11921, len 52, bad cksum
14! differs by cdc4)
The third and last handshake packet, sent by 'hercules' again with checksum whining from tcpdump.

Capture of the same three packets on the 'vintrax' web server

Code:
# tcpdump -vvi fxp0 'port 80'
tcpdump: listening on fxp0, link-type EN10MB

08:00:31.917555 hercules.utp.xnet.6746 > vintrax.utp.xnet.www: S
[tcp sum ok] 803558741:803558741(0) win 65535 <mss
1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 3524320952 0>
(DF) (ttl 64, id 28883, len 64)
While tcpdump on 'hercules' reported a checksum problem on this first packet,
'vintrax' finds not any fault.


Code:
08:00:31.917616 vintrax.utp.xnet.www > hercules.utp.xnet.6746: S
[tcp sum ok] 1399484825:1399484825(0) ack 803558742 win 16384 <mss
1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 1629307413
3524320952> (DF) (ttl 64, id 45620, len 64)
The reply and second packet of the TCP connection setup.


Code:
08:00:31.917732 hercules.utp.xnet.6746 > vintrax.utp.xnet.www: .
[tcp sum ok] 1:1(0) ack 1 win 65535 <nop,nop,timestamp 3524320952
1629307413> (DF) (ttl 64, id 11921, len 52)
In the concluding packet, no checksum problem found by 'vintrax' either.

Although a sample of two packets is not really much, it looks like we can accept Robert Watson's explanation.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 3rd January 2010 at 04:12 PM. Reason: typo
Reply With Quote