packet loss 2% when pinging to OpenBSD 5.5 host

[mbw]

Hi,

I've got an openbsd 5.5 host running a dual-port OCE emulex 10G card
and an optical transceiver and fiber uplink to the UW network

This box is also a PF bridging firewall and any host on the other side of the
firewall also has packet loss. It is possible that PF is involved in the packet loss.


When I ping the firewall directly from another subnet I get about 2% packet loss


root@nori:~# ping -f -c 1000 firewall
PING firewall (172.16.205.50) 56(84) bytes of data.
..............................
--- firewall ping statistics ---
1000 packets transmitted, 970 received, 3% packet loss, time 1753ms
rtt min/avg/max/mdev = 0.518/1.467/53.222/3.850 ms, pipe 4, ipg/ewma 1.754/0.680 ms
root@nori:~#


The load on the firewall appears minimal. The box is a server-grade Sun Fire x4170 with 8G ram


My Questions are:

1) How to troubleshoot an openbsd host to try to find the cause of the packet loss

2) What dual port 10G cards work best in OpenBSD - I need dual port Optical SFP+
capable 10G cards - is there a good list somewhere?

3) My emulex 10G card has internal firmware - is there a recommended firmware level to flash to
posted somewhere?

responses to any of these q's appreciated

thanks in advance,
Matt

[jggimi]

Welcome back!

I monitor the daily(8) reports from my servers, as network I/O errors are reported via netstat(8) -ivn options. This helps me monitor the health of cabling.

But when it comes to more detailed diagnostics, I run netsat(8) with the -s option, which produces more the 350 lines of network stack statistics.

[mbw]

Thank - I've been away since OpenBSD is so stable - no questions for a long time!

Heres the results of those commands - anything leap out at you?



[oBSD55: firewall:~ ] $ netstat -ivm
160 mbufs in use:
151 mbufs allocated to data
3 mbufs allocated to packet headers
6 mbufs allocated to socket names and addresses
150/530/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
1408 Kbytes allocated to network (24% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
[oBSD55: firewall:~ ] $ netstat -s
ip:
109127232 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (duplicates or out of space)
0 malformed fragments dropped
0 fragments dropped after timeout
0 packets reassembled ok
230690 packets for this host
53 packets for unknown/unsupported protocol
0 packets forwarded
21634680 packets not forwardable
0 redirects sent
1413320 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
91 output datagrams fragmented
91 fragments created
0 datagrams that can't be fragmented
0 fragment floods
0 packets with ip length > max ip packet size
0 tunneling packets that can't find gif
0 datagrams with bad address in header
1295 input datagrams software-checksummed
516933441201 output datagrams software-checksummed
29520517 multicast packets which we don't join
icmp:
61131523 calls to icmp_error
0 errors not generated because old message was icmp
666 errors not generated because of rate limitation
Output packet histogram:
echo reply: 45599
destination unreachable: 657798
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
0 echo requests to broadcast/multicast rejected
Input packet histogram:
echo reply: 29
destination unreachable: 55
echo: 45599
45599 message responses generated
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
ipencap:
0 total input packets
0 total output packets
0 packets shorter than header shows
0 packets dropped due to policy
0 packets with possibly spoofed local addresses
0 packets were dropped due to full output queue
0 input bytes
0 output bytes
0 protocol family mismatches
0 attempts to use tunnel with unspecified endpoint(s)
tcp:
54832 packets sent
53604 data packets (48268549 bytes)
3 data packets (288 bytes) retransmitted
0 fast retransmitted packets
1080 ack-only packets (14389 delayed)
0 URG only packets
0 window probe packets
18 window update packets
128 control packets
632674775 packets software-checksummed
50890 packets received
38433 acks (for 48268337 bytes)
76 duplicate acks
0 acks for unsent data
0 acks for old data
15216 packets (714253 bytes) received in-sequence
15 completely duplicate packets (1272 bytes)
0 old duplicate packets
0 packets with some duplicate data (0 bytes duplicated)
59 out-of-order packets (960 bytes)
0 packets (0 bytes) of data after window
0 window probes
3663 window update packets
0 packets received after close
305 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded for missing IPsec protection
0 discarded due to memory shortage
1574 packets software-checksummed
0 bad/missing md5 checksums
0 good md5 checksums
40 connection requests
53 connection accepts
89 connections established (including accepts)
1027 connections closed (including 0 drops)
0 connections drained
4 embryonic connections dropped
38466 segments updated rtt (of 26172 attempts)
4 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
12 keepalive timeouts
12 keepalive probes sent
0 connections dropped by keepalive
10504 correct ACK header predictions
7569 correct data packet header predictions
114 PCB cache misses
0 ECN connections accepted
0 ECE packets received
0 CWR packets received
0 CE packets received
0 ECT packets sent
0 ECE packets sent
0 CWR packets sent
cwr by fastrecovery: 0
cwr by timeout: 4
cwr by ecn: 0
0 bad connection attempts
58 SYN cache entries added
0 hash collisions
53 completed
0 aborted (no space to build PCB)
5 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
0 dropped due to RST
0 dropped due to ICMP unreachable
20 SYN,ACKs retransmitted
0 duplicate SYNs received for entries already in the cache
0 SYNs dropped (no route or no space)
0 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
3 SACK options received
14 SACK options sent
udp:
134179 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
0 with no checksum
36 input packets software-checksummed
12321 output packets software-checksummed
121914 dropped due to no socket
121967 broadcast/multicast datagrams dropped due to no socket
0 dropped due to missing IPsec protection
0 dropped due to full socket buffers
18446744073709441914 delivered
12330 datagrams output
121934 missed PCB cache
esp:
0 input ESP packets
0 output ESP packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets with bad encryption received
0 packets that failed verification received
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 possibly replayed packets received
0 packets with bad payload size or padding received
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed crypto processing
0 input UDP encapsulated ESP packets
0 output UDP encapsulated ESP packets
0 UDP packets for non-encapsulating TDB received
0 input bytes
0 output bytes
ah:
0 input AH packets
0 output AH packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets that failed verification received
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 possibly replayed packets received
0 packets with bad authenticator length received
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed crypto processing
0 input bytes
0 output bytes
etherip:
0 packets shorter than header shows
0 packets were dropped due to full output queue
0 packets were dropped because of no interface/bridge information
0 packets dropped due to policy
0 packets dropped for other reasons
0 input ethernet-in-IP packets
0 output ethernet-in-IP packets
0 input bytes
0 output bytes
ipcomp:
0 input IPCOMP packets
0 output IPCOMP packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed (de)compression processing
0 packets less than minimum compression length
0 input bytes
0 output bytes
carp:
0 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
0 discarded for unknown vhid
0 discarded because of a bad address list
0 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 transitions to master
pfsync:
0 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for bad ttl
0 packets shorter than header
0 packets discarded for bad version
0 packets discarded for bad HMAC
0 packets discarded for bad action
0 packets discarded for short packet
0 states discarded for bad values
0 stale states
0 failed state lookup/inserts
0 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 send error
divert:
0 total packets received
0 dropped due to no socket
0 dropped due to full socket buffers
0 packets output
0 errors
pflow:
0 flows sent
0 packets sent
0 send failed due to mbuf memory error
0 send error
ip6:
2668645 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (duplicates or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 packets reassembled ok
0 packets for this host
0 packets forwarded
0 packets not forwardable
0 redirects sent
19 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 packets that violated scope rules
0 multicast packets which we don't join
Input packet histogram:
hop by hop: 544
UDP: 2667834
ICMP6: 267
Mbuf statistics:
0 one mbufs
2668645 one ext mbufs
0 two or more ext mbufs
0 tunneling packets that can't find gif
0 packets discarded due to too many headers
0 failures of source address selection
0 forward cache hit
0 forward cache miss
divert6:
0 total packets received
0 dropped due to no socket
0 dropped due to full socket buffers
0 packets output
0 errors
icmp6:
5337265 calls to icmp6_error
0 errors not generated because old message was icmp6 or so
0 errors not generated because of rate limitation
Output packet histogram:
multicast listener report: 16
neighbor solicitation: 3
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Histogram of error messages to be generated:
0 no route
0 administratively prohibited
0 beyond scope
0 address unreachable
5337265 port unreachable
0 packet too big
0 time exceed transit
0 time exceed reassembly
0 erroneous header field
0 unrecognized next header
0 unrecognized option
0 redirect
0 unknown
0 message responses generated
0 messages with too many ND options
0 messages with bad ND options
0 bad neighbor solicitation messages
0 bad neighbor advertisement messages
0 bad router solicitation messages
0 bad router advertisement messages
0 bad redirect messages
0 path MTU changes
pim6:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 messages received with bad version
0 registers received
0 bad registers received
0 registers sent
rip6:
0 messages received
0 checksum calculations on inbound
0 messages with bad checksum
0 messages dropped due to no socket
0 multicast messages dropped due to no socket
0 messages dropped due to full socket buffers
0 delivered
0 datagrams output
[oBSD55:Thu Sep 17

Theres some suspicious looking UDP and ICMP section metrics there.... but not sure what I am looking at.
Care to enlighten me?

my pf.conf blocks all ipv6 as far as I know, BTW


Thanks for lookin'

- Matt

[jggimi]

The first thing that jumped out at me was that zero packets are forwarded, 21 million packets not forwardable. But then I recalled you are using bridging rather than packet forwarding.

Then this really jumped out at me: Twelve hundred input datagrams (Ethernet transmissions) software checksummed, followed by more than half a trillion output datagrams software checksummed.

In comparison, on my main firewall the ratio is about 6:1 input to output checksummed. While I have different network interfaces in use and have a different network infrastructure -- packet forwarding, trunking, vlans, and carp -- your ratio of input to output checksumming caught my eye.

Then you have 600 million TCP packets software checksummed, but only 50 thousand packets received or sent.

All of that may be due to your bridged infrastructure, but ... these seem odd to me.

I have 4 billion UDP "delivered" messages on my main firewall, and that's a tiny Alix machine with three 100BaseT NICs that's been up for 35 days (since the last 5.7-stable update). So the large number in your output may not be as wacky as it appears to be.

---
Edited to add: The UDP "delivered" statistic must be bytes, rather than packets.

What does netstat -ivn -- as in daily(8) reports, mentioned above -- show you? Any receive or transmit errors?

[mbw]

Thanks for looking at the #s - appreciate that

The OCE0 interface is a public interface exposed to the unfiltered internet... we get lots of scanning and probing ssh stuff that gets blocked - we also have a pretty restricted ruleset so lots of traffic is blocked - would that explain the
imbalance you saw? We bridge and use pf to filter.... those stats are on a box that is no more than 20 days uptime...
It wasnt clear to me if those stats reset at boot time....

heres the other piece you requested:

firewall:~ ] $ netstat -ivn 
Name    Mtu   Network     Address              Ipkts Ierrs    Opkts Oerrs Colls
lo0     33144 <Link>                            1366     0     1366     0     0
lo0     33144 ::1/128     ::1                   1366     0     1366     0     0
lo0     33144 fe80::%lo0/64 fe80::1%lo0           1366     0     1366     0     0
lo0     33144 127/8       127.0.0.1             1366     0     1366     0     0
em0*    1500  <Link>      00:14:4f:ca:cb:26        0     0        0     0     0
em1*    1500  <Link>      00:14:4f:ca:cb:27        0     0        0     0     0
em2*    1500  <Link>      00:14:4f:ca:cb:28        0     0        0     0     0
em3     1500  <Link>      00:14:4f:ca:cb:29 19920186     0    37892     0     0
em3     1500  10.64.0/24  10.64.0.50        19920186     0    37892     0     0
em3     1500  fe80::%em3/64 fe80::214:4fff:feca:cb29%em3 19920186     0    37892     0     0
oce0    1500  <Link>      00:90:fa:1e:e9:5e 158620998818   729 104115803019     0     0
oce0    1500  10.28.15/24 10.28.15.50     158620998818   729 104115803019     0     0
oce0    1500  fe80::%oce0/64 fe80::290:faff:fe1e:e95e%oce0 158620998818   729 104115803019     0     0
oce1    1500  <Link>      00:90:fa:1e:e9:62 104128089141   141 158566642918     0     0
oce1    1500  fe80::%oce1/64 fe80::290:faff:fe1e:e962%oce1 104128089141   141 158566642918     0     0
enc0*   0     <Link>                               0     0        0     0     0
bridge0 1500  <Link>                        262686796779     0 262680911891     0     0
pflog0  33144 <Link>                               0     0 109578690     0     0
[oBSD55 firewall: ]

about bridge0:

more /etc/hostname.bridge0                                                                               <
add oce0
add oce1
blocknonip oce0
blocknonip oce1
spanpriority 0
up

[oBSD55: #

[oBSD55:Fri Sep 18 21:07:26 root@firewall:~ ] $ w
 9:08PM  up 19 days,  3:06, 2 users, load averages: 0.08, 0.08, 0.08
USER    TTY FROM              LOGIN@  IDLE WHAT
root     p0 senpriv     Thu03PM     0 w 
root     p1 hostnamew Wed03PM     0 -ksh 
[oBSD55:Fri Sep 18 21:08:19 root@csde-firewall:~ ] $ uname -a
OpenBSD firewall 5.5 GENERIC.MP#315 amd64
[oBSD55:Fri Sep 18 21:08:32 root@firewall:~ ] $

[jggimi]

To my understanding the stats are reset at boot.

Thanks for the additional info. I was hoping to see huge numbers of Ierrs or Oerrs to explain your packet loss. I'm not seeing that. As an example, the oce0 NIC has, in round numbers, 159 billion inbound packets with 729 input errors. The oce1 NIC has 104 billion inbound packets with 141 input errors. That's it.

If no one else jumps into this discussion here, you might take the issue to the misc@ mailing list for consideration. It's a much larger community, and includes about half of the developers.

[TronDD]

I don't have much to offer other than some ideas.

What's between the system sending the pings and the firewall? Can you get statistics from any switches or routers and see if there are problems there? Failing that, have you watched the incoming port on the firewall with tcpdump to see if the packets reported as lost by ping ever even made it to the firewall? Does the firewall never see them or does it drop/not reply to them?

What if you send traffic from the firewall or behind it out to the other subnet?

Have you tried going to the firewall's incoming port from the same subnet (same switch even), eliminating most of the networking infrastructure between the two points?

Just some ideas to try narrowing things down.

Tim.

[mbw]

thanks for all the suggestions... im not seeing the problem anymore. I work on a large US campus with big juniper networks and aggregators, turns out there was a major network traffic redirect going on at the time - one of the main routes through campus was down and the backup aggregator path was active - it had more hops than the standard path, thats all I can really point to at this moment.

Pinging stuff on the same subnet was fine, no packet loss

other subnets were getting packet loss also

We also had the good netops folks "roll the fiber" (Reterminate at the switch)
and that may have helped. I didnt see anything on the switch logs (had to call netops to get them to look at that) or on the openbsd nic that suggested there was any communication problem. Netops did say something about a "low power signal" over the fiber - not sure what that means - maybe a marginal SFP+ module that couldnt make light bright enough to traverse the uplinl to the agg switch? Who knows

anyway, case closed. thanks for playing