|
OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below. |
|
Thread Tools | Display Modes |
|
||||
Welcome back!
I monitor the daily(8) reports from my servers, as network I/O errors are reported via netstat(8) -ivn options. This helps me monitor the health of cabling. But when it comes to more detailed diagnostics, I run netsat(8) with the -s option, which produces more the 350 lines of network stack statistics. |
|
||||
Thank - I've been away since OpenBSD is so stable - no questions for a long time!
Heres the results of those commands - anything leap out at you? [oBSD55: firewall:~ ] $ netstat -ivm 160 mbufs in use: 151 mbufs allocated to data 3 mbufs allocated to packet headers 6 mbufs allocated to socket names and addresses 150/530/6144 mbuf 2048 byte clusters in use (current/peak/max) 0/8/6144 mbuf 4096 byte clusters in use (current/peak/max) 0/8/6144 mbuf 8192 byte clusters in use (current/peak/max) 0/8/6144 mbuf 9216 byte clusters in use (current/peak/max) 0/8/6144 mbuf 12288 byte clusters in use (current/peak/max) 0/8/6144 mbuf 16384 byte clusters in use (current/peak/max) 0/8/6144 mbuf 65536 byte clusters in use (current/peak/max) 1408 Kbytes allocated to network (24% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines [oBSD55: firewall:~ ] $ netstat -s ip: 109127232 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (duplicates or out of space) 0 malformed fragments dropped 0 fragments dropped after timeout 0 packets reassembled ok 230690 packets for this host 53 packets for unknown/unsupported protocol 0 packets forwarded 21634680 packets not forwardable 0 redirects sent 1413320 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 91 output datagrams fragmented 91 fragments created 0 datagrams that can't be fragmented 0 fragment floods 0 packets with ip length > max ip packet size 0 tunneling packets that can't find gif 0 datagrams with bad address in header 1295 input datagrams software-checksummed 516933441201 output datagrams software-checksummed 29520517 multicast packets which we don't join icmp: 61131523 calls to icmp_error 0 errors not generated because old message was icmp 666 errors not generated because of rate limitation Output packet histogram: echo reply: 45599 destination unreachable: 657798 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length 0 echo requests to broadcast/multicast rejected Input packet histogram: echo reply: 29 destination unreachable: 55 echo: 45599 45599 message responses generated igmp: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 membership reports sent ipencap: 0 total input packets 0 total output packets 0 packets shorter than header shows 0 packets dropped due to policy 0 packets with possibly spoofed local addresses 0 packets were dropped due to full output queue 0 input bytes 0 output bytes 0 protocol family mismatches 0 attempts to use tunnel with unspecified endpoint(s) tcp: 54832 packets sent 53604 data packets (48268549 bytes) 3 data packets (288 bytes) retransmitted 0 fast retransmitted packets 1080 ack-only packets (14389 delayed) 0 URG only packets 0 window probe packets 18 window update packets 128 control packets 632674775 packets software-checksummed 50890 packets received 38433 acks (for 48268337 bytes) 76 duplicate acks 0 acks for unsent data 0 acks for old data 15216 packets (714253 bytes) received in-sequence 15 completely duplicate packets (1272 bytes) 0 old duplicate packets 0 packets with some duplicate data (0 bytes duplicated) 59 out-of-order packets (960 bytes) 0 packets (0 bytes) of data after window 0 window probes 3663 window update packets 0 packets received after close 305 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded for missing IPsec protection 0 discarded due to memory shortage 1574 packets software-checksummed 0 bad/missing md5 checksums 0 good md5 checksums 40 connection requests 53 connection accepts 89 connections established (including accepts) 1027 connections closed (including 0 drops) 0 connections drained 4 embryonic connections dropped 38466 segments updated rtt (of 26172 attempts) 4 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 12 keepalive timeouts 12 keepalive probes sent 0 connections dropped by keepalive 10504 correct ACK header predictions 7569 correct data packet header predictions 114 PCB cache misses 0 ECN connections accepted 0 ECE packets received 0 CWR packets received 0 CE packets received 0 ECT packets sent 0 ECE packets sent 0 CWR packets sent cwr by fastrecovery: 0 cwr by timeout: 4 cwr by ecn: 0 0 bad connection attempts 58 SYN cache entries added 0 hash collisions 53 completed 0 aborted (no space to build PCB) 5 timed out 0 dropped due to overflow 0 dropped due to bucket overflow 0 dropped due to RST 0 dropped due to ICMP unreachable 20 SYN,ACKs retransmitted 0 duplicate SYNs received for entries already in the cache 0 SYNs dropped (no route or no space) 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 3 SACK options received 14 SACK options sent udp: 134179 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 0 with no checksum 36 input packets software-checksummed 12321 output packets software-checksummed 121914 dropped due to no socket 121967 broadcast/multicast datagrams dropped due to no socket 0 dropped due to missing IPsec protection 0 dropped due to full socket buffers 18446744073709441914 delivered 12330 datagrams output 121934 missed PCB cache esp: 0 input ESP packets 0 output ESP packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 0 packets for which no TDB was found 0 input packets that failed to be processed 0 packets with bad encryption received 0 packets that failed verification received 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 0 possibly replayed packets received 0 packets with bad payload size or padding received 0 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed crypto processing 0 input UDP encapsulated ESP packets 0 output UDP encapsulated ESP packets 0 UDP packets for non-encapsulating TDB received 0 input bytes 0 output bytes ah: 0 input AH packets 0 output AH packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 0 packets for which no TDB was found 0 input packets that failed to be processed 0 packets that failed verification received 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 0 possibly replayed packets received 0 packets with bad authenticator length received 0 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed crypto processing 0 input bytes 0 output bytes etherip: 0 packets shorter than header shows 0 packets were dropped due to full output queue 0 packets were dropped because of no interface/bridge information 0 packets dropped due to policy 0 packets dropped for other reasons 0 input ethernet-in-IP packets 0 output ethernet-in-IP packets 0 input bytes 0 output bytes ipcomp: 0 input IPCOMP packets 0 output IPCOMP packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 0 packets for which no TDB was found 0 input packets that failed to be processed 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 0 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed (de)compression processing 0 packets less than minimum compression length 0 input bytes 0 output bytes carp: 0 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for wrong TTL 0 packets shorter than header 0 discarded for bad checksums 0 discarded packets with a bad version 0 discarded because packet too short 0 discarded for bad authentication 0 discarded for unknown vhid 0 discarded because of a bad address list 0 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 0 transitions to master pfsync: 0 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for bad ttl 0 packets shorter than header 0 packets discarded for bad version 0 packets discarded for bad HMAC 0 packets discarded for bad action 0 packets discarded for short packet 0 states discarded for bad values 0 stale states 0 failed state lookup/inserts 0 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 0 send error divert: 0 total packets received 0 dropped due to no socket 0 dropped due to full socket buffers 0 packets output 0 errors pflow: 0 flows sent 0 packets sent 0 send failed due to mbuf memory error 0 send error ip6: 2668645 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (duplicates or out of space) 0 fragments dropped after timeout 0 fragments that exceeded limit 0 packets reassembled ok 0 packets for this host 0 packets forwarded 0 packets not forwardable 0 redirects sent 19 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 packets that violated scope rules 0 multicast packets which we don't join Input packet histogram: hop by hop: 544 UDP: 2667834 ICMP6: 267 Mbuf statistics: 0 one mbufs 2668645 one ext mbufs 0 two or more ext mbufs 0 tunneling packets that can't find gif 0 packets discarded due to too many headers 0 failures of source address selection 0 forward cache hit 0 forward cache miss divert6: 0 total packets received 0 dropped due to no socket 0 dropped due to full socket buffers 0 packets output 0 errors icmp6: 5337265 calls to icmp6_error 0 errors not generated because old message was icmp6 or so 0 errors not generated because of rate limitation Output packet histogram: multicast listener report: 16 neighbor solicitation: 3 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Histogram of error messages to be generated: 0 no route 0 administratively prohibited 0 beyond scope 0 address unreachable 5337265 port unreachable 0 packet too big 0 time exceed transit 0 time exceed reassembly 0 erroneous header field 0 unrecognized next header 0 unrecognized option 0 redirect 0 unknown 0 message responses generated 0 messages with too many ND options 0 messages with bad ND options 0 bad neighbor solicitation messages 0 bad neighbor advertisement messages 0 bad router solicitation messages 0 bad router advertisement messages 0 bad redirect messages 0 path MTU changes pim6: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 messages received with bad version 0 registers received 0 bad registers received 0 registers sent rip6: 0 messages received 0 checksum calculations on inbound 0 messages with bad checksum 0 messages dropped due to no socket 0 multicast messages dropped due to no socket 0 messages dropped due to full socket buffers 0 delivered 0 datagrams output [oBSD55:Thu Sep 17 Theres some suspicious looking UDP and ICMP section metrics there.... but not sure what I am looking at. Care to enlighten me? my pf.conf blocks all ipv6 as far as I know, BTW Thanks for lookin' - Matt |
|
||||
The first thing that jumped out at me was that zero packets are forwarded, 21 million packets not forwardable. But then I recalled you are using bridging rather than packet forwarding.
Then this really jumped out at me: Twelve hundred input datagrams (Ethernet transmissions) software checksummed, followed by more than half a trillion output datagrams software checksummed. In comparison, on my main firewall the ratio is about 6:1 input to output checksummed. While I have different network interfaces in use and have a different network infrastructure -- packet forwarding, trunking, vlans, and carp -- your ratio of input to output checksumming caught my eye. Then you have 600 million TCP packets software checksummed, but only 50 thousand packets received or sent. All of that may be due to your bridged infrastructure, but ... these seem odd to me. I have 4 billion UDP "delivered" messages on my main firewall, and that's a tiny Alix machine with three 100BaseT NICs that's been up for 35 days (since the last 5.7-stable update). So the large number in your output may not be as wacky as it appears to be. --- Edited to add: The UDP "delivered" statistic must be bytes, rather than packets. What does netstat -ivn -- as in daily(8) reports, mentioned above -- show you? Any receive or transmit errors? Last edited by jggimi; 18th September 2015 at 12:41 PM. |
|
||||
Thanks for looking at the #s - appreciate that
The OCE0 interface is a public interface exposed to the unfiltered internet... we get lots of scanning and probing ssh stuff that gets blocked - we also have a pretty restricted ruleset so lots of traffic is blocked - would that explain the imbalance you saw? We bridge and use pf to filter.... those stats are on a box that is no more than 20 days uptime... It wasnt clear to me if those stats reset at boot time.... heres the other piece you requested: Code:
firewall:~ ] $ netstat -ivn Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Colls lo0 33144 <Link> 1366 0 1366 0 0 lo0 33144 ::1/128 ::1 1366 0 1366 0 0 lo0 33144 fe80::%lo0/64 fe80::1%lo0 1366 0 1366 0 0 lo0 33144 127/8 127.0.0.1 1366 0 1366 0 0 em0* 1500 <Link> 00:14:4f:ca:cb:26 0 0 0 0 0 em1* 1500 <Link> 00:14:4f:ca:cb:27 0 0 0 0 0 em2* 1500 <Link> 00:14:4f:ca:cb:28 0 0 0 0 0 em3 1500 <Link> 00:14:4f:ca:cb:29 19920186 0 37892 0 0 em3 1500 10.64.0/24 10.64.0.50 19920186 0 37892 0 0 em3 1500 fe80::%em3/64 fe80::214:4fff:feca:cb29%em3 19920186 0 37892 0 0 oce0 1500 <Link> 00:90:fa:1e:e9:5e 158620998818 729 104115803019 0 0 oce0 1500 10.28.15/24 10.28.15.50 158620998818 729 104115803019 0 0 oce0 1500 fe80::%oce0/64 fe80::290:faff:fe1e:e95e%oce0 158620998818 729 104115803019 0 0 oce1 1500 <Link> 00:90:fa:1e:e9:62 104128089141 141 158566642918 0 0 oce1 1500 fe80::%oce1/64 fe80::290:faff:fe1e:e962%oce1 104128089141 141 158566642918 0 0 enc0* 0 <Link> 0 0 0 0 0 bridge0 1500 <Link> 262686796779 0 262680911891 0 0 pflog0 33144 <Link> 0 0 109578690 0 0 [oBSD55 firewall: ] Code:
more /etc/hostname.bridge0 < add oce0 add oce1 blocknonip oce0 blocknonip oce1 spanpriority 0 up [oBSD55: # [oBSD55:Fri Sep 18 21:07:26 root@firewall:~ ] $ w 9:08PM up 19 days, 3:06, 2 users, load averages: 0.08, 0.08, 0.08 USER TTY FROM LOGIN@ IDLE WHAT root p0 senpriv Thu03PM 0 w root p1 hostnamew Wed03PM 0 -ksh [oBSD55:Fri Sep 18 21:08:19 root@csde-firewall:~ ] $ uname -a OpenBSD firewall 5.5 GENERIC.MP#315 amd64 [oBSD55:Fri Sep 18 21:08:32 root@firewall:~ ] $ Last edited by J65nko; 19th September 2015 at 04:18 AM. Reason: added [code] and [/code] tags ;) |
|
||||
To my understanding the stats are reset at boot.
Thanks for the additional info. I was hoping to see huge numbers of Ierrs or Oerrs to explain your packet loss. I'm not seeing that. As an example, the oce0 NIC has, in round numbers, 159 billion inbound packets with 729 input errors. The oce1 NIC has 104 billion inbound packets with 141 input errors. That's it. If no one else jumps into this discussion here, you might take the issue to the misc@ mailing list for consideration. It's a much larger community, and includes about half of the developers. |
|
|||
I don't have much to offer other than some ideas.
What's between the system sending the pings and the firewall? Can you get statistics from any switches or routers and see if there are problems there? Failing that, have you watched the incoming port on the firewall with tcpdump to see if the packets reported as lost by ping ever even made it to the firewall? Does the firewall never see them or does it drop/not reply to them? What if you send traffic from the firewall or behind it out to the other subnet? Have you tried going to the firewall's incoming port from the same subnet (same switch even), eliminating most of the networking infrastructure between the two points? Just some ideas to try narrowing things down. Tim. |
|
||||
thanks for all the suggestions... im not seeing the problem anymore. I work on a large US campus with big juniper networks and aggregators, turns out there was a major network traffic redirect going on at the time - one of the main routes through campus was down and the backup aggregator path was active - it had more hops than the standard path, thats all I can really point to at this moment.
Pinging stuff on the same subnet was fine, no packet loss other subnets were getting packet loss also We also had the good netops folks "roll the fiber" (Reterminate at the switch) and that may have helped. I didnt see anything on the switch logs (had to call netops to get them to look at that) or on the openbsd nic that suggested there was any communication problem. Netops did say something about a "low power signal" over the fiber - not sure what that means - maybe a marginal SFP+ module that couldnt make light bright enough to traverse the uplinl to the agg switch? Who knows anyway, case closed. thanks for playing |
Tags |
10g, network, packet loss, troubleshoot |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
How-To : Vpn IKEv2 between a Windows 7 Road Warrior Host and an OpenBSD gateway | wesley | Guides | 1 | 15th July 2013 04:38 PM |
OpenBSD as host for VirtualBox | gpatrick | OpenBSD General | 15 | 9th May 2012 02:22 AM |
Packet Sniffer for OpenBSD. | bsdnewbie999 | OpenBSD General | 6 | 26th October 2008 02:28 AM |
Problem pinging internal nic | JustDoIt | OpenBSD General | 16 | 19th August 2008 08:16 PM |