looks like my NSServers were attempting to use vhids 1 and 2 - they also had a advskew of 0, which could account for the high numbers in the netstat -ss.
i moved them to two unused vhids and bumped up the advskew to 20. It'll take a couple hours to see if that works. Intuition tells me that was part of the issue.
a side note,
my cross-over cable interface [bge0]
on one machine the media options show
Code:
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:10:18:14:8a:6a
priority: 0
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
inet 192.168.1.2 netmask 0xfffffffc broadcast 192.168.1.3
inet6 fe80::210:18ff:fe14:8a6a%bge0 prefixlen 64 scopeid 0x1
The "master" part, is b/c its a cross over (the pest01 interface doesn't show as master)? If so, how does it decide who is master. I couldn't find reference to this on the web, except that state pointed to carp interfaces.