CARP(?) issues after upgrading one side from 5.0 to 5.6

[kbeaucha]

I finally got around to upgrading our paired firewall set, and started with the primary. This is a big jump in versions: 5.0 to 5.6

The primary is a Dell, the secondary an HP. They've been paired before and ran fine. I left the secondary at 5.0 and upgraded the primary to 5.6

After the upgrade I booted the primary up and let it assume the states and the isakmpd tunnels we have. Both seemed to transfer cleanly but my users were telling me that the sites at the other end of the tunnels were offline.

In the log files on the primary I was seeing entries saying "got bad state MASTER from peer" - it looked like the secondary was fighting the primary. I tried bumping my advskew up from 100 to 150 but that didn't make a difference.

I logged onto my secondary and downed the CARP interfaces, which stopped the logs and brought my remote sites online.

I'm hoping that this issue will disappear when I update the secondary, but I wanted to ask: Has anyone else seen this problem?

I noticed that there were changes to CARP back in 5.1, but I couldn't find any references to the warning message that was in my logs.

dmesgs follow:

Primary:

OpenBSD 5.6 (GENERIC.MP) #333: Fri Aug  8 00:20:21 MDT 2014
    deraadt@amd64.openbsd.org:/usr/src/s...ile/GENERIC.MP
real mem = 2129338368 (2030MB)
avail mem = 2063933440 (1968MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.5 @ 0x7ff9c000 (46 entries)
bios0: vendor Dell Inc. version "1.0.0" date 10/24/2007
bios0: Dell Inc. PowerEdge R200
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC SPCR HPET MCFG WD__ SLIC ERST HEST BERT EINJ SSDT SSDT SSDT
acpi0: wakeup devices PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM)2 Duo CPU E4500 @ 2.20GHz, 2200.39 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM,NXE,LONG,LAHF,PERF
cpu0: 2MB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 200MHz
cpu0: mwait min=64, max=64, C-substates=0.2.2.0.0, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Core(TM)2 Duo CPU E4500 @ 2.20GHz, 2200.08 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM,NXE,LONG,LAHF,PERF
cpu1: 2MB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 2
acpihpet0 at acpi0: 14318179 Hz
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEX1)
acpiprt2 at acpi0: bus 2 (SBE0)
acpiprt3 at acpi0: bus 6 (SBE4)
acpiprt4 at acpi0: bus 7 (SBE5)
acpiprt5 at acpi0: bus 8 (COMP)
acpicpu0 at acpi0: PSS
acpicpu1 at acpi0: PSS
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 2200 MHz: speeds: 2200, 2000, 1800, 1600, 1400, 1200 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 3200/3210 Host" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 3200/3210 PCIE" rev 0x01: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: msi
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0e
pci3 at ppb2 bus 3
ppb3 at pci3 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0e
pci4 at ppb3 bus 4
em0 at pci4 dev 0 function 0 "Intel 82571EB" rev 0x06: apic 2 int 19, address 00:15:17:63:82:1d
em1 at pci4 dev 0 function 1 "Intel 82571EB" rev 0x06: apic 2 int 18, address 00:15:17:63:82:1c
ppb4 at pci3 dev 4 function 0 "IDT 89HPES12N3A" rev 0x0e
pci5 at ppb4 bus 5
em2 at pci5 dev 0 function 0 "Intel 82571EB" rev 0x06: apic 2 int 17, address 00:15:17:63:82:1f
em3 at pci5 dev 0 function 1 "Intel 82571EB" rev 0x06: apic 2 int 16, address 00:15:17:63:82:1e
ppb5 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02
pci6 at ppb5 bus 6
bge0 at pci6 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1 (0x4201): msi, address 00:19:b9:f9:c0:41
brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
ppb6 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02
pci7 at ppb6 bus 7
bge1 at pci7 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1 (0x4201): msi, address 00:19:b9:f9:c0:42
brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
uhci0 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 2 int 21
uhci1 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 2 int 20
uhci2 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 2 int 21
ehci0 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 2 int 21
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb7 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci8 at ppb7 bus 8
radeondrm0 at pci8 dev 5 function 0 "ATI ES1000" rev 0x02
drm0 at radeondrm0
radeondrm0: apic 2 int 19
pcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02
pciide0 at pci0 dev 31 function 2 "Intel 82801I SATA" rev 0x02: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide0: using apic 2 int 23 for native-PCI interrupt
wd0 at pciide0 channel 0 drive 0: <WDC WD800AAJS-18TDA1>
wd0: 16-sector PIO, LBA48, 76293MB, 156250000 sectors
atapiscsi0 at pciide0 channel 0 drive 1
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <HL-DT-ST, DVD-ROM GDR8084N, 1.01> ATAPI 5/cdrom removable
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6
cd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 2
wd1 at pciide0 channel 1 drive 0: <WDC WD800AAJS-18TDA1>
wd1: 16-sector PIO, LBA48, 76293MB, 156250000 sectors
wd1(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 6
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhub4 at uhub0 port 5 "Cypress Semiconductor USB2 Hub" rev 2.00/0.0b addr 2
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on wd0a (9b4e92d7e67cc8b7.a) swap on wd0b dump on wd0b
drm: initializing kernel modesetting (RV100 0x1002:0x515E 0x1028:0x023C).
radeondrm0: VRAM: 128M 0x00000000D0000000 - 0x00000000D7FFFFFF (32M used)
radeondrm0: GTT: 512M 0x00000000B0000000 - 0x00000000CFFFFFFF
drm: PCI GART of 512M enabled (table at 0x0000000005671000).
error: [drm:pid0:r100_cp_init_microcode] *ERROR* radeon_cp: Failed to load firmware "radeon-r100_cp"
error: [drm:pid0:r100_cp_init] *ERROR* Failed to load firmware!
error: [drm:pid0:r100_startup] *ERROR* failed initializing CP (-2).
error: [drm:pid0:r100_init] *ERROR* Disabling GPU acceleration
drm: radeon: cp finalized
radeondrm0: 1024x768
wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using wskbd0
wsdisplay0: screen 1-5 added (std, vt100 emulation)

Secondary:

OpenBSD 5.0 (GENERIC.MP) #63: Wed Aug 17 10:14:30 MDT 2011
    deraadt@amd64.openbsd.org:/usr/src/s...ile/GENERIC.MP
real mem = 4283621376 (4085MB)
avail mem = 4155469824 (3962MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xdf7fe000 (127 entries)
bios0: vendor HP version "P64" date 10/01/2009
bios0: HP ProLiant DL360 G6
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP SPCR MCFG HPET FFFF SPMI ERST APIC SRAT FFFF BERT HEST DMAR SSDT SSDT SSDT SSDT SSDT
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-63
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz, 2000.35 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: apic clock running at 133MHz
cpu1 at mainbus0: apid 4 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz, 2000.07 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG
cpu1: 256KB 64b/line 8-way L2 cache
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz, 2000.07 MHz
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG
cpu2: 256KB 64b/line 8-way L2 cache
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz, 2000.07 MHz
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG
cpu3: 256KB 64b/line 8-way L2 cache
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 0 pa 0xfec80000, version 20, 24 pins
acpiprt0 at acpi0: bus 1 (IP2P)
acpiprt1 at acpi0: bus -1 (IPT1)
acpiprt2 at acpi0: bus 3 (PT01)
acpiprt3 at acpi0: bus 11 (PT02)
acpiprt4 at acpi0: bus 7 (PT03)
acpiprt5 at acpi0: bus 12 (PT04)
acpiprt6 at acpi0: bus 13 (PT05)
acpiprt7 at acpi0: bus 14 (PT06)
acpiprt8 at acpi0: bus 15 (PT07)
acpiprt9 at acpi0: bus 2 (PT08)
acpiprt10 at acpi0: bus 4 (PT09)
acpiprt11 at acpi0: bus 16 (PT0A)
acpiprt12 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C3, C3, C1
acpicpu1 at acpi0: C3, C3, C1
acpicpu2 at acpi0: C3, C3, C1
acpicpu3 at acpi0: C3, C3, C1
acpitz0 at acpi0: critical temperature is 31 degC
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 5520 Host" rev 0x13
ppb0 at pci0 dev 1 function 0 "Intel X58 PCIE" rev 0x13
pci1 at ppb0 bus 3
ciss0 at pci1 dev 0 function 0 "Hewlett-Packard Smart Array" rev 0x01: apic 0 int 4
ciss0: 1 LD, HW rev 2, FW 2.74/2.74, 64bit fifo rro
scsibus0 at ciss0: 1 targets
sd0 at scsibus0 targ 0 lun 0: <HP, LOGICAL VOLUME, 2.74> SCSI3 0/direct fixed
sd0: 69973MB, 512 bytes/sector, 143305920 sectors
ppb1 at pci0 dev 2 function 0 "Intel X58 PCIE" rev 0x13
pci2 at ppb1 bus 11
ppb2 at pci0 dev 3 function 0 "Intel X58 PCIE" rev 0x13
pci3 at ppb2 bus 7
ppb3 at pci3 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0e
pci4 at ppb3 bus 8
ppb4 at pci4 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0e
pci5 at ppb4 bus 9
em0 at pci5 dev 0 function 0 "Intel PRO/1000 QP (82571EB)" rev 0x06: msi, address 00:15:17:63:75:99
em1 at pci5 dev 0 function 1 "Intel PRO/1000 QP (82571EB)" rev 0x06: msi, address 00:15:17:63:75:98
ppb5 at pci4 dev 4 function 0 "IDT 89HPES12N3A" rev 0x0e
pci6 at ppb5 bus 10
em2 at pci6 dev 0 function 0 "Intel PRO/1000 QP (82571EB)" rev 0x06: msi, address 00:15:17:63:75:9b
em3 at pci6 dev 0 function 1 "Intel PRO/1000 QP (82571EB)" rev 0x06: msi, address 00:15:17:63:75:9a
ppb6 at pci0 dev 4 function 0 "Intel X58 PCIE" rev 0x13
pci7 at ppb6 bus 12
ppb7 at pci0 dev 5 function 0 "Intel X58 PCIE" rev 0x13
pci8 at ppb7 bus 13
ppb8 at pci0 dev 6 function 0 "Intel X58 PCIE" rev 0x13
pci9 at ppb8 bus 14
ppb9 at pci0 dev 7 function 0 "Intel X58 PCIE" rev 0x13
pci10 at ppb9 bus 15
ppb10 at pci0 dev 8 function 0 "Intel X58 PCIE" rev 0x13
pci11 at ppb10 bus 2
bnx0 at pci11 dev 0 function 0 "Broadcom BCM5709" rev 0x20: apic 0 int 7
bnx1 at pci11 dev 0 function 1 "Broadcom BCM5709" rev 0x20: apic 0 int 15
ppb11 at pci0 dev 9 function 0 "Intel X58 PCIE" rev 0x13
pci12 at ppb11 bus 4
ppb12 at pci0 dev 10 function 0 "Intel X58 PCIE" rev 0x13
pci13 at ppb12 bus 16
pchb1 at pci0 dev 13 function 0 vendor "Intel", unknown product 0x343a rev 0x13
pchb2 at pci0 dev 13 function 1 vendor "Intel", unknown product 0x343b rev 0x13
pchb3 at pci0 dev 13 function 2 vendor "Intel", unknown product 0x343c rev 0x13
pchb4 at pci0 dev 13 function 3 vendor "Intel", unknown product 0x343d rev 0x13
pchb5 at pci0 dev 13 function 4 "Intel 5520/X58 QuickPath" rev 0x13
pchb6 at pci0 dev 13 function 5 "Intel 5520 QuickPath" rev 0x13
pchb7 at pci0 dev 13 function 6 vendor "Intel", unknown product 0x341a rev 0x13
pchb8 at pci0 dev 14 function 0 vendor "Intel", unknown product 0x341c rev 0x13
pchb9 at pci0 dev 14 function 1 vendor "Intel", unknown product 0x341d rev 0x13
pchb10 at pci0 dev 14 function 2 vendor "Intel", unknown product 0x341e rev 0x13
pchb11 at pci0 dev 14 function 3 vendor "Intel", unknown product 0x341f rev 0x13
pchb12 at pci0 dev 14 function 4 vendor "Intel", unknown product 0x3439 rev 0x13
"Intel X58 Misc" rev 0x13 at pci0 dev 20 function 0 not configured
"Intel X58 GPIO" rev 0x13 at pci0 dev 20 function 1 not configured
"Intel X58 RAS" rev 0x13 at pci0 dev 20 function 2 not configured
uhci0 at pci0 dev 29 function 0 "Intel 82801JI USB" rev 0x00: apic 8 int 20
uhci1 at pci0 dev 29 function 1 "Intel 82801JI USB" rev 0x00: apic 8 int 23
uhci2 at pci0 dev 29 function 2 "Intel 82801JI USB" rev 0x00: apic 8 int 22
uhci3 at pci0 dev 29 function 3 "Intel 82801JI USB" rev 0x00: apic 8 int 23
ehci0 at pci0 dev 29 function 7 "Intel 82801JI USB" rev 0x00: apic 8 int 20
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb13 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x90
pci14 at ppb13 bus 1
vga1 at pci14 dev 3 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
radeondrm0 at vga1: apic 8 int 23
drm0 at radeondrm0
"Compaq iLO" rev 0x03 at pci14 dev 4 function 0 not configured
"Compaq iLO" rev 0x03 at pci14 dev 4 function 2 not configured
uhci4 at pci14 dev 4 function 4 "Hewlett-Packard USB" rev 0x00: apic 8 int 22
"Hewlett-Packard IPMI" rev 0x00 at pci14 dev 4 function 6 not configured
usb1 at uhci4: USB revision 1.0
uhub1 at usb1 "Hewlett-Packard UHCI root hub" rev 1.00/1.00 addr 1
pcib0 at pci0 dev 31 function 0 "Intel 82801JIB LPC" rev 0x00
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com1: probed fifo depth: 0 bytes
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
mtrr: Pentium Pro MTRR support
uhidev0 at uhub1 port 1 configuration 1 interface 0 "HP Virtual Keyboard" rev 1.10/0.02 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes, country code 33
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub1 port 1 configuration 1 interface 1 "HP Virtual Keyboard" rev 1.10/0.02 addr 2
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (5c44d0c4f88a0485.a) swap on sd0b dump on sd0b
bnx0: address f4:ce:46:b9:15:48
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
bnx1: address f4:ce:46:b9:15:4a
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8

Thanks
kmb

[jggimi]

The "got bad state" message is from sasyncd(8). That tool has had more changes than carp in the 3.5 years between releases.

[kbeaucha]

So, chances are that this will disappear once the two firewalls are running the same version.

I'll probably let the primary run as is and leave the CARP interfaces on the secondary stay down for a week or two, then upgrade the secondary.

Thanks for the feedback.

As a postscript, I just placed my pre-order for 5.7

kmb

[jggimi]

I've ordered my copy too. FYI, in 5.7 carp(4) will require explicit carpdev settings if you are not already using them. That will likely get a mention in the 5.7 Upgrade Guide when published, as it's already discussed in the Following -current FAQ.

http://www.openbsd.org/faq/current.html#20150121

[kbeaucha]

Thanks for the heads-up. I'll update my hostname files.

[kbeaucha]

An update. I brought the secondary firewall up to 5.6 and rebooted it, hoping it would just quietly go into BACKUP mode but it behaved the same as it did when it was still at 5.0 - the primary was logging the "got bad MASTER state from peer" and my remote tunnels seemed to be having trouble.

Since this pair of firewalls used to sync up and nothing has changed in the physical connections since I brought the old primary firewall up on 5.6, I'm guessing that I've got an outdated configuration somewhere and the first place I'm going to look is in sasyncd.conf

[jggimi]

If you're unable to determine the root cause by looking at sasyncd.conf, you might try turning on verbose logging for sasyncd(8), which may help you determine why the state is "bad."

You may get further if you open an informal problem discussion thread on the misc@ mailing list. Its a far larger user community, so you may find someone who has seen this or a similar issue who will offer a solution. I understand about half of the developers subscribe, so you may be asked to conduct additional diagnostics; that can only aid in problem isolation and resolution.

[kbeaucha]

I checked the sasyncd.conf files on both systems. There are just four lines in each (although one had an extra blank line after the "peer" value):

# Comment identifying primary/secondary
interface carp0
peer 129.128.x.x
sharedkey "blah"

The interface and sharedkey lines match, and the peer lines differ in that they each have the address of one of the other firewall's physical interfaces.

I am considering posting this to misc@, but am reluctant to. Just a minor mental block. I'll see about putting together a post.

kmb

[kbeaucha]

I emailed an edited version of this to misc@openbsd.org, but it just kind of disappeared into the ether. It didn't bounce and it didn't appear in my inbox.

[jggimi]

I saw it, it appeared on the list and is in mailing list archives. I thought it was a very well articulated informal problem report.

http://marc.info/?l=openbsd-misc&m=142868387732620&w=2