One would think it would be helpful.
It is possible that PF may be getting in the way through either gateway. I haven't examined your PF configurations at all.
The VPN setup, key change management, and VPN teardown is done by isakmpd(8), all via UDP, on port 500 by default. The ESP tunnels don't change routing. So to reach a 10.4 address from a 10.2 address, routing must be established end-to-end.
PF can then be used to restrict traffic to VPN only, if desired.