I'm not sure I was sufficiently clear, so I'll try to add more information.
Per your pf.conf, all outbound traffic is currently permitted, regardless of source. But traffic inbound is only permitted on the internal network for a limited set of UDP and TCP destination ports. No inbound traffic from the external interface is permitted, unless applicable to an existing state.
- ESP needs to be passed, both directions. At the moment, it's not permitted at all.
- UDP destination ports 500 and 4500 need to be passed in both directions
While I'm not sure what NAT traversal techniques might be needed other than merely having destination port 4500 open in both directions...it's my belief (without testing) that you won't need more than that.