Packets are being blocked. The first one is the first of the 3-way TCP handshake to set up a TCP connection. The others are blocked UDP packets.
Add this rule and retry.
Code:
pass out quick on egress inet proto { tcp, udp } to VPN_IP port VPN_port
BTW telling us which port you are using for VPN could give us a clue We are not interested in the IP address, only the port