As you described your goal, it appeared to me that you wished to have someone at any external IP address establish a tunnel to a local address,
then use that local address as an initiation for further communication outbound. That's not the picture you drew, nor does it match the configuration files and output that you shared with us.
Did I understand what you wanted to accomplish? If so, IPSec alone won't provide that. You will need to establish tunnels within an IPSec flow, and
gif(4) would be one likely candidate. The gif(4) man page has an example of this using bridge(4) and the etherip protocol.
The reason you need additional tunnels is because IPSec uses flows to determine whether to apply IPSec to a packet, and Security Associations (SAs) to determine the various IPSec options to apply to a packet within a flow. By itself, it doesn't provide for the "local virtual IP address for a road warrior" that you apparently need.