I'd make allowance for guaranteed highest priority bandwidth to the IP acks and other signaling traffic for the whole wire (subnet), regardless of host origins.
Code:
pass in quick on $int_if inet proto { tcp udp } \
from ($int_if:network) to any \
queue (up_host1, up_highest)
You'll have to fit
up_highest into your queue schema.
/S