Vista is the first OS from Microsoft which has TCP window scaling (defined in RFC 1323) enabled as default.
To properly deal with this, a stateful packetfilter has to create state on the first packet of the 3 way TCP handshake, where this scaling is proposed by the TCP connection initiatior.
Your
pass out keep state rule violates this principle.
The issue is rather clearly explained in
http://undeadly.org/cgi?action=artic...20060928081238 under the section "Create TCP states on the initial SYN packet"
The release of Vista has been one of the reasons why for some time
pf now defaults to
flags S/SA keep state for stateful TCP rules.
Don't know if the OS you run, already has this as default
In OpenBSD you can disable this TCP window scaling by setting the sysctl varibiable
net.inet.tcp.rfc1323=1 to 0.
In Vista it probably is hidden somewhere in the registry