Here is a simple pair of PF rules which may work. The first rule is the general case, blocking all traffic to the inbound ssh port. It would not be necessary if you already have a default block rule in place. The second rule is the specific case, permitting traffic to pass.
Code:
block proto tcp from any to self port ssh
pass proto tcp from 10.255.255.101 to self port ssh
More information can be found in the PF User's Guide.