@Carpetsmoker: Sorry I should have been clearer in my description. It works fine the way you have it, but not if I invert the selection with a '!'. As there are only a few servers I wish to exclude from the filtering (rather than include) it would be easier this way around.
@DutchDaemon: Yeah that was the way I tried it initially but it gives a config error if I put ! before the list name. Putting the ! in the list was the only way I could launch pf successfully.
Any other thoughts?