I tried once, but I haven't found a way to trick ftp-proxy to run on a single interface.
You could add the ftp rules to a pf anchor and disable them when you don't need ftp. Remember that the ports use ftp
Another option could be to store the allowed ftp sites in a table and make the ftp rules only applicable to these sites. But if you add software by using the FBSD ports mechanism, this becomes very difficult.