If you want to shut off a connection after so many bytes, you'll have to do it via an manual examination of PF state tables. This might be as simple as a cron job that examines the output of # pfctl -vs state.
I used to do this with a cron job, to limit script kiddies from looping stupid ftpd attacks on "User Administrator" or other common userids that did not exist. I found a simple modification to ftpd can drop those sessions, and that's a much easier solution.
I submitted it as a patch last month. It was partially accepted, but what was accepted did not stop kiddie scripts from looping, so I still have a slightly modified ftpd.
PF can handle state creations via stateful tracking options, but it only direct limits already established and valid sessions -- not killed via stateful "overload flush" -- via traffic shaping from queue management. You have choices here, such as simple packet priority, bps, or percentage of bps of a parent queue.