Hmm... you should really just use pf. :P
With pf you can specify that if somebody tries to connect to your machine X times in Y seconds, pf will block it from there on.
E.g. with 3 lines of code, I can make it so anyone can connect to my server by default, and allow all outgoing connection:
table <brute> persist
block in
pass in from !<brute> keep state (max-src-conn 50, max-src-conn-rate 50/5, overload <brute> flush global)
pass out
Anyone tries to make a 50 new connection in 5 seconds will be blocked by pf automatically (max-src-conn-rate 50/5), and the same if they make any more than 50 connections at any one time (max-src-conn 50). Of course you will need to adjust these accordingly.
And yea if you just copy and paste those 3 lines into /etc/pf.conf, and turn on pf, it should just works.
Look at
http://www.bgnett.no/~peter/pf/en/bruteforce.html for more details.