Have seen the OpenBSD man page for starttls? Are the permissons on the certificates OK?
I would first try to get STARTTLS working and then adding in SMTP AUTH.
Some pitfalls are discussed in
http://herolsen.org/2009/OpenBSDSMTPS.html
I am afraid I cannot be of much further help
