In the OpenBSD world, such unnecessary "optimizations" leave the user with an unsupported kernel.
But as with all of the BSD projects, an unmodified kernel will not slow things down at all.. initial device discovery/probing is usually quite fast, all you could possibly gain is some negligible amount of free space.. a couple MB max shaved off of the final kernel binary.
FreeBSD makes use of loadable kernel modules, so presumably some drivers can be made separate.. I do not know much about this functionality.
Hope that helps.