Unfortunately, avoiding this aperture is difficult.. but once it's opened by a running X server no other process has direct access.
In the future when KMS is available for more vendors/chipsets, I believe the X aperture will no longer be required.. only
drm(4).
If you're using a workstation, you can have
xdm(1) run at boot to securely open the aperture device.. non-root users have no directly permissions to read/write to it either, if a user had root they could do more harm using a plethora of other methods.
For non-vesa, it's sometimes possible to set
machdep.allowaperture to 1 instead of 2, this does work for some drivers/chipsets but not all, please read
xf86(4) to appreciate complexity of the problem.
You have no choice other than to not use X.