I think you may have a configuration problem of some kind. When you start an X session, the drm device is chowned:
Code:
0 crw------- 1 josh wheel 88, 0 Jun 26 20:15 drm0
0 crw------- 1 root wheel 88, 1 Jun 26 20:15 drm1
0 crw------- 1 root wheel 88, 2 Jun 26 20:15 drm2
0 crw------- 1 root wheel 88, 3 Jun 26 20:15 drm3
How are you starting your X session? xinit? startx? xdm? other?