If I've understood the problem correctly, you are still able to use X, only without 3D or acceleration. If you are not able to use X, I believe you have two choices:
- write a custom xorg.conf to force the VESA driver to be used, or
- disable radeondrm via config(8), which will cause X to automatically use the VESA driver.