Xorg is necessary, everything graphical runs atop of it on Unix-like systems... some utilities you don't use personally, may be used by some other 3rd party program.
Just deal with it, the overall size of Xorg is small considering the capacity of modern drives.
