A lot of them are libraries. Your window manager (be it gnome, kde, fluxbox, or whatever else) uses those libraries to access the X server and do things on your monitor.
A minimal X install would be xorg-server, enough libraries for xterm to work, and xterm...
Or, if you wanted to get really minimal, you could forgo xterm, though you wouldn't be able to do
anything.