You can't do it easily. In theory, one could build and install a 32-bit world, as you should be able to run a 32-bit userland on top of a 64-bit kernel. Then you could build and install a 32-bit kernel. Then rebuilt all your ports.
However, considering you have to rebuild the entire OS, as well as all the apps, anyway, you may as well just reinstall from the CD.
And the bootonly CD is only a handful of megs, not like it'd take a day to download.