You have to start the
vncserver before you can connect, otherwise there is nothing to connect to !
Without configuring some thing to handle such issues (probably
inetd on Linux/BSD/Unix systems), there is no way for any computer to know what service you want started on receiving a request at port XYZ from host ZYX -- it's common sense. So without doing some thing like that you'll need to have the service running before connecting.
I don't see any rc scripts in the ports pkg-plist files so you probably will need to create them in /usr/local/etc/rc.d or add an entry to /etc/rc.local if you want to start the service (and leave it running) when ever the machine starts. rc.local is a shell script, the rc.d system is a system of shell scripts -> read the rc man page for details.
I used VNC on a headless machine ~two years ago and it worked fine. If you don't want to use TWM on login, you probably want to edit (or create) the ~/.vnc/xstartup shell script for the user on the VNC Server machine to handle launching the environment you want.
I used to use KDE via VNC since at the time I required my only monitor for gaming, meaning it was attached to a Windows machine ;-).
Personally I find SSH better then VNC or running an X Server whether the client is a Microsoft or Unix based system but to each their own. VNC + TWM however does work better over the network then say, VNC + Windows NT or VNC+KDE which are heavier on performance then TWM.