You're misunderstanding pkg_info(1) -Q, or misunderstanding $PKG_PATH:
Code:
-Q query
Show all packages in $PKG_PATH which match the given query.
Instead, try
$ pkg_info | grep xfce. You'll get the empty set. And, now that you've uninstalled the manually added xfce, you can also remove all unneeded dependencies with
# pkg_delete -a.