What is the best way to do it?

By uninstalling the corresponding packages, I guess … Don’t know in detail how Debian works nowadays, but there will likely a meta package or a package group you can remove to remove Xfce with all dependencies