Do configure --help and see what other options are available.
It is very common to provide different options to override different locations. By standard, --prefix overrides all of them, so you need to override config location after specifying the prefix. This course of actions usually works for every automake-based project.
The worse case scenario is when you need to modify the configure script, or even worse, generated makefiles and config.h headers. But yeah, for Xfce you can try something like this:
./configure --prefix=/home/me/somefolder/mybuild/output/target --sysconfdir=/etc
I believe that should do it.