If you want to remove derived data manually just run:
rm -rf ~/Library/Developer/Xcode/DerivedData
If you want to free up more disk space there's a few other directories you might want to clear out as well though.
I have created a Bash script for removing all kinds of files generated by Xcode. Removing DerivedData content can be done by running:
./xcode-clean.sh -d
More info at https://github.com/niklasberglund/xcode-clean.sh