An excerpt from an apple technical note (Thanks to matthias-bauch)
Xcode includes all your command-line tools. If it is installed on your system, remove it to uninstall your tools.
If your tools were downloaded separately from Xcode, then they are located at
/Library/Developer/CommandLineTools
on your system. Delete the CommandLineTools folder to uninstall them.
you could easily delete using terminal:
Here is an article that explains how to remove the command line tools but do it at your own risk.Try this only if any of the above doesn't work.