You should be pointing it towards the Developer
directory, not the Xcode application bundle. Run this:
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
With recent versions of Xcode, you can go to Xcode ? Preferences… ? Locations and pick one of the options for Command Line Tools to set the location.