In my case I noticed that disabling my internet connection would make it run as fast as with ctrl-f5, so I went to debug->options->symbols and just unchecked all .pdb locations.
Seems like VS was trying to connect to these servers every time a debug session was launched.
Note that disabling Debug->Options->Debugging->General "Enable source support" or "Require source files to exactly match the original version" wouldn't make any difference.