Configure in Tools, Options, Debugging, Symbols.
You can watch the output window (view, output) to see what it's doing usually. If it's really slow that probably means it's hitting a symbol server, probably Microsoft's, to download missing symbols. This takes three HTTP hits for each file it can't find on every startup - you can sometimes see this in the status bar at the bottom or in e.g. Fiddler. You can see which modules have loaded symbols in Debug, Windows, Modules whilst you're debugging.
Symbols mean you get useful stack trace information into third party and system assemblies. You definitely need them for your own code, but I think those get loaded regardless. Your best bet is to turn off any non-local symbol sources in that menu and, if you're loading lots of symbols for system assemblies that you don't need to debug into you can temporarily disable loading those to speed up debug start - but they're often useful to have loaded.