Debug symbols (.pdb) and XML doc (.xml) files make up a large percentage of the total size and should not be part of the regular deployment package. But it should be possible to access them in case they are needed.
One possible approach: at the end of the TFS build process, move them to a separate artifact.