Step 1:
Add .vs/ to your .gitignore file (as said in other answers).
Step 2:
It is important to understand, that step 1 WILL NOT remove files within .vs/ from your current branch index, if they have already been added to it. So clear your active branch by issuing:
git rm --cached -r .vs/*
Step 3:
Best to immediately repeat steps 1 and 2 for all other active branches of your project as well.
Otherwise you will easily face the same problems again when switching to an uncleaned branch.
Pro tip:
Instead of step 1 you may want to to use this official .gitingore template for VisualStudio that covers much more than just the .vs path:
https://github.com/github/gitignore/blob/master/VisualStudio.gitignore
(But still don't forget steps 2 and 3.)