Just tell git to not assume it is changed never matter what:
git update-index --assume-unchanged src/file/to/ignore
yes, you can remove the files from the git repository. But if your team all use the same IDE or you are by yourself, you probably don't want to do that. For yourself, you want to have an ok starting point to resume working, for your teammates as well.