I've dealt with this for as long as Visual Studio existed. And yes, even in the current version it still fails (especially for large projects.)
I want to share small free tool that my friend and I wrote to address this exact same issue. You basically close your solution, drag its folder into the icon for this tool and it will reset all intermediary files for you. (Read the text manual inside if you want to know which ones. It's not just one file.)
I use it to clean up all my VS projects. So here you go: