Ivan Branets 's solution is essentially what fixed this for me, but some more details could be shared.
In my case, I was in VS 2015 using Auto Package restore and TFS. This is all pretty default stuff.
The problem was that when another developer tried to get a solution from TFS, some packages were not fully getting restored. (Why, that I'm not quite sure yet.) But the packages folder contained a folder for the reference and the NuGet package, but it wasn't getting expanded (say a lib folder containing a .dll was missing.) This half there, but not really quite right concept was preventing the package from being restored.
You'll recognize this because the reference will have a yellow exclamation point about not being resolved.
So, the solution of deleting the folder within packages removes the package restore blocking issue. Then you can right click at the top solution level to get the option to restore packages, and now it should work.