The DLL must be present at all times - as the name indicates, a reference only tells VS that you're trying to use stuff from the DLL. In the project file, VS stores the actual path and file name of the referenced DLL. If you move or delete it, VS is not able to find it anymore.
I usually create a libs
folder within my project's folder where I copy DLLs that are not installed to the GAC. Then, I actually add this folder to my project in VS (show hidden files in VS, then right-click and "Include in project"). I then reference the DLLs from the folder, so when checking into source control, the library is also checked in. This makes it much easier when more than one developer will have to change the project.
(Please make sure to set the build type to "none" and "don't copy to output folder" for the DLL in your project.)
PS: I use a German Visual Studio, so the captions I quoted may not exactly match the English version...