I don't know if this will be helpful for someone, but I had the same problem.
I wrote pip install xlrd
in the anaconda prompt while in the specific environment and it said it was installed, but when I looked at the installed packages it wasn't there.
What solved the problem was "moving" (I don't know the terminology for it) into the Scripts
folder of the specific environment and do the pip install xlrd
there.
Hope this is useful for someone :D