I had exact issue and here is how I fixed:
I found out that I had first installed Keras then installed pandas in my virtual env. When you install keras, pandas is shipped with it. Do not need to pip install pandas.
I tested this hypothesis by creating new virtual environment and wala... pandas appeared without me installing it. Thus I came to the conclusion that pandas is automatically installed when you pip install keras.