I followed everything from here: https://cloud.google.com/compute/docs/instances/connecting-to-instance#generatesshkeypair
But still there was an error and SSH keys in my instance metadata wasn't getting recognized.
Solution: Check if your ssh key has any new-line. When I copied my public key using cat, it added into-lines into the key, thus breaking the key. Had to manually check any line-breaks and correct it.