Another possible cause of this is trying to use the ;x509; module on something that is not X.509.
The server certificate is X.509 format, but the private key is RSA.
So:
openssl rsa -noout -text -in privkey.pem
openssl x509 -noout -text -in servercert.pem