I had to do following:
2.Go to App releases -> Manage production -> Create release
3.Click continue on Google Play App Signing
4.Create upload certificate by running "keytool -genkey -v -keystore c:\path\to\cert.keystore -alias uploadKey -keyalg RSA -keysize 2048 -validity 10000"
5.Sign your apk with generated certificate (c:\path\to\cert.keystore)
6.Upload signed apk in App releases -> Manage production -> Edit release
7.By uploading apk, certificate generated in step 4 has been added to App Signing certificates and became your signing cert for all future builds.