If you are getting this error when you run stuffs on automated cluster and you are downloading the stable version of the google chrome every time then you can use the below shell script to download the compatible version of the chrome driver dynamically every time even if the stable version of the chrome gets updated.
%sh
#downloading compatible chrome driver version
#getting the current chrome browser version
**chromeVersion=$(google-chrome --product-version)**
#getting the major version value from the full version
**chromeMajorVersion=${chromeVersion%%.*}**
# setting the base url for getting the release url for the chrome driver
**baseDriverLatestReleaseURL=https://chromedriver.storage.googleapis.com/LATEST_RELEASE_**
#creating the latest release driver url based on the major version of the chrome
**latestDriverReleaseURL=$baseDriverLatestReleaseURL$chromeMajorVersion**
**echo $latestDriverReleaseURL**
#file name of the file that gets downloaded which would contain the full version of the chrome driver to download
**latestDriverVersionFileName="LATEST_RELEASE_"$chromeMajorVersion**
#downloading the file that would contain the full release version compatible with the major release of the chrome browser version
**wget $latestDriverReleaseURL**
#reading the file to get the version of the chrome driver that we should download
**latestFullDriverVersion=$(cat $latestDriverVersionFileName)**
**echo $latestFullDriverVersion**
#creating the final URL by passing the compatible version of the chrome driver that we should download
**finalURL="https://chromedriver.storage.googleapis.com/"$latestFullDriverVersion"/chromedriver_linux64.zip"**
**echo $finalURL**
**wget $finalURL**
I was able to get the compatible version of chrome browser and chrome driver using the above approach when running scheduled job on the databricks environment and it worked like a charm without any issues.
Hope it helps others in one way or other.