How can I save all cookies in Python's Selenium WebDriver to a .txt file, and then load them later?
The documentation doesn't say much of anything about the getCookies function.
This question is related to
python
python-2.7
selenium
webdriver
Based on the answer by Eduard Florinescu, but with newer code and the missing imports added:
$ cat work-auth.py
#!/usr/bin/python3
# Setup:
# sudo apt-get install chromium-chromedriver
# sudo -H python3 -m pip install selenium
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--user-data-dir=chrome-data")
driver = webdriver.Chrome('/usr/bin/chromedriver',options=chrome_options)
chrome_options.add_argument("user-data-dir=chrome-data")
driver.get('https://www.somedomainthatrequireslogin.com')
time.sleep(30) # Time to enter credentials
driver.quit()
$ cat work.py
#!/usr/bin/python3
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--user-data-dir=chrome-data")
driver = webdriver.Chrome('/usr/bin/chromedriver',options=chrome_options)
driver.get('https://www.somedomainthatrequireslogin.com') # Already authenticated
time.sleep(10)
driver.quit()
Remember, you can only add a cookie for the current domain.
If you want to add a cookie for your Google account, do
browser.get('http://google.com')
for cookie in cookies:
browser.add_cookie(cookie)
Just a slight modification for the code written by Roel Van de Paar, as all credit goes to him. I am using this in Windows and it is working perfectly, both for setting and adding cookies:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--user-data-dir=chrome-data")
driver = webdriver.Chrome('chromedriver.exe',options=chrome_options)
driver.get('https://web.whatsapp.com') # Already authenticated
time.sleep(30)
This is a solution that saves the profile directory for Firefox (similar to the user-data-dir
(user data directory) in Chrome) (it involves manually copying the directory around. I haven't been able to find another way):
It was tested on Linux.
Short version:
driver.execute_script("window.close()")
time.sleep(0.5)
currentProfilePath = driver.capabilities["moz:profile"]
profileStoragePath = "/tmp/abc"
shutil.copytree(currentProfilePath, profileStoragePath,
ignore_dangling_symlinks=True
)
driver = Firefox(executable_path="geckodriver-v0.28.0-linux64",
firefox_profile=FirefoxProfile(profileStoragePath)
)
Long version (with demonstration that it works and a lot of explanation -- see comments in the code)
The code uses localStorage
for demonstration, but it works with cookies as well.
#initial imports
from selenium.webdriver import Firefox, FirefoxProfile
import shutil
import os.path
import time
# Create a new profile
driver = Firefox(executable_path="geckodriver-v0.28.0-linux64",
# * I'm using this particular version. If yours is
# named "geckodriver" and placed in system PATH
# then this is not necessary
)
# Navigate to an arbitrary page and set some local storage
driver.get("https://DuckDuckGo.com")
assert driver.execute_script(r"""{
const tmp = localStorage.a; localStorage.a="1";
return [tmp, localStorage.a]
}""") == [None, "1"]
# Make sure that the browser writes the data to profile directory.
# Choose one of the below methods
if 0:
# Wait for some time for Firefox to flush the local storage to disk.
# It's a long time. I tried 3 seconds and it doesn't work.
time.sleep(10)
elif 1:
# Alternatively:
driver.execute_script("window.close()")
# NOTE: It might not work if there are multiple windows!
# Wait for a bit for the browser to clean up
# (shutil.copytree might throw some weird error if the source directory changes while copying)
time.sleep(0.5)
else:
pass
# I haven't been able to find any other, more elegant way.
#`close()` and `quit()` both delete the profile directory
# Copy the profile directory (must be done BEFORE driver.quit()!)
currentProfilePath = driver.capabilities["moz:profile"]
assert os.path.isdir(currentProfilePath)
profileStoragePath = "/tmp/abc"
try:
shutil.rmtree(profileStoragePath)
except FileNotFoundError:
pass
shutil.copytree(currentProfilePath, profileStoragePath,
ignore_dangling_symlinks=True # There's a lock file in the
# profile directory that symlinks
# to some IP address + port
)
driver.quit()
assert not os.path.isdir(currentProfilePath)
# Selenium cleans up properly if driver.quit() is called,
# but not necessarily if the object is destructed
# Now reopen it with the old profile
driver=Firefox(executable_path="geckodriver-v0.28.0-linux64",
firefox_profile=FirefoxProfile(profileStoragePath)
)
# Note that the profile directory is **copied** -- see FirefoxProfile documentation
assert driver.profile.path!=profileStoragePath
assert driver.capabilities["moz:profile"]!=profileStoragePath
# Confusingly...
assert driver.profile.path!=driver.capabilities["moz:profile"]
# And only the latter is updated.
# To save it again, use the same method as previously mentioned
# Check the data is still there
driver.get("https://DuckDuckGo.com")
data = driver.execute_script(r"""return localStorage.a""")
assert data=="1", data
driver.quit()
assert not os.path.isdir(driver.capabilities["moz:profile"])
assert not os.path.isdir(driver.profile.path)
What doesn't work:
Firefox(capabilities={"moz:profile": "/path/to/directory"})
-- the driver will not be able to connect.options=Options(); options.add_argument("profile"); options.add_argument("/path/to/directory"); Firefox(options=options)
-- same as above.When you need cookies from session to session, there is another way to do it. Use the Chrome options user-data-dir in order to use folders as profiles. I run:
# You need to: from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("user-data-dir=selenium")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get("www.google.com")
Here you can do the logins that check for human interaction. I do this and then the cookies I need now every time I start the Webdriver with that folder everything is in there. You can also manually install the Extensions and have them in every session.
The second time I run, all the cookies are there:
# You need to: from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("user-data-dir=selenium")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get("www.google.com") # Now you can see the cookies, the settings, extensions, etc., and the logins done in the previous session are present here.
The advantage is you can use multiple folders with different settings and cookies, Extensions without the need to load, unload cookies, install and uninstall Extensions, change settings, change logins via code, and thus no way to have the logic of the program break, etc.
Also, this is faster than having to do it all by code.
This is code I used in Windows. It works.
for item in COOKIES.split(';'):
name,value = item.split('=', 1)
name=name.replace(' ', '').replace('\r', '').replace('\n', '')
value = value.replace(' ', '').replace('\r', '').replace('\n', '')
cookie_dict={
'name':name,
'value':value,
"domain": "", # Google Chrome
"expires": "",
'path': '/',
'httpOnly': False,
'HostOnly': False,
'Secure': False
}
self.driver_.add_cookie(cookie_dict)
Source: Stackoverflow.com