[python] Sending "User-agent" using Requests library in Python

I want to send a value for "User-agent" while requesting a webpage using Python Requests. I am not sure is if it is okay to send this as a part of the header, as in the code below:

debug = {'verbose': sys.stderr}
user_agent = {'User-agent': 'Mozilla/5.0'}
response  = requests.get(url, headers = user_agent, config=debug)

The debug information isn't showing the headers being sent during the request.

Is it acceptable to send this information in the header? If not, how can I send it?

This question is related to python web-crawler python-requests

The answer is


It's more convenient to use a session, this way you don't have to remember to set headers each time:

session = requests.Session()
session.headers.update({'User-Agent': 'Custom user agent'})

session.get('https://httpbin.org/headers')

By default, session also manages cookies for you. In case you want to disable that, see this question.


Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to web-crawler

TypeError: can't use a string pattern on a bytes-like object in re.findall() Finding the layers and layer sizes for each Docker image Sending "User-agent" using Requests library in Python How to find sitemap.xml path on websites? How to request Google to re-crawl my website? python: [Errno 10054] An existing connection was forcibly closed by the remote host How to get a web page's source code from Java Python: maximum recursion depth exceeded while calling a Python object Python Web Crawlers and "getting" html source code How do I make a simple crawler in PHP?

Examples related to python-requests

Requests (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.") Error in PyCharm requesting website Use python requests to download CSV Download and save PDF file with Python requests module ImportError: No module named 'Queue' How to use cookies in Python Requests Saving response from Requests to file How to get Python requests to trust a self signed SSL certificate? How to install requests module in Python 3.4, instead of 2.7 Upload Image using POST form data in Python-requests SSL InsecurePlatform error when using Requests package