By user1289853

2012-05-15 17:48:44 8 Comments

I want to send a value for "User-agent" while requesting a webpage using Python Requests. I am not sure is if it is okay to send this as a part of the header, as in the code below:

debug = {'verbose': sys.stderr}
user_agent = {'User-agent': 'Mozilla/5.0'}
response  = requests.get(url, headers = user_agent, config=debug)

The debug information isn't showing the headers being sent during the request.

Is it acceptable to send this information in the header? If not, how can I send it?


@user 2017-06-14 04:16:13

It's more convenient to use a session, this way you don't have to remember to set headers each time:

session = requests.Session()
session.headers.update({'User-Agent': 'Custom user agent'})


By default, session also manages cookies for you. In case you want to disable that, see this question.

@wkl 2012-05-15 17:58:21

The user-agent should be specified as a field in the header.

Here is a list of HTTP header fields, and you'd probably be interested in request-specific fields, which includes User-Agent.

If you're using requests v2.13 and newer

The simplest way to do what you want is to create a dictionary and specify your headers directly, like so:

import requests

url = 'SOME URL'

headers = {
    'User-Agent': 'My User Agent 1.0',
    'From': '[email protected]'  # This is another valid field

response = requests.get(url, headers=headers)

If you're using requests v2.12.x and older

Older versions of requests clobbered default headers, so you'd want to do the following to preserve default headers and then add your own to them.

import requests

url = 'SOME URL'

# Get a copy of the default headers that requests would use
headers = requests.utils.default_headers()

# Update the headers with your custom ones
# You don't have to worry about case-sensitivity with
# the dictionary keys, because default_headers uses a custom
# CaseInsensitiveDict implementation within requests' source code.
        'User-Agent': 'My User Agent 1.0',

response = requests.get(url, headers=headers)

@here 2014-05-04 04:29:22

You can also access the headers you sent with response.request.headers , this works because the original request object is an attribute of the response object. See also‌​uest-and-response-ob‌​jects

@nealmcb 2015-10-30 17:00:52

The default value is also available as requests.utils.default_user_agent() if you want to just augment that with your own info.

@Chad Miller 2016-10-14 17:17:29

It's not correct. It clobbers the rest of the headers. He should get a copy of defaults from requests.utils.default_user_agent() and update it, and send those.

@erm3nda 2016-12-23 06:53:40

for easyness, on (downloadable stuff) you can get the browser headers then make your query appear you

@Jmills 2017-05-31 00:02:20

At least in 2.13.0, the headers are not clobbered and the docs just tell you to use the headers kwarg.

Related Questions

Sponsored Content

39 Answered Questions

[SOLVED] How to make a flat list out of list of lists?

26 Answered Questions

25 Answered Questions

[SOLVED] How can I safely create a nested directory in Python?

40 Answered Questions

[SOLVED] How do I check whether a file exists without exceptions?

18 Answered Questions

[SOLVED] Does Python have a string 'contains' substring method?

55 Answered Questions

[SOLVED] Calling an external command in Python

26 Answered Questions

[SOLVED] Difference between append vs. extend list methods in Python

28 Answered Questions

[SOLVED] Finding the index of an item given a list containing it in Python

  • 2008-10-07 01:39:38
  • Eugene M
  • 3080323 View
  • 2543 Score
  • 28 Answer
  • Tags:   python list

32 Answered Questions

[SOLVED] How to get the current time in Python

  • 2009-01-06 04:54:23
  • user46646
  • 2519313 View
  • 2205 Score
  • 32 Answer
  • Tags:   python datetime time

15 Answered Questions

[SOLVED] What are metaclasses in Python?

Sponsored Content