By user1289853

2012-05-15 17:48:44 8 Comments

I want to send a value for "User-agent" while requesting a webpage using Python Requests. I am not sure is if it is okay to send this as a part of the header, as in the code below:

debug = {'verbose': sys.stderr}
user_agent = {'User-agent': 'Mozilla/5.0'}
response  = requests.get(url, headers = user_agent, config=debug)

The debug information isn't showing the headers being sent during the request.

Is it acceptable to send this information in the header? If not, how can I send it?


@user 2017-06-14 04:16:13

It's more convenient to use a session, this way you don't have to remember to set headers each time:

session = requests.Session()
session.headers.update({'User-Agent': 'Custom user agent'})


By default, session also manages cookies for you. In case you want to disable that, see this question.

@wkl 2012-05-15 17:58:21

The user-agent should be specified as a field in the header.

Here is a list of HTTP header fields, and you'd probably be interested in request-specific fields, which includes User-Agent.

If you're using requests v2.13 and newer

The simplest way to do what you want is to create a dictionary and specify your headers directly, like so:

import requests

url = 'SOME URL'

headers = {
    'User-Agent': 'My User Agent 1.0',
    'From': '[email protected]'  # This is another valid field

response = requests.get(url, headers=headers)

If you're using requests v2.12.x and older

Older versions of requests clobbered default headers, so you'd want to do the following to preserve default headers and then add your own to them.

import requests

url = 'SOME URL'

# Get a copy of the default headers that requests would use
headers = requests.utils.default_headers()

# Update the headers with your custom ones
# You don't have to worry about case-sensitivity with
# the dictionary keys, because default_headers uses a custom
# CaseInsensitiveDict implementation within requests' source code.
        'User-Agent': 'My User Agent 1.0',

response = requests.get(url, headers=headers)

@here 2014-05-04 04:29:22

You can also access the headers you sent with response.request.headers , this works because the original request object is an attribute of the response object. See also‌​uest-and-response-ob‌​jects

@nealmcb 2015-10-30 17:00:52

The default value is also available as requests.utils.default_user_agent() if you want to just augment that with your own info.

@Chad Miller 2016-10-14 17:17:29

It's not correct. It clobbers the rest of the headers. He should get a copy of defaults from requests.utils.default_user_agent() and update it, and send those.

@erm3nda 2016-12-23 06:53:40

for easyness, on (downloadable stuff) you can get the browser headers then make your query appear you

@Jmills 2017-05-31 00:02:20

At least in 2.13.0, the headers are not clobbered and the docs just tell you to use the headers kwarg.

Related Questions

Sponsored Content

56 Answered Questions

[SOLVED] Calling an external command in Python

27 Answered Questions

39 Answered Questions

[SOLVED] How to make a flat list out of list of lists

15 Answered Questions

[SOLVED] What are metaclasses in Python?

28 Answered Questions

42 Answered Questions

[SOLVED] How do I check whether a file exists without exceptions?

29 Answered Questions

[SOLVED] Finding the index of an item given a list containing it in Python

  • 2008-10-07 01:39:38
  • Eugene M
  • 3203553 View
  • 2642 Score
  • 29 Answer
  • Tags:   python list

25 Answered Questions

[SOLVED] How can I safely create a nested directory in Python?

33 Answered Questions

[SOLVED] How to get the current time in Python

  • 2009-01-06 04:54:23
  • user46646
  • 2649604 View
  • 2316 Score
  • 33 Answer
  • Tags:   python datetime time

17 Answered Questions

[SOLVED] Does Python have a string 'contains' substring method?

Sponsored Content