By user1289853

2012-05-15 17:48:44 8 Comments

I want to send a value for "User-agent" while requesting a webpage using Python Requests. I am not sure is if it is okay to send this as a part of the header, as in the code below:

debug = {'verbose': sys.stderr}
user_agent = {'User-agent': 'Mozilla/5.0'}
response  = requests.get(url, headers = user_agent, config=debug)

The debug information isn't showing the headers being sent during the request.

Is it acceptable to send this information in the header? If not, how can I send it?


@user 2017-06-14 04:16:13

It's more convenient to use a session, this way you don't have to remember to set headers each time:

session = requests.Session()
session.headers.update({'User-Agent': 'Custom user agent'})


By default, session also manages cookies for you. In case you want to disable that, see this question.

@wkl 2012-05-15 17:58:21

The user-agent should be specified as a field in the header.

Here is a list of HTTP header fields, and you'd probably be interested in request-specific fields, which includes User-Agent.

If you're using requests v2.13 and newer

The simplest way to do what you want is to create a dictionary and specify your headers directly, like so:

import requests

url = 'SOME URL'

headers = {
    'User-Agent': 'My User Agent 1.0',
    'From': '[email protected]'  # This is another valid field

response = requests.get(url, headers=headers)

If you're using requests v2.12.x and older

Older versions of requests clobbered default headers, so you'd want to do the following to preserve default headers and then add your own to them.

import requests

url = 'SOME URL'

# Get a copy of the default headers that requests would use
headers = requests.utils.default_headers()

# Update the headers with your custom ones
# You don't have to worry about case-sensitivity with
# the dictionary keys, because default_headers uses a custom
# CaseInsensitiveDict implementation within requests' source code.
        'User-Agent': 'My User Agent 1.0',

response = requests.get(url, headers=headers)

@here 2014-05-04 04:29:22

You can also access the headers you sent with response.request.headers , this works because the original request object is an attribute of the response object. See also‌​uest-and-response-ob‌​jects

@nealmcb 2015-10-30 17:00:52

The default value is also available as requests.utils.default_user_agent() if you want to just augment that with your own info.

@Chad Miller 2016-10-14 17:17:29

It's not correct. It clobbers the rest of the headers. He should get a copy of defaults from requests.utils.default_user_agent() and update it, and send those.

@erm3nda 2016-12-23 06:53:40

for easyness, on (downloadable stuff) you can get the browser headers then make your query appear you

@Jmills 2017-05-31 00:02:20

At least in 2.13.0, the headers are not clobbered and the docs just tell you to use the headers kwarg.

Related Questions

Sponsored Content

33 Answered Questions

[SOLVED] How to get the current time in Python

  • 2009-01-06 04:54:23
  • user46646
  • 2888936 View
  • 2488 Score
  • 33 Answer
  • Tags:   python datetime time

21 Answered Questions

[SOLVED] Does Python have a ternary conditional operator?

28 Answered Questions

[SOLVED] Finding the index of an item given a list containing it in Python

  • 2008-10-07 01:39:38
  • Eugene M
  • 3361137 View
  • 2776 Score
  • 28 Answer
  • Tags:   python list indexing

16 Answered Questions

[SOLVED] What are metaclasses in Python?

12 Answered Questions

[SOLVED] How can I make a time delay in Python?

25 Answered Questions

[SOLVED] Is there a way to run Python on Android?

57 Answered Questions

[SOLVED] Calling an external command in Python

10 Answered Questions

[SOLVED] Does Python have a string 'contains' substring method?

11 Answered Questions

[SOLVED] How to substring a string in Python?

  • 2009-03-19 17:29:41
  • Joan Venge
  • 2599886 View
  • 1925 Score
  • 11 Answer
  • Tags:   python string

7 Answered Questions

[SOLVED] Understanding Python super() with __init__() methods

Sponsored Content