I am attempting to extract a specific request header using python, the 'plib' to be exact after logging into my account to automate the login. I have successful logged in and printed out all the request headers using seleniumwire but i need one saved to a variable.
for request in driver.requests:
#print(request.url) # <--------------- Request url
print(request.headers) # <----------- Request headers
#print(request.response.headers) # <-- Response headers
that is what i am using to print all the request header but i just need one. Can someone please assist ?
Thanks,
You can index into the headers property like you would with a dictionary.
print(request.headers["plib"])
To quote from the documentation:
headers
A dictionary-like object of request headers. Headers are case-insensitive and duplicates are permitted. Asking for
request.headers['user-agent'] will return the value of the User-Agent
header. If you wish to replace a header, make sure you delete the
existing header first with del request.headers['header-name'],
otherwise you’ll create a duplicate.
Related
I try to read JSON-formatted data from the following public URL: http://ws-old.parlament.ch/factions?format=json. Unfortunately, I was not able to convert the response to JSON as I always get the HTML-formatted content back from my request. Somehow the request seems to completely ignore the parameters for JSON formatting passed with the URL:
import urllib.request
response = urllib.request.urlopen('http://ws-old.parlament.ch/factions?format=json')
response_text = response.read()
print(response_text) #why is this HTML?
Does somebody know how I am able to get the JSON formatted content as displayed in the web browser?
You need to add "Accept": "text/json" to request header.
For example using requests package:
r = requests.get(r'http://ws-old.parlament.ch/factions?format=json',
headers={'Accept':'text/json'})
print(r.json())
Result:
[{'id': 3, 'updated': '2022-02-22T14:59:17Z', 'abbreviation': ...
Sorry for you but these web services have a misleading implementation. The format query parameter is useless. As pointed out by #maciek97x only the header Accept: <format> will be considered for the formatting.
So your can directly call the endpoint without the ?format=json but with the header Accept: text/json.
I'm sending a POST request, with python-requests in Python 3.5, using:
r = requests.post(apiEndpoint, data=jsonPayload, headers=headersToMergeIn)
I can inspect the headers and body after sending the request like this:
print(r.request.headers)
print(r.request.body)
Is there any way to inspect the full request headers (not just the ones i'm merging in) and body before sending the request?
Note: I need this for an api which requires me to build a hash off of a subset of the headers, and another hash off the full body.
You probably want Prepared Requests
I am a python newbie and i have a controler that get Post requests.
I try to print to log file the request that it receive, i am able to print the body but how can i extract all the request include the headers?
I am using request.POST.get() to get the body/data from the request.
Thanks
To get request headers, you can use request.META, as it returns dictionary containing all available HTTP headers.
headers = request.META
request.POST should give you the POST body if it is get use request.GET
if the request body is json use request.data
When I get the Set-Cookie and try to use it, I wont seem that I'm logged in Facebook...
import urllib, urllib2
data = urllib.urlencode({"email":"swagexample#hotmail.com", "pass":"password"})
request = urllib2.Request("http://www.facebook.com/login.php", data)
request.add_header("User-Agent", "Mozilla 5.0")
response = urllib2.urlopen(request)
cookie = response.headers.get("Set-Cookie")
new_request = urllib2.Request("http://www.facebook.com/login.php")
new_request.add_header("User-Agent", "Mozilla 5.0")
new_request.add_header("Cookie", cookie)
new_response = urllib2.urlopen(new_request)
if "Logout" in new_response.read():
print("Logged in.") #No output
Why?
First, Set-Cookie header format is different from Cookie header.
Set-Cookie header contains additional information (doamin, expire, ...), you need to convert them to use it for Cookie header.
cookie = '; '.join(
x.split(';', 1)[0] for x in response.headers.getheaders("Set-Cookie")
)
Even though you do above, you will still not get what you want, because default urllib2 handler does not handle cookie for redirect.
Why don't you use urllib2.HTTPCookieProcessor as you did before?
I'm working with the CrawlSpider class to crawl a website and I would like to modify the headers that are sent in each request. Specifically, I would like to add the referer to the request.
As per this question, I checked
response.request.headers.get('Referer', None)
in my response parsing function and the Referer header is not present. I assume that means the Referer is not being submitted in the request (unless the website doesn't return it, I'm not sure on that).
I haven't been able to figure out how to modify the headers of a request. Again, my spider is derived from CrawlSpider. Overriding CrawlSpider's _requests_to_follow or specifying a process_request callback for a rule will not work because the referer is not in scope at those times.
Does anyone know how to modify request headers dynamically?
You can pass REFERER manually to each request using headers argument:
yield Request(parse=..., headers={'referer':...})
RefererMiddleware does the same, automatically taking the referrer url from the previous response.
You have to enable the SpiderMiddleware that will populate the referer for responses. See the documentation for scrapy.contrib.spidermiddleware.referer.RefererMiddleware
In short, you need to add this middleware to your project's settings file.
SPIDER_MIDDLEWARES = {
'scrapy.contrib.spidermiddleware.referer.RefererMiddleware': True,
}
Then in your response parsing method, you can use, response.request.headers.get('Referrer', None), to get the referer.