I'm using Requests http://docs.python-requests.org/en/latest/
I'm trying to do a simple POST request with some extra headers. I don't understand the following behavior.
opener = requests.Session()
data = {"payload" : { "id": 1, "pwd": "mypass"}
headers = {"Content-Type":"application/json"}
url = "https://mysite.com/login"
# THIS WORKS
opener.post(url, data)
# THIS DOES NOT WORK
opener.post(url, json.dumps(data))
# THIS DOES NOT WORK
opener.post(url, data=data, headers=headers)
# THIS WORKS
opener.post(url, data=json.dumps(data), headers=headers)
It seems that the post method expects a dict normally and does not work when I convert that dict to a string. I can login using the former but I can't using the latter.
However, when I supply extra headers information, it seems the post method works the extact opposite way, it expects a string for the data. On the server side, the first version throws an error while the second version works with headers.
What is the reason for this behavior?
Related
I try to read JSON-formatted data from the following public URL: http://ws-old.parlament.ch/factions?format=json. Unfortunately, I was not able to convert the response to JSON as I always get the HTML-formatted content back from my request. Somehow the request seems to completely ignore the parameters for JSON formatting passed with the URL:
import urllib.request
response = urllib.request.urlopen('http://ws-old.parlament.ch/factions?format=json')
response_text = response.read()
print(response_text) #why is this HTML?
Does somebody know how I am able to get the JSON formatted content as displayed in the web browser?
You need to add "Accept": "text/json" to request header.
For example using requests package:
r = requests.get(r'http://ws-old.parlament.ch/factions?format=json',
headers={'Accept':'text/json'})
print(r.json())
Result:
[{'id': 3, 'updated': '2022-02-22T14:59:17Z', 'abbreviation': ...
Sorry for you but these web services have a misleading implementation. The format query parameter is useless. As pointed out by #maciek97x only the header Accept: <format> will be considered for the formatting.
So your can directly call the endpoint without the ?format=json but with the header Accept: text/json.
I'm trying to build an app that alerts when air quality rises above a certain level. I'm trying to get some json data from the api at https://api-docs.iqair.com, and they kindly provide simple copy and paste code. However, when I run this (with my API key), I get this error message:
requests.exceptions.MissingSchema: Invalid URL '{{urlExternalAPI}}v2/city?city=Los Angeles&state=California&country=USA&key={{my_key}}': No schema supplied. Perhaps you meant http://{{urlExternalAPI}}v2/city?city=Los Angeles&state=California&country=USA&key={{my_key}}?`
I tried putting in the http, but then nothing happened.
Here's the code they provide:
import requests
url = "{{urlExternalAPI}}v2/city?city=Los Angeles&state=California&country=USA&key={{YOUR_API_KEY}}"
payload = {}
headers= {}
response = requests.request("GET", url, headers=headers, data = payload)
print(response.text.encode('utf8'))
First of all, you have to put in the URL, and not use the curly brackets. Also, I couldn't find the correct URL, but after googling it, I found that I merely had to use the correct URL, which was https://api.airvisual.com.
I'm trying to send a POST Request with form data to some form. I'm using requests package.
#!/bin/python3.6
import requests
FFF_search = 'https://www.fff.fr/la-vie-des-clubs/resultats'
data = {'address':75000}
session = requests.session()
r = session.post(FFF_search, data=data, headers={'User-Agent':'test'})
print(r.text)
This gives me the result page where no result is found.
Maybe my problem is more about the data I'm sending.
Here's the POST request as it must be done on the website (Chrome dev tools).
You should consider several issues:
Your's headers dict is not valid. There is no 'User-Agnet' that called test. You should use only valid headers headers. Try to send all the headers you see in the network tab.
Your's data us clearly wrong. You should send the described Form Data ad a dictionary and pass it with data=data, as you do (but you do it with a wrong data). Maybe the data you are passing now is the params?
Try to send the exact request like your's browser sending (with the same params and headers), and see the results you are receiving, with paying attention to the issues above.
rocksteady's solution worked
He did originally refer to dictionaries. But the following code to send the JSON string also worked wonders using requests:
import requests
headers = {
'Authorization': app_token
}
url = api_url + "/b2api/v1/b2_get_upload_url"
content = json.dumps({'bucketId': bucket_id})
r = requests.post(url, data = content, headers = headers)
I'm working with an API that requires me to send JSON as a POST request to get results. Problem is that Python 3 won't allow me to do this.
The following Python 2 code works fine, in fact it's the official sample:
request = urllib2.Request(
api_url +'/b2api/v1/b2_get_upload_url',
json.dumps({ 'bucketId' : bucket_id }),
headers = { 'Authorization': account_authorization_token }
)
response = urllib2.urlopen(request)
However, using this code in Python 3 only makes it complain about data being invalid:
import json
from urllib.request import Request, urlopen
from urllib.parse import urlencode
# -! Irrelevant code has been cut out !-
headers = {
'Authorization': app_token
}
url = api_url + "/b2api/v1/b2_get_upload_url"
# Tested both with encode and without
content = json.dumps({'bucketId': bucket_id}).encode('utf-8')
request = Request(
url=url,
data=content,
headers=headers
)
response = urlopen(req)
I've tried doing urlencode(), like you're supposed to. But this returns a 400 status code from the web server, because it's expecting pure JSON. Even if the pure JSON data is invalid, I need to somehow force Python into sending it.
EDIT: As requested, here are the errors I get. Since this is a flask application, here's a screenshot of the debugger:
Screenshot
Adding .encode('utf-8') gives me an "Expected string or buffer" error
EDIT 2: Screenshot of the debugger with .encode('utf-8') added
Since I have a similar application running, but the client still was missing, I tried it myself.
The server which is running is from the following exercise:
Miguel Grinberg - designing a restful API using Flask
That's why it uses authentication.
But the interesting part: Using requests you can leave the dictionary as it is.
Look at this:
username = 'miguel'
password = 'python'
import requests
content = {"title":"Read a book"}
request = requests.get("http://127.0.0.1:5000/api/v1.0/projects", auth=(username, password), params=content)
print request.text
It seems to work :)
Update 1:
POST requests are done using requests.post(...)
This here describes it well : python requests
Update 2:
In order to complete the answer:
requests.post("http://127.0.0.1:5000/api/v1.0/projects", json=content)
sends the json-string.
json is a valid parameter of the request and internally uses json.dumps()...
I am a bit confused with using Request, urlopen and JSONDecoder().decode().
Currently I have:
hdr = {'User-agent' : 'anything'} # header, User-agent header describes my web browser
I am assuming that the server uses this to determine which browsers are acceptable? Not sure
my url is:
url = 'http://wwww.reddit.com/r/aww.json'
I set a req variable
req = Request(url,hdr) #request to access the url with header
json = urlopen(req).read() # read json page
I tried using urlopen in terminal and I get this error:
TypeError: must be string or buffer, not dict # This has to do with me header?
data = JSONDecoder().decode(json) # translate json data so I can parse through it with regular python functions?
I'm not really sure why I get the TypeError
If you look at the documentation of Request, you can see that the constructor signature is actually Request(url, data=None, headers={}, …). So the second parameter, the one after the URL, is the data you are sending with the request. But if you want to set the headers instead, you will have to specify the headers parameter.
You can do this in two different ways. Either you pass None as the data parameter:
Request(url, None, hdr)
But, well, this requires you to pass the data parameter explicitely and you have to make sure that you pass the default value to not cause any unwanted effects. So instead, you can tell Python to explicitely pass the header parameter instead, without specifying data:
Request(url, headers=hdr)