I'm calling the google API myself instead of using their python Library because I'm behind an inconvenient corporate proxy which kills their library, so I have to do it all myself.
This works fine:
requests.get('https://www.googleapis.com/webmasters/v3/sites', params =
{'access_token':'my_access_token_here'})
This on the other hand, doesn't:
site = https://www.my_website_from_the_above_function.com
site = urllib.parse.quote_plus(site)
def get_website_info():
url = 'https://www.googleapis.com/webmasters/v3/sites/{}/searchAnalytics/query'.format(site)
params = {
"endDate": "2017-12-10",
"startDate": "2017-12-01",
"access_token": my_access_token
}
r = requests.post(url, params = params)
return r
x = get_website_info().json()
All I get is this error code:
{'error': {'code': 500,
'errors': [{'domain': 'global',
'message': 'Backend Error',
'reason': 'backendError'}],
'message': 'Backend Error'}}
Even with the reccomended 'Exponential backoff'
Using googles API explorer seems to work fine:
Aditionally: This also seems give similar errors:
r = requests.post(url, params = auth_params, data = json.dumps(params))
and finally:
r = requests.post(url, params = auth_params, data = params)
just gives
{'error': {'code': 400,
'errors': [{'domain': 'global',
'message': 'This API does not support parsing form-encoded input.',
'reason': 'parseError'}],
'message': 'This API does not support parsing form-encoded input.'}}
So, you can think of the contents of a request as just text, right? Not only text, but text that accepts a relatively limited number of characters.
With that in mind, it all boils down on how to serialize "complex" data structures into text. I recently answered another question about files that is kinddddaaa similar idea.
If you have a bunch of key=value parameters, you could use a simple "trick":
Control names and values are escaped. Space characters are replaced by
+, and then reserved characters are escaped as described in
[RFC1738], section 2.2: Non-alphanumeric characters are replaced by
%HH, a percent sign and two hexadecimal digits representing the
ASCII code of the character. Line breaks are represented as "CR LF"
pairs (i.e., %0D%0A).
The control names/values are listed in the
order they appear in the document. The name is separated from the
value by = and name/value pairs are separated from each other by
&.
So this data:
{a="foo", b="bar baz"}
Could be serialized into text following the specification above like: a=foo&b=bar+baz
That serialization format is identified as application/x-www-form-urlencoded in the Content-type request's header. That request's header is telling the server that receives it something like "Hey! The data that is coming in my body is serialized following that convention that separates keys from values using the = symbol and splits key/value pairs using &, changes whitespaces by +... and so on"
(!) Very important: That is the format used by the requests module on a POST unless told otherwise.
Another format, which allows more flexibility (such as maintaining basic types or nesting structures) is JSON. That is the format that the Google server "wants", and in order to tell servers that the "text" contained in the request's body follows the Json standard (or convention), the Content-Type header must be set to 'application/json'.
What appears that your Google server was doing upon receiving a request was checking the Content-type header and if it wasn't Json, it gave you a 400 error to indicate "Oh, I don't understand this format... I want Json!"
That's why you have to specify the Json header.
There's an example comparing both formats here.
You can also see it more clearly since the latest versions of requests module can do the JSON parsing for you. Since the JSON format has become so common, you can pass data provided in a Python structure (a dict, for instance) through the json= argument, and the module will do the json.dumps and set the header for you. This also allows you to "introspect" a little how the body will look like (to see the differences maybe more clearly).
Check this out:
from requests import Request
data = {
'a': 'foo-1 baz',
'b': 5,
'c': [1, 2, 3],
'd': '6'
}
req = Request('POST', 'http://foo.bar', data=data)
prepped = req.prepare()
print("Normal headers: %s" % prepped.headers)
print("Normal body: %s" % prepped.body)
req = Request('POST', 'http://foo.bar', json=data)
prepped = req.prepare()
print("Json headers: %s" % prepped.headers)
print("Json body: %s" % prepped.body)
Outputs:
Normal headers: {'Content-Length': '31', 'Content-Type': 'application/x-www-form-urlencoded'}
Normal body: d=6&a=foo-1+baz&c=1&c=2&c=3&b=5
Json headers: {'Content-Length': '52', 'Content-Type': 'application/json'}
Json body: b'{"d": "6", "a": "foo-1 baz", "c": [1, 2, 3], "b": 5}'
See the difference? JSON is capable of making a difference between the strings foo-1 or 6 (using ") as opposed to 5 being an integer, while the x-www-form can't (see how the form encoding doesn't differentiate between the integer 5 or the string 6). Same with the list. By using the character [, the server will be able to tell that c is a list (and of integers)
I got it! The solution:
was to pass in header information with:
headers = {'Content-type': 'application/json',
'Authorization' : 'Bearer %s' % access_token}
and make sure the json data was dumped to a string:
r = requests.post(url,data = json.dumps(params), headers = headers)
If someone could explain the reason behind my answer, that would be great.
Related
I am making a post request and I have a payload variable but I am trying to set some of the data in the payload variable as user input so I am doing:
payload = "{\r\"user\": \"{}\",\r\"password\": \"{}\",\r\"gdn\": 0\r}".format(name, pass)
I get this error:
File "testing.py", line 24, in sndReq
payload = "{\r\"user\": \"{}\",\r\"password\": \"{}\",\r\"gdn\": 0\r}".format(name, pass)
KeyError: '\r"user"
If I take out .format and put hard values in there it works. (I also tried using an 'f' string)
My request line is this if it matters:
r = requests.request("POST", url, data=payload, headers=headers)
It seems your string is not formatted/escaped properly. How about doing something which is more readable, such as:
import requests
# some example values because your question doesn't give too many details
user, password ="user", "password"
url="http://google.com" # example url
headers={'pragma': 'no-cache'} # whatever your headers are
# now do the request, note we're not passing a single string here
requests.post(
url,
json={
"user": user,
"password": password
},
headers=headers
)
Additionally, I'd encourage you to work with something a bit easier than those unreadable strings! Put it into something that python (and us) can work with more easily. The first thing I'd do is get your string into a python object:
import json
payload = "{\r\"user\": \"{}\",\r\"password\": \"{}\",\r\"gdn\": 0\r}"
data=json.loads(payload)
data.update({'password': 'secret', 'user': 'me'})
# {u'password': 'secret', u'user': 'me', u'gdn': 0}
And finally, if you truly want to use the {} as the format string (which I hope the rest of my answer shows you is much more difficult than you need to make it), you'll need to remove the leading and trailing braces in the formatted text:
"{" + "\r\"user\": \"{}\",\r\"password\": \"{}\",\r\"gdn\": 0\r".format("user", "secret") + "}"
# '{\r"user": "user",\r"password": "secret",\r"gdn": 0\r}
I am trying to do the following with requests:
data = {'hello', 'goodbye'}
json_data = json.dumps(data)
headers = {
'Access-Key': self.api_key,
'Access-Signature': signature,
'Access-Nonce': nonce,
'Content-Type': 'application/json',
'Accept': 'text/plain'
}
r = requests.post(url, headers=headers, data=json_data,
files={'file': open('/Users/david/Desktop/a.png', 'rb')})
However, I get the following error:
ValueError: Data must not be a string.
Note that if I remove the files parameter, it works as needed. Why won't requests allow me to send a json-encoded string for data if files is included?
Note that if I change data to be just the normal python dictionary (and not a json-encoded string), the above works. So it seems that the issue is that if files is not json-encoded, then data cannot be json-encoded. However, I need to have my data encoded to match a hash signature that's being created by the API.
When you specify your body to a JSON string, you can no longer attach a file since file uploading requires the MIME type multipart/form-data.
You have two options:
Encapsulate your JSON string as part as the form data (something like json => json.dumps(data))
Encode your file in Base64 and transmit it in the JSON request body. This looks like a lot of work though.
1.Just remove the line
json_data = json.dumps(data)
and change in request as data=data.
2.Remove 'Content-Type': 'application/json' inside headers.
This worked for me.
Alternative solution to this problem is to post data as file.
You can post strings as files. Read more here:
http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file
Here is explained how to post multiple files:
http://docs.python-requests.org/en/latest/user/advanced/#post-multiple-multipart-encoded-files
removing the following helped me in my case:
'Content-Type': 'application/json'
then the data should be passed as dictionary
If your files are small, you could simply convert the binary (image or anything) to base64 string and send that as JSON to the API. That is much simpler and more straight forward than the suggested solutions. The currently accepted answer claims that is a lot of work, but it's really simple.
Client:
with open('/Users/houmie/Downloads/log.zip','rb') as f:
bytes = f.read()
tb = b64encode(bytes)
tb_str = tb.decode('utf-8')
body = {'logfile': tb_str}
r = requests.post('https://prod/feedback', data=json.dumps(body), headers=headers)
API:
def create(event, context):
data = json.loads(event["body"])
if "logfile" in data:
tb_back = data["logfile"].encode('utf-8')
zip_data = base64.b64decode(tb_back)
I am trying to do the following with requests:
data = {'hello', 'goodbye'}
json_data = json.dumps(data)
headers = {
'Access-Key': self.api_key,
'Access-Signature': signature,
'Access-Nonce': nonce,
'Content-Type': 'application/json',
'Accept': 'text/plain'
}
r = requests.post(url, headers=headers, data=json_data,
files={'file': open('/Users/david/Desktop/a.png', 'rb')})
However, I get the following error:
ValueError: Data must not be a string.
Note that if I remove the files parameter, it works as needed. Why won't requests allow me to send a json-encoded string for data if files is included?
Note that if I change data to be just the normal python dictionary (and not a json-encoded string), the above works. So it seems that the issue is that if files is not json-encoded, then data cannot be json-encoded. However, I need to have my data encoded to match a hash signature that's being created by the API.
When you specify your body to a JSON string, you can no longer attach a file since file uploading requires the MIME type multipart/form-data.
You have two options:
Encapsulate your JSON string as part as the form data (something like json => json.dumps(data))
Encode your file in Base64 and transmit it in the JSON request body. This looks like a lot of work though.
1.Just remove the line
json_data = json.dumps(data)
and change in request as data=data.
2.Remove 'Content-Type': 'application/json' inside headers.
This worked for me.
Alternative solution to this problem is to post data as file.
You can post strings as files. Read more here:
http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file
Here is explained how to post multiple files:
http://docs.python-requests.org/en/latest/user/advanced/#post-multiple-multipart-encoded-files
removing the following helped me in my case:
'Content-Type': 'application/json'
then the data should be passed as dictionary
If your files are small, you could simply convert the binary (image or anything) to base64 string and send that as JSON to the API. That is much simpler and more straight forward than the suggested solutions. The currently accepted answer claims that is a lot of work, but it's really simple.
Client:
with open('/Users/houmie/Downloads/log.zip','rb') as f:
bytes = f.read()
tb = b64encode(bytes)
tb_str = tb.decode('utf-8')
body = {'logfile': tb_str}
r = requests.post('https://prod/feedback', data=json.dumps(body), headers=headers)
API:
def create(event, context):
data = json.loads(event["body"])
if "logfile" in data:
tb_back = data["logfile"].encode('utf-8')
zip_data = base64.b64decode(tb_back)
I need to send this JSON array in a GET request
{"user": "jähn", "id": 3}
I tried to use
data = '{"user": "jähn", "id": 3}'
headers = {
'Content-type': 'application/json',
'Accept': 'text/plain'
}
request = urllib.request.Request(self.update_url, data=data,
headers=headers, method='GET')
response = urllib.request.urlopen(request)
But its failing with: TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str.
Another thing that I find quite weird is that it tells me about POST data although I set the method on the Request to GET.
Since this is a simple script I'd prefer not to use a library like python-requests
You cannot make a GET request with a JSON-encoded body, as a GET request only ever consists of the URL and the headers. Parameters are encoded into the URL using URL encoding instead, there is no option to encode such parameters to JSON instead.
You create URL-encoded parameters with the urllib.parse.urlencode() function, then appended to the URL with ?.
from request.parse import urlencode
data = {"user": "jähn", "id": 3} # note, a Python dictionary, not a JSON string
parameters = urlencode(data)
response = urllib.request.urlopen('?'.join((self.update_url, parameters)))
Do not use the data parameter; using that keyword argument forces a request to use the POST method:
data must be a bytes object specifying additional data to send to the server, or None if no such data is needed. Currently HTTP requests are the only ones that use data; the HTTP request will be a POST instead of a GET when the data parameter is provided. data should be a buffer in the standard application/x-www-form-urlencoded format.
I'm trying to make a RESTful api call in python/django with requests.post
I can get requests.get(url=url, auth=auth) to work. Similar call in the same api family for this company
I'm trying to do:
data = {'start': 13388, 'end': 133885, 'name': 'marcus0.5'}
r = requests.post(url=url, auth=auth, headers={'Accept': 'application/json'}, data=data)
and I get the following error:
>>> r.text
u'{"status":"error","errorCode":"COMMON_UNSUPPORTED_MEDIA_TYPE","incidentId":"czEtbXNyZXBvcnRzMDQuc3RhZ2V4dHJhbmV0LmFrYW1haS5jb20jMTM3NTgxMzc3MTk4NQ==","errorMessage":"The server is refusing to service the request because the entity of the request is in a format not supported by the requested resource for the requested method.. Content type \'application/x-www-form-urlencoded;charset=UTF-8\' not supported."}'
I think it has something to do with the json, but I'm not sure what and I'm not sure how to fix it. Any ideas?
Extra info [not sure if it applies]:
I imported
import requests, django
I know the the auth is correct and I tested it with the get method
You want to set the Content-Type parameter of your request to 'application/json', not the Accept parameter.
Taken from w3.org:
The Accept request-header field can be used to specify certain media types which are acceptable for the response.
Try this instead:
import json
data = {'start': 13388, 'end': 133885, 'name': 'marcus0.5'}
r = requests.post(url=url, auth=auth, data=json.dumps(data),
headers={'content-type': 'application/json'})
EDIT:
There is a little bit of confusion (for me as well) about when to send data as a dict or a json encoded string (ie. the result of json.dumps). There is an excellent post here that explains the problem. For a brief summary send a dict when the API requires form-encoded data, and a json encoded string when it requires json-encoded data.