I have been working on solving an HTTP 500 (bad syntax/string) error for far too long, and after doing some searching I cannot find a solution anywhere. I have a nested json PUT request that I have been able to make work using a couple API tools (both browser extensions and stand-alone programs), but when I try to use the json in Python's HTTP Requests module, I keep getting the 500 error code returned.
I have gotten other, simplier jsons (e.g. data={"RequestID": "71865"}) to work using similar code to the following, which leaves me to believe something is not getting formatted correctly, and I am unfortunately too new to this json-python thing to figure it out. I think the issue is because of the way python handles the nested json.
# -*- coding: utf-8 -*-
#!/usr/bin/env python
import requests
import json
USER_NAME=u"myusername"
USER_PASS=u"mypassword"
PUT_URL="https://webservice.url.com/A/Path/To/Create/"
headers = {"Content-Type": "application/json"}
data = {
"ListOfFields": {
"Field": [
{"fieldname": "summary","value": "test summary"},
{"fieldname": "notes","value": "an example json PUT"},
{"fieldname": "user","value": "myuser"}
]
}
}
data_json = json.dumps(data)
payload = {'json_playload': data_json } ## I have tried with and without this line.
r = requests.put('{}'.format(PUT_URL), data=data_json, headers=headers, auth=(USER_NAME, USER_PASS), timeout=10)
# r = requests.put('{}'.format(PUT_URL), data=payload, headers=headers, auth=(USER_NAME, USER_PASS), timeout=10)
I have tried putting the data value into quotes, a single line, and making some other slight tweaks, but I keep getting the 500 error.
print(r.status_code)
>> 500
As mentioned before, I have gotten similar code to work in python using GET and POST and the same web server, but this one is giving me a headache!
The Requests library has a nasty habit of clobbering data when passing in nested JSON to the data param. To avoid this, pass it into the json param instead:
r = requests.put(PUT_URL, json=data_json, headers=headers, auth=(USER_NAME, USER_PASS), timeout=10)
For more details, take a look at this answer to a similar question: Post JSON using Python Requests
Do you want to pretty print your JSON data? Try this:
data_json = json.dumps(data, sort_keys=True, indent=4, separators=(',', ': '))
See https://docs.python.org/2/library/json.html
Related
I had a script to send a file to anonfiles.com for sharing using requests. It emulates the suggested curl command:
curl -F "file=#test.txt" https://api.anonfiles.com/upload
with open(filename, 'rb') as file:
req = requests.post(
'https://api.anonfiles.com/upload',
files={'file': file},
)
res = json.loads(req.text)
This worked properly for files smaller than 2GB, but when the file grew bigger, it started printing a error message about the filesize.
I tried to using requests-toolbelt to send the file with no success.
with open(filename, 'rb') as file:
encoder = MultipartEncoder({'file': file})
req = requests.post(
'https://api.anonfiles.com/upload',
data=encoder,
headers={'Content-Type': encoder.content_type}
)
res = json.loads(req.text)
Which returns a the following error from anonfiles.com:
{
"status": false,
"error": {
"message": "No file chosen.",
"type": "ERROR_FILE_NOT_PROVIDED",
"code": 10
}
}
I'm probably missing something obvious, but can't figure it out right now.
P.S.: the use of requests-toolbelt.MultipartEncoder is not mandatory, I tried it and didn't manage to make it work. I will accept any answer even if it doesn't include this. A solution without request is not optimal but would also work.
Using a tuple seems to work:
with open(filename, 'rb') as file:
encoder = MultipartEncoder({'file': (filename, file)})
req = requests.post(
'https://api.anonfiles.com/upload',
data=encoder,
headers={'Content-Type': encoder.content_type}
)
res = json.loads(req.text)
EDIT:
Upon further examination, I really am not sure what's going on. I originally thought you could fix it by sending json=json_file with the request, but then I realized you were using the encoder. I do not know if the MultipartEncoder will work the same way with the json object.
Anyway...good luck!
I think this is a simple case of json.load vs json.loads:
json.load is for files
json.loads is for strings
In your original code you never loaded the file because you used json.loads, which is for strings.
Although I can't explain why it worked for smaller files so maybe I'm just off base!! you might try some sort of streaming for the larger file as well.
I haven't seen the syntax you used for the response before so I rewrote it a tiny bit to simplify it to understand it a little better, but mostly I think the issue is what I wrote above:
url = 'www.url.com'
headers = 'header stuff'
with open(filename, 'rb') as file:
json_file = json.load(file)
encoder = MultipartEncoder({'file': json_file})
response = requests.post(url, data=encoder, headers=headers).text
# or instead of appending .text you could create
# data = response.text for easier use in the rest of your code.
Summarize the problem: Uncompressed JSON payload updates successfully. However, compressed JSON fails to upload to the website.(using Python requests module)
2.
I am trying to POST comparatively large JSON data(1 MB +) to a website
I wish to compress the JSON data(stream) and post it to a wesite. The catch is- compressed JSON returns success code, but data is not reflected on the website.
On the contrary, the same JSON data is making through when not compressed.
3.
When appropriate, show some code:
Works fine:
payload without compression - goes though to website without issues:
payload={"eventType": "check", "status": "Fail","testCategory": "Test"}
headers = {‘Content-Type’: ‘application/json’,‘X-Insert-Key’: ‘XXXXXXX’}
r = requests.post(url, data=json.dumps(payload), headers=headers)
NOT working
the payload that is compressed not making it to the website:
payload={"eventType": "check", "status": "Fail","testCategory": "Test"}
headers = {‘Content-Type’: ‘application/json’,‘X-Insert-Key’: ‘XXXXXXX’, ‘Content-Encoding’:‘gzip’}
request_body = zlib.compress(json.dumps(payload))
r = requests.post(url, data=request_body, headers=headers)
I am banging my head around but it seems I am missing on something subtle. Any help/tips will be highly appreciated
Changing - ‘Content-Encoding’:‘gzip’} to ‘Content-Encoding’:‘deflate’} fixed the issue for me
Note: This answer is specific to New Relic custom events
I'm trying to post to a server using the following script:
import requests
data = {
'query': 'GetProcess',
'getFrom': '2018-12-06 10:10:10.000',
}
response = requests.post('http://localhost/monitor', data=data)
I cannot find where exactly, but the space character in the getFrom element is being replaced with a +: '2018-12-06+10:10:10.000'
This doesn't match the syntax SQL expects on our server, so the query fails.
I read here (https://stackoverflow.com/a/12528097) that setting the Content-type might help. I tried text/html, text/plain, application/json, and nothing seems to change.
Interestingly, the following (equivalent?) bash command succeeds:
curl -d 'query=GetProcess&getFrom=2018-12-06 10:10:10.000' localhost/monitor
I'm looking for a way to make my server receive "getFrom" : "2018-12-06 10:10:10.000" in the header.
I found a way to make this work: the problem I was having was due to the use of the urlencode function used in requests. In the requests documentation, it is shown how to go around this default behavior using PreparedRequests: http://docs.python-requests.org/en/master/user/advanced/#prepared-requests
Essentially, instead of using the requests.post() wrapper, make the function calls explicitly. This way, you will be able to control exactly what is sent. In my case, the solution was to do:
import requests
data = {
'query': 'GetProcess',
'getFrom': '2018-12-06 10:10:10.000'
}
s = requests.Session()
req = requests.Request('POST', 'http://'+ipAddress+'/monitor', data=data)
prepped = s.prepare_request(req)
prepped.body = prepped.body.replace("+", " ")
response = s.send(prepped)
I am trying to import a csv of responses into Qualtrics using the API shown here: https://api.qualtrics.com/docs/import-responses. But, since I'm a noob at Python and (by extension) at Requests, I'm having trouble figuring out why I keep getting a 413. I've gotten this far:
formTest = {
'surveyId': 'my_id',
'file': {
'value': open('dataFiles/myFile.csv', 'rb'),
'options': {
'contentType': 'text/csv'
}
}
}
headersTest = {
"X-API-TOKEN": "my_token",
'content-type': "multipart/form-data"
}
r = requests.request("POST", url, data=formTest, headers=headersTest)
print(r.text)
The format for the formTest variable is something I found when looking through other code bases for an angular implementation of this, which may not apply to a python version of the code. I can successfully use cUrl, but Python Requests, in my current situation is the way to go (for various reasons).
In a fit of desperation, I tried directly translating the cUrl request to python requests, but that didn't seem to help much either.
Has anyone done something like this before? I took a look at posts for importing contacts and the like, but there was no luck there either (since the data that needs to be sent is formatted differently). Is there something I am missing?
It's best not to mix post data and files but use two separate dictionaries. For the files you should use the files= parameter, because it encodes the POST data as a Multipart Form data and creates the required Content-Type headers.
import requests
url = 'Qualtrics API'
file_path = 'path/to/file'
file_name = 'file.name'
data = {'surveyId':'my_id'}
files = {'file' : (file_name, open(file_path, 'rb'), 'text/csv')}
headers = {'X-API-TOKEN': 'my_token'}
r = requests.post(url, data=data, files=files, headers=headers)
print(r.text)
The first value in files['file'] is the file name (optional), followed by the file object, followed by the file content type (optional).
You will find more info in the docs: Requests, POST a Multipart-Encoded File.
I'm working on an API wrapper. The spec I'm trying to build to has the following request in it:
curl -H "Content-type:application/json" -X POST -d data='{"name":"Partner13", "email":"example#example.com"}' http://localhost:5000/
This request produces the following response from a little test server I setup to see exatly what headers/params etc are sent as. This little script produces:
uri: http://localhost:5000/,
method: POST,
api_key: None,
content_type: application/json,
params: None,
data: data={"name":"Partner13", "email":"example#example.com"}
So that above is the result I want my python script to create when it hits the little test script.
I'm using the python requests module, which is the most beautiful HTTP lib I have ever used. So here is my python code:
uri = "http://localhost:5000/"
headers = {'content-type': 'application/json' }
params = {}
data = {"name":"Partner13", "email":"example#exmaple.com"}
params["data"] = json.dumps(data)
r = requests.post(uri, data=params, headers=headers)
So simple enough stuff. Set the headers, and create a dictionary for the POST parameters. That dictionary has one entry called "data" which is the JSON string of the data I want to send to the server. Then I call the post. However, the result my little test script gives back is:
uri: http://localhost:5000/,
method: POST,
api_key: None,
content_type: application/json,
params: None,
data: data=%7B%22name%22%3A+%22Partner13%22%2C+%22email%22%3A+%22example%40example.com%22%7D
So essentially the json data I wanted to send under the data parameter has been urlendcoded.
Does anyone know how to fix this? I have looked through the requests documentation and cannot seem to find a way to not auto urlencode the send data.
Thanks very much,
Kevin
When creating the object for the data keyword, simply assign a variable the result of json.dumps(data).
Also, because HTTP POST can accept both url parameters as well as data in the body of the request, and because the requests.post function has a keyword argument named "params", it might be better to use a different variable name for readability. The requests docs use the variable name "payload", so thats what I use.
data = {"name":"Partner13", "email":"example#exmaple.com"}
payload = json.dumps(data)
r = requests.post(uri, data=payload, headers=headers)
Requests automatically URL encodes dictionaries passed as data here. John_GG's solution works because rather than posting a dictionary containing the JSON encoded string in the 'data' field it simply passes the JSON encoded string directly: strings are not automatically encoded. I can't say I understand the reason for this behaviour in Requests but regardless, it is what it is. There is no way to toggle this behaviour off that I can find.
Best of luck with it, Kevin.