We Keep Coding

Python requests PUT request with json parameter fails and data parameter succeeds

Problem
I've looked at some of the documentation about the json and data parameters and the differences between them. I think I understand the difference, best explained here, in my opinion.
However, I have a specific request that fails on PUT using json, but fails using data, and I'm not sure why. Can someone clarify why this is the case? Could it be that there is a list in the payload?
Context
I have requests==2.28.0 installed. Below is the code that submits the PUT requests to an API for PagerDuty, the incident management software, one using data (successful) and one using json (failing). Otherwise they are identical.
The weird thing is that their examples use the json parameter.
payload = f'{{"source_incidents": [{{"id": "{child_incident_id}", "type": "incident_reference"}}]}}'
headers = {
'Content-Type': "application/json",
'Accept': "application/vnd.pagerduty+json;version=2",
'From': email,
'Authorization': f"Token token={read_write_api_token}"
}
response = requests.put(f'https://api.pagerduty.com/incidents/{parent_incident_id}/merge', data=payload, headers=headers)
print("response: ", response)
Result: response: <Response [200]>
payload = f'{{"source_incidents": [{{"id": "{child_incident_id}", "type": "incident_reference"}}]}}'
headers = {
'Content-Type': "application/json",
'Accept': "application/vnd.pagerduty+json;version=2",
'From': email,
'Authorization': f"Token token={read_write_api_token}"
}
response = requests.put(f'https://api.pagerduty.com/incidents/{parent_incident_id}/merge', json=payload, headers=headers)
print("response: ", response)
Result: response: <Response [400]>

Your payload is a string while json parameter takes a dictionary. That's the whole point of the json argument (you don't have to encode it yourself):
If you need that header set and you don’t want to encode the dict yourself, you can also pass it directly using the json parameter (added in version 2.4.2) and it will be encoded automatically:
You should pass a dictionary if you want to use the json parameter:
payload = {
"source_incidents": [
{
"id": child_incident_id,
"type": "incident_reference"
}
]
}
which is more readable anyway.

Alternatively you could use json.loads to parse your string:
import json
payload = f'{{"source_incidents": [{{"id": "{child_incident_id}", "type": "incident_reference"}}]}}'
headers = {
'Content-Type': "application/json",
'Accept': "application/vnd.pagerduty+json;version=2",
'From': email,
'Authorization': f"Token token={read_write_api_token}"
}
response = requests.put(f'https://api.pagerduty.com/incidents/{parent_incident_id}/merge', data=json.loads(payload), headers=headers)
print("response: ", response)

That is what the requests library does with json data.
Converts your Python object to JSON using the json encoder.
Sets the "content-type" header to "application/json".
It is possible to implement it as shown here.
def put(uri, data=None, json=None):
if json and data:
raise Exception()
if json:
payload = json.dumps(json)
else:
payload = data
...
So the first request returns 200 because you passed valid JSON OBJECT through the "data" parameter.
And the second request fails because you passed STRING through the JSON that will be dumped with "json.dumps(obj)" for example.
As a result, it would be nothing more than a string, which is also a valid JSON object but not a javascript object.
As shown here, if you pass a string through "json.dumps" and a dictionary, it returns two different values: a string and an object.
>>> json.dumps("{}")
'"{}"'
>>> json.dumps({})
'{}'
>>>

Convert a PDF to DOCX using Adobe PDF Services via REST API (with Python)

I am trying to query Adobe PDF services API to generate (export) DOCX from PDF documents.
I just wrote a python code to generate a Bearer Token in order to be identified from Adobe PDF services (see the question here: https://stackoverflow.com/questions/68351955/tunning-a-post-request-to-reach-adobe-pdf-services-using-python-and-a-rest-api). Then I wrote the following piece of code, where I tried to follow the instruction in this page concerning the EXPORT option of Adobe PDF services (here: https://documentcloud.adobe.com/document-services/index.html#post-exportPDF).
Here is the piece of code :
import requests
import json
from requests.structures import CaseInsensitiveDict
N/B: I didn't write the part of the code generating the Token and enabling identification by the server
>> This part is a POST request to upload my PDF file via form parameters
URL = "https://cpf-ue1.adobe.io/ops/:create?respondWith=%257B%2522reltype%2522%253A%2520%2522http%253A%252F%252Fns.adobe.com%252Frel%252Fprimary%2522%257D"
headers = CaseInsensitiveDict()
headers["x-api-key"] = "client_id"
headers["Authorization"] = "Bearer MYREALLYLONGTOKENIGOT"
headers["Content-Type"] = "application/json"
myfile = {"file":open("absolute_path_to_the_pdf_file/input.pdf", "rb")}
j="""
{
"cpf:engine": {
"repo:assetId": "urn:aaid:cpf:Service-26c7fda2890b44ad9a82714682e35888"
},
"cpf:inputs": {
"params": {
"cpf:inline": {
"targetFormat": "docx"
}
},
"documentIn": {
"dc:format": "application/pdf",
"cpf:location": "C:/Users/a-bensghir/Downloads/P_D_F/trs_pdf_file_copy.pdf"
}
},
"cpf:outputs": {
"documentOut": {
"dc:format": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"cpf:location": "C:/Users/a-bensghir/Downloads/P_D_F/output.docx"
}
}
}"""
resp = requests.post(url=URL, headers=headers, json=json.dumps(j), files=myfile)
print(resp.text)
print(resp.status_code)
The status of the code is 400
I am tho well authentified by the server
But I get the following as a result of print(resp.text) :
{"requestId":"the_request_id","type":"Bad Request","title":"Not a multipart request. Aborting.","status":400,"report":"{\"error_code\":\"INVALID_MULTIPART_REQUEST\"}"}
I think that I have problems understanding the "form parameters" from the Adobe Guide concerning POST method for the EXPORT job of the API (https://documentcloud.adobe.com/document-services/index.html).
Would you have any ideas for improvement. thank you !

Make you variable j as a python dict first then create a JSON string from it.
What's also not super clear from Adobe's documentation is the value for documentIn.cpf:location needs to be the same as the key used for you file. I've corrected this to InputFile0 in your script. Also guessing you want to save your file so I've added that too.
import requests
import json
import time
URL = "https://cpf-ue1.adobe.io/ops/:create?respondWith=%257B%2522reltype%2522%253A%2520%2522http%253A%252F%252Fns.adobe.com%252Frel%252Fprimary%2522%257D"
headers = {
'Authorization': f'Bearer {token}',
'Accept': 'application/json, text/plain, */*',
'x-api-key': client_id,
'Prefer': "respond-async,wait=0",
}
myfile = {"InputFile0":open("absolute_path_to_the_pdf_file/input.pdf", "rb")}
j={
"cpf:engine": {
"repo:assetId": "urn:aaid:cpf:Service-26c7fda2890b44ad9a82714682e35888"
},
"cpf:inputs": {
"params": {
"cpf:inline": {
"targetFormat": "docx"
}
},
"documentIn": {
"dc:format": "application/pdf",
"cpf:location": "InputFile0"
}
},
"cpf:outputs": {
"documentOut": {
"dc:format": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"cpf:location": "C:/Users/a-bensghir/Downloads/P_D_F/output.docx"
}
}
}
body = {"contentAnalyzerRequests": json.dumps(j)}
resp = requests.post(url=URL, headers=headers, data=body, files=myfile)
print(resp.text)
print(resp.status_code)
poll = True
while poll:
new_request = requests.get(resp.headers['location'], headers=headers)
if new_request.status_code == 200:
open('test.docx', 'wb').write(new_request.content)
poll = False
else:
time.sleep(5)

I don't know why the docx file (its well created by the way) doesn't open, telling via popup that the content is not readable. maybe it's due to the 'wb' parsing methos
I had the same issue. Typecasting to 'bytes' the request contents solved it.
poll = True
while poll:
new_request = requests.get(resp.headers['location'], headers=headers)
if new_request.status_code == 200:
with open('test.docx', 'wb') as f:
f.write(bytes(new_request.content))
poll = False
else:
time.sleep(5)

HTTP 400 error using Python Requests POST and Intercom API

I am trying to pull a list of conversations from Intercom using their API, restricted based on the date that they were updated.
Their site (https://developers.intercom.com/intercom-api-reference/reference#search-for-conversations) says
To search for conversations, you need to send a POST request to
https://api.intercom.io/conversations/search. This will accept a query
object in the body which will define your filters in order to search
for conversations.
So I tried:
import requests
url = 'https://api.intercom.io/conversations/search'
data_params = {
"query": {
"field": "updated_at",
"operator": ">",
"value": 1560436784
}
}
headers = {'Authorization' : 'Bearer ******************', 'Accept':'application/json', 'Content-Type':'application/json'}
r = requests.post(url, headers = headers, data = data_params)
r.status_code
This consistently runs into a 400 'Bad Request' error, but I'm struggling to see why. I've also tried json = data_params in the post call but same result. I appreciate the code might not be fully reproducible as it requires an authenticated API connection, but if there's anything obviously wrong it would be good to know!

Obviously, there seems no obviously error here.
for the 400 Bad Request . I think you should do the check of you data_params if there's something missing or in bad format. also the headers

You should try converting the data_params dictionary to json.
I tested with json.dumps and this seems to have worked:
import json
url = 'https://api.intercom.io/conversations/search'
data_params = {
"query": {
"field": "updated_at",
"operator": ">",
"value": 1560436784
}
}
headers = {'Authorization' : 'Bearer ******************', 'Accept':'application/json', 'Content-Type':'application/json'}
r = requests.post(url, headers = headers, data = json.dumps(data_params))
print(r.status_code)

Python requests Json body includes some thai values, needs to be encoded as it is

I have some json containing some thai values. It looks something like
{
"TitleName": "คุณ",
"FirstName": "Simar"
}
I need to make a Http POST request with this json body with the exact thai value.I am using Python 3 requests library to make the request.
I tried this
headers = {
'Content-Type': "application/json",
'Authorization': "xxx",
'cache-control': "no-cache",
'Postman-Token': "xxx"
}
response = requests.request("POST", url, json=request, headers=headers)
It generates json values as
"TitleName": "\\u0e04\\u0e38\\u0e13",
"FirstName": "Simar"
I also tried this
json_request = json.dumps(self.request_data,ensure_ascii=False).encode('utf8')
response = requests.request("POST", url, json=json_request, headers=headers)
It generates json values as
"TitleName": "\xe0\xb8\x84\xe0\xb8\xb8\xe0\xb8\x93",
"FirstName": "Simar"
But I want json values to be generated as
"TitleName": "คุณ",
"FirstName": "Simar"
Help will be appreciated. Thanks in advance.

To preserve non-ascii characters in POST requests you need to serialise to json manually, and explicitly set the content-type header.
data = json.dumps(my_dict, ensure_ascii=False)
r = requests.post(url, headers={'content-type': 'application/json'},
data=data.encode('utf-8'))

Send body of POST request

I inspected how a request was sent to a website in firefox:
(Unfortunately I had to change the website URL to a fake one to prevent the server form being requested too much).
I tried to do this request in python:
import requests
import json
seq = 'ATGGCAGACTCTATTGAGGTC'
url = 'http://www.test.com'
body = {'QUERY': seq}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(body), headers=headers)
print(r.text)
However when doing this the website says: Empty gene sequence passed for blast analysis. Please enter a valid gene sequence. So that means that the sequence (i.e. QUERY) is not sent correctly to the server. What am I missing here?
(P.s. hopefully missing of the website is not a problem to answer this question, if it is please let me know maybe I can ask to mention their website)

I am guessing the string / sequence that you are submitting to that particular website is the problem. I ran your sample code against a POST accepting website:
import requests
import json
seq = 'ATGGCAGACTCTATTGAGGTC'
url = 'http://httpbin.org/post'
body = {'QUERY': seq}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(body), headers=headers)
print(r.text)
And got this result, which shows your query properly formed:
{
"args": {},
"data": "{\"QUERY\": \"ATGGCAGACTCTATTGAGGTC\"}",
"files": {},
"form": {},
"headers": {
"Accept": "text/plain",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "34",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.22.0"
},
"json": {
"QUERY": "ATGGCAGACTCTATTGAGGTC"
},
"origin": "2.122.222.8, 2.122.222.8",
"url": "https://httpbin.org/post"
}

Are you sure the it's supposed to be "QUERY=<>?" Cause it could be incorrect formatting of the body. Normally it's in JSON format as in "title: information." Note the ':' rather than the '='