We Keep Coding

Parse JSON nested data from Requests POST response

I'm working on a project using Python(3.7) in which I have to parse a JSON returned from the POST request using Requests library.
I googled a lot and tried too many solutions but nothing helped me, so
don't mark this as duplicate, please!
Here's what I have tried:
def process_req(payload):
try:
headers = {
'Content-Type': 'application/json'
}
data = payload
resp = requests.post(
'http://<EXAMPLE_URL>',
data=data,
headers=headers
)
print('returned data: {}'.format(resp.content.decode('utf8').replace("'", '"')))
resp = resp.content.decode('utf8').replace("'", '"')
When I print the resp it provide the following JSON:
{
"code": "00",
"message": "Successful",
"data": "{\"requestId\":\"0012602\",\"responseCode\":\"68\",\"responseDescription\":\"Invalid Institution Code\"}"
}
Now, I need to access the data field of that JSON, here what I tried:
resp['data']
But it returns an error as:
string indices must be integers

You're retrieving the data as raw bytes by using resp.content.
Try resp.json() instead. This will decode the JSON into Python objects.

Pulling data from an api when the JSON record does not have a container name

I am trying to pull data from the api for our HR System. But when individual photo is pulled the JSON file does not have the container name anymore which breaks my current script.
For the available api when you request all employee photos in 1 request it only returns the thumbnail photo.To get the large photo you have to do it 1 employee at a time and inlclude the employeeid in the url. However the issue is that doing the latter does not present the object name only the items in the document.
import json
import pyodbc
import requests
url = "https://someurl.com/api/PersonPhoto"
headers = {
'Accept': "application/json",
'Authorization': "apikey xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
'Content-Type': "application/json",
'cache-control': "no-cache"
}
response = requests.request("GET", url, headers=headers)
data = json.loads(response.text)
ID,Photo,PhotoDate,PhotoType = [],[],[],[]
for device in data['PersonPhoto']:
ID.append(device[u'ID'])
Photo.append(device[u'Photo'])
PhotoDate.append(device[u'PhotoDate'])
PhotoType.append(device[u'PhotoType'])
connStr = pyodbc.connect(
"DRIVER={SQL Server};"
"SERVER=TestSrver;"
"Database=MyDB;"
"Trusted_Connection=yes;"}"
)
cursor = connStr.cursor()
sql = "INSERT INTO dbo.Employee_Photo2 ([EmployeeID],[PhotoBinary],[PhotoDate],[FileType]) VALUES (?,?,?,?)"
vals = [(int(device[u'ID']), device[u'Photo'], device[u'PhotoDate'], device[u'PhotoType']) \
for device in data['PersonPhoto']]
cursor.executemany(sql, vals)
connStr.commit()
cursor.close()
connStr.close()
The above code is working for when I pull all the employees. To pull 1 employee to get the large photo is the same api call except you have to put the employee id on the end of the url.
The issue I am having is that when you make this api call it no longer includes the object name of PersonPhoto in the resulting JSON it just returns the 4 attibutes inside a the curly brackets. So I am not sure how to change the above code to handle the lack of the object name.
JSON that works with above script:
{
"PersonPhoto": [
{
"ID": 123,
"Name": "Test User",
"Photo": "/9j/4AAQSkZJRgABAQEAYABgAAD",
"PhotoDate": "2019-05-02T00:00:00",
"PhotoType": ".jpg"
}
]
}
The JSON that gets returned when you pass the employeeid at the end of the URL:
{
"ID": 123,
"Name": "Test User",
"Photo": "/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAgGBgcGBQgHBwcJCQgKD",
"PhotoDate": "2019-05-02T00:00:00",
"PhotoType": ".jpg"
}

The short answer is that you can add exception handling with a try: except: statement to handle the error that occurs when referencing a non existent dictionary key.
From a purely functional standpoint that's a fine solution, but a better practice would probably be to write two separate functions to handle the expected JSON differently depending on the api call you are making
If you are simply ONLY going to handle the new use case, you can just reference the data this way:
user = "12345"
url = "https://someurl.com/api/PersonPhoto/{}".format(user)
response = requests.request("GET", url, headers=headers)
data = json.loads(response.text)
ID = [data['ID']]
Photo = [data['Photo']]
PhotoDate = [data['PhotoDate']]
PhotoType = [data['PhotoType']]

logging into the MATCHBOOK API with Python Requests

I am trying (legitimately and with the go ahead from the site)to log into the betting exchange matchbook.com through their api.
The documentation states:
To Login: https://www.matchbook.com/bpapi/rest/security/session
and
Example Request
POST /security/session
{
"username": "j_henry",
"password": "******"
}
Example Response
{
"session-token": "1418_1234567890",
"user-id": 1418,
"account": { // Same as GET /account API response.
...
}
}
I am using Requests and have the following code:
payload = {"username": "********", "password": "************"}
r = requests.post('https://www.matchbook.com/edge/rest/security/session', data=payload)
print (r.status_code)
I get error code 415? I must be getting the wrong type of response??
I have looked at a lot of very similar posts on here, and I am about to ask matchbook's team, but before I do has anybody got any ideas?

You might have to specify Content-Type, try to add a header to tell the server it's JSON formatted:
payload = {"username": "********", "password": "************"}
headers = {"Content-Type": "application/json;"}
r = requests.post('https://www.matchbook.com/edge/rest/security/session', data=payload, headers=headers)
print (r.status_code)

It does not appear from your code that you are JSON-encoding your payload. The endpoint is likely expecting JSON.
Try this:
payload = '{"username": "********", "password": "************"}'

How can I make a Post Request on Python with urllib3?

I've been trying to make a request to an API, I have to pass the following body:
{
"description":"Tenaris",
"ticker":"TS.BA",
"industry":"Metalúrgica",
"currency":"ARS"
}
Altough the code seems to be right and it finished with "Process finished with exit code 0", it's not working well. I have no idea of what I'm missing but this is my code:
http = urllib3.PoolManager()
http.urlopen('POST', 'http://localhost:8080/assets', headers={'Content-Type':'application/json'},
data={
"description":"Tenaris",
"ticker":"TS.BA",
"industry":"Metalúrgica",
"currency":"ARS"
})
By the way, this the first day working with Python so excuse me if I'm not specific enough.

Since you're trying to pass in a JSON request, you'll need to encode the body as JSON and pass it in with the body field.
For your example, you want to do something like:
import json
encoded_body = json.dumps({
"description": "Tenaris",
"ticker": "TS.BA",
"industry": "Metalúrgica",
"currency": "ARS",
})
http = urllib3.PoolManager()
r = http.request('POST', 'http://localhost:8080/assets',
headers={'Content-Type': 'application/json'},
body=encoded_body)
print r.read() # Do something with the response?
Edit: My original answer was wrong. Updated it to encode the JSON. Also, related question: How do I pass raw POST data into urllib3?

I ran into this issue when making a call to Gitlab CI. Since the above did not work for me (gave me some kind of error about not being able to concatenate bytes to a string), and because the arguments I was attempting to pass were nested, I thought I would post what ended up working for me:
API_ENDPOINT = "https://gitlab.com/api/v4/projects/{}/pipeline".format(GITLAB_PROJECT_ID)
API_TOKEN = "SomeToken"
data = {
"ref": ref,
"variables": [
{
"key": "ENVIRONMENT",
"value": some_env
},
{ "key": "S3BUCKET",
"value": some_bucket
},
]
}
req_headers = {
'Content-Type': 'application/json',
'PRIVATE-TOKEN': API_TOKEN,
}
http = urllib3.PoolManager()
encoded_data = json.dumps(data).encode('utf-8')
r = http.request('POST', API_ENDPOINT,
headers=req_headers,
body=encoded_data)
resp_body = r.data.decode('utf-8')
resp_dict = json.loads(r.data.decode('utf-8'))
logger.info('Response Code: {}'.format(r.status))
logger.info('Response Body: {}'.format(resp_body))
if 'message' in resp_body:
logfile_msg = 'Failed Gitlab Response-- {} {message}'.format(r.status, **resp_dict)

I recently became interested in using urllib3, and came across this problem. If you read the urllib3 "User Guide" page, you will see this:
For POST and PUT requests, you need to manually encode query parameters in the URL
Your code should be adjusted to look like this:
import urllib3
from urllib.parse import urlencode
data = {"description":"Tenaris",
"ticker":"TS.BA",
"industry":"Metalúrgica",
"currency":"ARS"}
http = urllib3.PoolManager()
encoded_data = urlencode(data)
http.request('POST',
'http://localhost:8080/assets?'+encoded_data,
headers={'Content-Type':'application/json'})

post request to change permissions for a file in google drive is failing

Im using python requests library to make google a drive api request to change permissions of a file, in this case the owner.
Here is what my code looks like
fileId = "123abcEfJl-mNooP45Kl6u" #fake file id
url = https://www.googleapis.com/drive/v2/files/%s/permissions' % fileId
payload = {"role":"owner", "type":"user", "value":"<some_user>#gmail.com"}
headers = {'Authorization': 'Bearer %s'%access_token, 'Content-Type':'application/json'}
permResponse = requests.post(url, data=payload, headers=headers)
print permResponse.text
When I run this, I get the following response:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "parseError",
"message": "Parse Error"
}
],
"code": 400,
"message": "Parse Error"
}
}
I've been following the google docs api for this and have not been able to figure out what im doing incorrectly.
https://developers.google.com/drive/v2/reference/permissions/insert
Even their Try It! section is broken because there isn't an option to add the required 'value' field.
What am I doing incorrectly? Is anyone else running into these issues?
Thanks

I'm using the urllib.request module, and It works fine. This is my code:
key = "?key=" + MY_API_KEY
url_destino = ("https://www.googleapis.com/drive/v2/files/%s/permissions" % source_id)+ key
values = "{"role":"owner", "type":"user", "value":"<some_user>#gmail.com"}"
data = values.encode('utf-8')
request = urllib.request.Request(url_destino, data, method='POST')
request.add_header("Authorization", "Bearer " + token)
request.add_header("Content-Length", len(data))
request.add_header("Content-Type", "application/json")
print(request.header_items()) # for debugging purpouse
f = urllib.request.urlopen(request)
print(f.read())
I've thought to replace the urllib.request by Requests module (it's more clean to work with) in my little library but, now works.
Because I use Python 3 I can't use the google-api-python-client.