We Keep Coding

Parse JSON nested data from Requests POST response

I'm working on a project using Python(3.7) in which I have to parse a JSON returned from the POST request using Requests library.
I googled a lot and tried too many solutions but nothing helped me, so
don't mark this as duplicate, please!
Here's what I have tried:
def process_req(payload):
try:
headers = {
'Content-Type': 'application/json'
}
data = payload
resp = requests.post(
'http://<EXAMPLE_URL>',
data=data,
headers=headers
)
print('returned data: {}'.format(resp.content.decode('utf8').replace("'", '"')))
resp = resp.content.decode('utf8').replace("'", '"')
When I print the resp it provide the following JSON:
{
"code": "00",
"message": "Successful",
"data": "{\"requestId\":\"0012602\",\"responseCode\":\"68\",\"responseDescription\":\"Invalid Institution Code\"}"
}
Now, I need to access the data field of that JSON, here what I tried:
resp['data']
But it returns an error as:
string indices must be integers

You're retrieving the data as raw bytes by using resp.content.
Try resp.json() instead. This will decode the JSON into Python objects.

Basic authentication not working in node-rest-client node

I'm using node-rest-client library to call my rest API running a python flask app in a test server. The Node setup and request code is below.
If I add a user token to the request header it words fine, but in the call to obtain the user token using basic auth my python system is successfully authenticating and returning the token with a 200 status, but the flask server is then changing this to a 400 Bad Request. It works fine making the call using Postman.
Is there something missing from my two config objects for node-rest-client?
Cheers.
var options_auth = {
user: "bob",
password: "password",
mimetypes: {
json: ["application/json", "application/json;charset=utf-8"]
}
};
var Client = require('node-rest-client').Client;
var client = new Client(options_auth);
var method = "/authtoken";
var args = {
headers: {
"Content-Type": "application/json",
"api-key": "asf89a7assa98d7sd98d98ds",
//"token": "dsf98s7dsf98dsf7sd98f7dsf",
"Accept": "application/json"
},
responseConfig: {
timeout: 1000 //response timeout
}
};
client.get(Api.url+method, args, function (data, response) {
// parsed response body as js object
// raw response
//console.log(response);
if(Buffer.isBuffer(data)){
data = data.toString('utf8');
}
console.log(data);
var stringData = data.toString('utf8');
console.log("String data = "+stringData);
}).on('error', function (err) {
console.error('Something went wrong with the http client', err);
});
Also, spotted these differences between the request headers received by the server:
// Node Request fails: 400
'headers': EnvironHeaders([
('Authorization', u'Basic XXXXXXXXXXXXXX=='),
('Vga-Api-Key', u'xxxxxxxxxxxxxxxxxxxxxxxx'),
('Content-Length', u'0'),
('Connection', u'close'),
('Host', u'127.0.0.1:8080'),
('Accept', u'*/*'),
('Content-Type', u'application/json')]),
// Postman Request works: 200
'headers': EnvironHeaders([
('Authorization', u'Basic XXXXXXXXXXXXXX=='),
('Vga-Api-Key', u'xxxxxxxxxxxxxxxxxxxxxxxx'),
* ('Content-Length', u''),
* ('Connection', u'keep-alive'),
('Host', u'127.0.0.1:8080'),
* ('Cache-Control', u'no-cache'),
('Accept', u'*/*'),
('Content-Type', u''),
* ('Accept-Encoding', u'gzip, deflate')]),

The problem is your setting of the header Content-Type: application/json and the probable calling in the server of request.get_json() directly, or indirectly via the (deprecated) request.json property.
When get_json() is called Flask will check to see that a JSON payload has been sent in the body of the request and then parse it if present. That's OK if the request actually contains JSON in the body, but, being a GET request, yours doesn't. In this case, its JSON expectation being unfulfilled, the server raises a BadRequest error and returns a HTTP 400 error response.
From what you've shown your request doesn't need to be JSON because the authorisation username and password are passed in the Authorization: Basic xxxxxxxx header.
The easiest and best way to fix the problem is to simply omit the content type header.
Alternatively you can tell Flask not to complain if there is no JSON data to parse by passing silent=True to get_json, but this just papers over the problem and is not a good idea.

logging into the MATCHBOOK API with Python Requests

I am trying (legitimately and with the go ahead from the site)to log into the betting exchange matchbook.com through their api.
The documentation states:
To Login: https://www.matchbook.com/bpapi/rest/security/session
and
Example Request
POST /security/session
{
"username": "j_henry",
"password": "******"
}
Example Response
{
"session-token": "1418_1234567890",
"user-id": 1418,
"account": { // Same as GET /account API response.
...
}
}
I am using Requests and have the following code:
payload = {"username": "********", "password": "************"}
r = requests.post('https://www.matchbook.com/edge/rest/security/session', data=payload)
print (r.status_code)
I get error code 415? I must be getting the wrong type of response??
I have looked at a lot of very similar posts on here, and I am about to ask matchbook's team, but before I do has anybody got any ideas?

You might have to specify Content-Type, try to add a header to tell the server it's JSON formatted:
payload = {"username": "********", "password": "************"}
headers = {"Content-Type": "application/json;"}
r = requests.post('https://www.matchbook.com/edge/rest/security/session', data=payload, headers=headers)
print (r.status_code)

It does not appear from your code that you are JSON-encoding your payload. The endpoint is likely expecting JSON.
Try this:
payload = '{"username": "********", "password": "************"}'

How can I make a Post Request on Python with urllib3?

I've been trying to make a request to an API, I have to pass the following body:
{
"description":"Tenaris",
"ticker":"TS.BA",
"industry":"Metalúrgica",
"currency":"ARS"
}
Altough the code seems to be right and it finished with "Process finished with exit code 0", it's not working well. I have no idea of what I'm missing but this is my code:
http = urllib3.PoolManager()
http.urlopen('POST', 'http://localhost:8080/assets', headers={'Content-Type':'application/json'},
data={
"description":"Tenaris",
"ticker":"TS.BA",
"industry":"Metalúrgica",
"currency":"ARS"
})
By the way, this the first day working with Python so excuse me if I'm not specific enough.

Since you're trying to pass in a JSON request, you'll need to encode the body as JSON and pass it in with the body field.
For your example, you want to do something like:
import json
encoded_body = json.dumps({
"description": "Tenaris",
"ticker": "TS.BA",
"industry": "Metalúrgica",
"currency": "ARS",
})
http = urllib3.PoolManager()
r = http.request('POST', 'http://localhost:8080/assets',
headers={'Content-Type': 'application/json'},
body=encoded_body)
print r.read() # Do something with the response?
Edit: My original answer was wrong. Updated it to encode the JSON. Also, related question: How do I pass raw POST data into urllib3?

I ran into this issue when making a call to Gitlab CI. Since the above did not work for me (gave me some kind of error about not being able to concatenate bytes to a string), and because the arguments I was attempting to pass were nested, I thought I would post what ended up working for me:
API_ENDPOINT = "https://gitlab.com/api/v4/projects/{}/pipeline".format(GITLAB_PROJECT_ID)
API_TOKEN = "SomeToken"
data = {
"ref": ref,
"variables": [
{
"key": "ENVIRONMENT",
"value": some_env
},
{ "key": "S3BUCKET",
"value": some_bucket
},
]
}
req_headers = {
'Content-Type': 'application/json',
'PRIVATE-TOKEN': API_TOKEN,
}
http = urllib3.PoolManager()
encoded_data = json.dumps(data).encode('utf-8')
r = http.request('POST', API_ENDPOINT,
headers=req_headers,
body=encoded_data)
resp_body = r.data.decode('utf-8')
resp_dict = json.loads(r.data.decode('utf-8'))
logger.info('Response Code: {}'.format(r.status))
logger.info('Response Body: {}'.format(resp_body))
if 'message' in resp_body:
logfile_msg = 'Failed Gitlab Response-- {} {message}'.format(r.status, **resp_dict)

I recently became interested in using urllib3, and came across this problem. If you read the urllib3 "User Guide" page, you will see this:
For POST and PUT requests, you need to manually encode query parameters in the URL
Your code should be adjusted to look like this:
import urllib3
from urllib.parse import urlencode
data = {"description":"Tenaris",
"ticker":"TS.BA",
"industry":"Metalúrgica",
"currency":"ARS"}
http = urllib3.PoolManager()
encoded_data = urlencode(data)
http.request('POST',
'http://localhost:8080/assets?'+encoded_data,
headers={'Content-Type':'application/json'})

How to make a connection to Parse via requests instead of httplib

I'm trying to connect to a custom endpoint I've set up on Parse Cloud Code. In the docs they show how to make a connection:
import json,httplib
connection = httplib.HTTPSConnection('api.parse.com', 443)
connection.connect()
connection.request('POST', '/1/functions/userData', json.dumps({
"userID": "1234"
}), {
"X-Parse-Application-Id": "####",
"X-Parse-REST-API-Key": "####",
"Content-Type": "application/json"
})
result = json.loads(connection.getresponse().read())
print result
This work great.
I'm trying to write it with requests but I continually get a 404:
import json, requests
PARSE_HOSTNAME = 'https://api.parse.com:443'
PARSE_APP_ID = '####'
PARSE_REST_API_KEY = '####'
endpoint = '/1/function/userData/'
headers = {"X-Parse-Application-Id": PARSE_APP_ID,
"X-Parse-REST-API-Key": "PARSE_REST_API_KEY",
"Content-Type": "application/json"}
payload = {'userID': '1234'}
r = requests.post(PARSE_HOSTNAME + endpoint, data=json.dumps(payload), headers=headers)
print r.json
Which prints out:
<bound method Response.json of <Response [404]>>
I feel like I'm missing something really obvious. Sorry for the ignorance.
How do I accomplish the same result as the first but with requests?

URL misspelling in the second example. You have
/1/function/userData/
must be
/1/functions/userData