We Keep Coding

Search for specific information within a json file

So I am trying to locate and acquire data from an api, I am fine with actually getting the data which is in json format from the api into my python program, however I am having troubles searching through the json for the specific data I want.
Here is a basic idea of what the json file from the api looks like:
{
"data": {
"inventory": {
"object_name": {
"amount": 8,
},
(Obviously the } close, I just didn't copy them)
I am trying to locate the amount within the json file of a specific object.
So far, here is the code I have, however, I have run into the error json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
I have researched the error and it appears to be caused usually by a faulty json file, however as I have imported the json file from an api, it having issues is not the case and must be an issue with some of the converting to strings and such I have done in my code.
data = requests.get('[api]',
headers={
"[api key name]" : "[api key]"
})
dataJson = data.json()
dataStr = str(dataJson)
amt = json.loads(dataStr)['data'].get('inventory').get('object_name').get('amount')
As stated previously, the main issue I have is actually collecting the data I need from the json endpoint, everything is fine with getting the data into the python script.

dataJson = data.json() is already python dict no need to json.loads(dataStr) just use
data = requests.get('[api]',
headers={
"[api key name]" : "[api key]"
})
dataJson = data.json()
amt = dataStr['data'].get('inventory').get('object_name').get('amount')

Pass a pickle buffer from Node to Python

I have a Node application that subscribes to JSON data streams. I would like to extend this to subscribe to Python pickle data streams (I am willing to drop or convert non primitive types). The node-pickle & jpickle packages have failed me. I now wish to write my own Python script to convert pickles to JSON.
I fiddled with the node-pickle source code to get part of it to work (can pass JSON from Node to Python and get back a pickle string, can also use a predefined Python dict and pass to Node as JSON). My problem is getting Python to recognize the data from Node as pickled data. I am passing the data stream buffer from Node to Python and trying desparately to get the string buffer argument into a format for me to pickle.loads it.
After much trial and error I have ended up with this:
main.js
const pickle = require('node-pickle');
const amqp = require('amqplib/callback_api');
amqp.connect(`amqp://${usr}:${pwd}#${url}`, (err, conn) => {
if (err) {
console.error(err);
}
conn.createChannel((err, ch) => {
if (err) {
console.error(err);
}
ch.assertExchange(ex, 'fanout', { durable: false });
ch.assertQueue('', {}, (err, q) => {
ch.bindQueue(q.queue, ex, '');
console.log('consuming');
ch.consume(q.queue, msg => {
console.log('Received [x]');
const p = msg.content.toString('base64');
pickle.loads(p).then(r => console.log('Res:', r));
// conn.close();
});
});
});
});
index.js (node-pickle)
const spawn = require('child_process').spawn,
Bluebird = require('bluebird');
module.exports.loads = function loads(pickle) {
return new Bluebird((resolve, reject) => {
const convert = spawn('python', [__dirname + '/convert.py', '--loads']),
stdout_buffer = [];
convert.stdout.on('data', function(data) {
stdout_buffer.push(data);
});
convert.on('exit', function(code) {
const data = stdout_buffer.join('');
// console.log('buffer toString', stdout_buffer[0] ? stdout_buffer[0].toString() : null);
if (data == -1) {
resolve(false);
} else {
let result;
try {
result = JSON.parse(data);
} catch (err) {
console.log('failed parse');
result = false;
}
resolve(result);
}
});
convert.stdin.write(pickle);
convert.stdin.end();
});
};
convert.py (node-pickle)
import sys
try:
import simplejson as json
except ImportError:
import json
try:
import cPickle as pickle
except ImportError:
import pickle
import codecs
import jsonpickle
def main(argv):
try:
if argv[0] == '--loads':
buffer = sys.stdin.buffer.read()
decoded = codecs.decode(buffer, 'base64')
d = pickle.loads(decoded, encoding='latin1')
j = jsonpickle.encode(d,False)
sys.stdout.write(j)
elif argv[0] == '--dumps':
d = json.loads(argv[1])
p = pickle.dumps(d)
sys.stdout.write(str(p))
except Exception as e:
print('Error: ' + str(e))
sys.stdout.write('-1')
if __name__ == '__main__':
main(sys.argv[1:])
The error I come up against at the moment is:
invalid load key, '\xef'
EDIT 1:
I am now sending the buffer string representation, instead of the buffer, to Python. I then use stdin to read it in as bytes. I started writing the bytes object to a file to compare to the data received from Node, to the buffer received when I subscribe to the data stream from a Python script. I have found that they seem to be identical, apart from certain \x.. sequences found when subscribing from Python, being represented as \xef\xbf\xbd when subscribing from Node. I assume this has something to do with string encoding?? Some examples of the misrepresented sequences are: \x80 (this is the first sequence after the b'; however \x80 does appear elsewhere), \xe3, and \x85.
EDIT 2:
I have now encoded the string I'm sending to Python as base64, then, in Python, decoding the stdin buffer using codecs.decode. The buffer I'm writing to the file now looks more identical to the Python only stream, with no more \xef\xbf\xbd substitutions. However, I now come up against this error:
'ascii' codec can't decode byte 0xe3 in position 1: ordinal not in range(128)
Also, I found a slight difference when trying to match the last 1000 characters of each stream. The is a section in the Python stream (\x0c,'\x023) that looks like this (\x0c,\'\x023) in the stream from Node. Not sure how much that'll affect things.
EDIT 3 (Success!):
After searching up my new error, I found the last piece of this encoding puzzle. Since I was working in Python 3, and the pickle came from Python 2.x, I needed to specify the encoding for pickle.loads as bytes or latin1(the one I needed). I was then able to make use of the wonderful jsonpickle package to do the work of JSON serializing the dict, changing datetime objects into date strings.

So I was able to get the node-pickle npm package to work. My flow of getting a buffer of pickled data from Node to Python to get back JSON is:
In Node
Encode the buffer as a base64 string
Send the string to the Python child process as a stdin input, not an argument
In Python
Read in the buffer from stdin as bytes
Use codecs to decode it from base64
If using Python 3, specify bytes or latin1 encoding for pickle.loads
Use jsonpickle to serialize python objects in JSON
In Node
Collect the buffer from stdout and JSON.parse it

String to dict/json like format [duplicate]

This question already has answers here:
Get the data received in a Flask request
(23 answers)
How to get POSTed JSON in Flask?
(13 answers)
Closed 4 years ago.
I'm finding it difficult to turn a json like string into a json/dict object, I've got the data like this
{"key1":"value1","key2":[{"key2.1":"value2.1"},{"key2.2":"value2.2"}]}
When running type on the variable it tells me it is of class str i.e <class 'str'>. I need to get the string into a json or dict like format so I can extract all of the values named value...
I've tried json.loads, json.load, json.dumps, ast.literal_eval but nothing seems to be working. I've tried adding [' and '] either side of the string but still no luck. Any ideas? Thanks
EDIT:
I'm using a nodeJS backend sending a request to a flask server, the nodeJS axios request is this
get_values: (key_one, key_two) =>
axios.post('http://localhost:5000/example', {
key_one: key_one,
key_two: key_two
}).then(res => res.data),
on the flask side I'm doing this
#app.route('/example', methods=["POST","OPTIONS"])
def example():
convert_data = request.get_data()
string_data = convert_data.decode('utf8').replace("'", '"')
new_string = "'''" + string_data + "'''"
print(json.loads(new_string))
Then I get the an error

I modified your function a bit:
#app.route('/example', methods=["POST","OPTIONS"])
def example():
convert_data = request.get_data()
string_data = convert_data.decode('utf8')
print(json.loads(string_data))
It seems string_data is already a perfectly formatted json string that you can pass into loads. I removed the replacement of ' with " which seems unnecessary, and definitely the addition of ''', which will make this string become Python multiline string literal syntax but definitely break json compatibility, which is what your error tells you too:
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
This should be a lot closer to what you wanted. Otherwise let me know what exactly is inside string_data when you run this.

Not sure if I understand the question correctly. But, let's say you have a string as follows:
var a = {
"key1": "value1",
"key2": [{
"key2.1": "value2.1"
}, {
"key2.2": "value2.2"
}]
}
Then you can do something like this:
try {
const dictionary = JSON.parse(a)
console.log(dictionary.key1) // This will print value1
Object.keys(dictionary).map((key) => {
console.log(dictionary[key]);
});
} catch(e){
// do something with error here
}

Non-latin text outputting as nonsense in Python

I've got a script which makes a JSON request that may return text in any script, then outputs the text (I dont have any control over the text being returned).
It works fine with latin characters, but other scripts output as a mojibake, and I'm not sure what's going wrong.
In the response, the problematic characters are encoded using \u syntax. In particular, I have a string containing \u00d0\u00b8\u00d1\u0081\u00d0\u00bf\u00d1\u008b\u00d1\u0082\u00d0\u00b0\u00d0\u00bd\u00d0\u00b8\u00d0\u00b5 which should output as испытание but instead outputs as иÑпÑÑание.
Obviously this is something to do with how python deals with unicode and UTF, but I despite all I've read I don't understand what's going on well enough to know how to solve it.
I've tried to extract the salient points from the code below:
response = requests.get(url, params=params, cookies=self.cookies, auth=self.auth)
text = response.text
print text
status = json.loads(text)
print status
for folder in status['folders']
print folder['name']
Output:
{ "folders": [ { "name": "\u00d0\u00b8\u00d1\u0081\u00d0\u00bf\u00d1\u008b\u00d1\u0082\u00d0\u00b0\u00d0\u00bd\u00d0\u00b8\u00d0\u00b5" } ] }
{u'folders': [{ u'name': u'\xd0\xb8\xd1\x81\xd0\xbf\xd1\x8b\xd1\x82\xd0\xb0\xd0\xbd\xd0\xb8\xd0\xb5' }]}
иÑпÑÑание
I've also tried
status = response.json();
for folder in status['folders']:
print folder['name']
With the same result.
Note, I'm really passing the string to a GTKMenuItem to be displayed, but the output from printing the string is the same as from showing it in the menu.

As #Ricardo Cárdenes said in the comment the server sends incorrect response. The response that you've got is double encoded:
>>>> u = u'\xd0\xb8\xd1\x81\xd0\xbf\xd1\x8b\xd1\x82\xd0\xb0\xd0\xbd\xd0\xb8\xd0\xb5'
>>>> print u.encode('latin-1').decode('utf-8')
испытание
The correct string would look like:
>>>> s = {u"name": u"испытание"}
>>>> import json
>>>> print json.dumps(s)
{"name": "\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435"}
>>>> print s['name']
испытание
>>>> print s['name'].encode('unicode-escape')
\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435
>>>> print s['name'].encode('utf-8')
испытание
>>>> s['name'].encode('utf-8')
'\xd0\xb8\xd1\x81\xd0\xbf\xd1\x8b\xd1\x82\xd0\xb0\xd0\xbd\xd0\xb8\xd0\xb5'

Android , Read in binary data and write it to file

Im trying to read in image file from a server , with the code below . It keeps going into the exception. I know the correct number of bytes are being sent as I print them out when received. Im sending the image file from python like so
#open the image file and read it into an object
imgfile = open (marked_image, 'rb')
obj = imgfile.read()
#get the no of bytes in the image and convert it to a string
bytes = str(len(obj))
#send the number of bytes
self.conn.send( bytes + '\n')
if self.conn.sendall(obj) == None:
imgfile.flush()
imgfile.close()
print 'Image Sent'
else:
print 'Error'
Here is the android part , this is where I'm having the problem. Any suggestions on the best way to go about receiving the image and writing it to a file ?
//read the number of bytes in the image
String noOfBytes = in.readLine();
Toast.makeText(this, noOfBytes, 5).show();
byte bytes [] = new byte [Integer.parseInt(noOfBytes)];
//create a file to store the retrieved image
File photo = new File(Environment.getExternalStorageDirectory(), "PostKey.jpg");
DataInputStream dis = new DataInputStream(link.getInputStream());
try{
os =new FileOutputStream(photo);
byte buf[]=new byte[1024];
int len;
while((len=dis.read(buf))>0)
os.write(buf,0,len);
Toast.makeText(this, "File recieved", 5).show();
os.close();
dis.close();
}catch(IOException e){
Toast.makeText(this, "An IO Error Occured", 5).show();
}
EDIT: I still cant seem to get it working. I have been at it since and the result of all my efforts have either resulted in a file that is not the full size or else the app crashing. I know the file is not corrupt before sending server side. As far as I can tell its definitely sending too as the send all method in python sends all or throws an exception in the event of an error and so far it has never thrown an exception. So the client side is messed up . I have to send the file from the server so I cant use the suggestion suggested by Brian .

The best way to get a bitmap from a server is to execute the following.
HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet("http://yoururl");
HttpResponse response = client.execute(get);
InputStream is = response.getEntity().getContent();
Bitmap image = BitmapFactory.decodeStream(is);
That will give you your bitmap, to save it to a file do something like the following.
FileOutputStream fos = new FileOutputStream("yourfilename");
image.compress(CompressFormat.PNG, 1, fos);
fos.close();
You can also combine the two and just write straight to disk
HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet("http://yoururl");
HttpResponse response = client.execute(get);
InputStream is = response.getEntity().getContent();
FileOutputStream fos = new FileOutputStream("yourfilename");
byte[] buffer = new byte[256];
int read = is.read(buffer);
while(read != -1){
fos.write(buffer, 0, read);
read = is.read(buffer);
}
fos.close();
is.close();
Hope this helps;

I'm not sure I understand your code. You are calling dis.readFully(bytes); to write the content of dis into your byte array. But then you don't do anything with the array, and then try to write the content of dis through a buffer into your FileOutputStream.
Try commenting out the line dis.readFully(bytes);.
As a side note, I would write to the log rather than popping up a toast for things like the number of bytes or when an exception occurs:
...
} catch (IOException e) {
Log.e("MyTagName","Exception caught " + e.toString());
e.printStackTrace();
}
You could look at these links for examples of writing a file to the SD card:
Android download binary file problems
Android write to sd card folder

I solved it with the help of a Ubuntu Forums member. It was the reading of the bytes that was the problem . It was cutting some of the bytes from the image. The solution was to just send the image whole and remove the sending of the bytes from the equation all together