I have a Node application that subscribes to JSON data streams. I would like to extend this to subscribe to Python pickle data streams (I am willing to drop or convert non primitive types). The node-pickle & jpickle packages have failed me. I now wish to write my own Python script to convert pickles to JSON.
I fiddled with the node-pickle source code to get part of it to work (can pass JSON from Node to Python and get back a pickle string, can also use a predefined Python dict and pass to Node as JSON). My problem is getting Python to recognize the data from Node as pickled data. I am passing the data stream buffer from Node to Python and trying desparately to get the string buffer argument into a format for me to pickle.loads it.
After much trial and error I have ended up with this:
main.js
const pickle = require('node-pickle');
const amqp = require('amqplib/callback_api');
amqp.connect(`amqp://${usr}:${pwd}#${url}`, (err, conn) => {
if (err) {
console.error(err);
}
conn.createChannel((err, ch) => {
if (err) {
console.error(err);
}
ch.assertExchange(ex, 'fanout', { durable: false });
ch.assertQueue('', {}, (err, q) => {
ch.bindQueue(q.queue, ex, '');
console.log('consuming');
ch.consume(q.queue, msg => {
console.log('Received [x]');
const p = msg.content.toString('base64');
pickle.loads(p).then(r => console.log('Res:', r));
// conn.close();
});
});
});
});
index.js (node-pickle)
const spawn = require('child_process').spawn,
Bluebird = require('bluebird');
module.exports.loads = function loads(pickle) {
return new Bluebird((resolve, reject) => {
const convert = spawn('python', [__dirname + '/convert.py', '--loads']),
stdout_buffer = [];
convert.stdout.on('data', function(data) {
stdout_buffer.push(data);
});
convert.on('exit', function(code) {
const data = stdout_buffer.join('');
// console.log('buffer toString', stdout_buffer[0] ? stdout_buffer[0].toString() : null);
if (data == -1) {
resolve(false);
} else {
let result;
try {
result = JSON.parse(data);
} catch (err) {
console.log('failed parse');
result = false;
}
resolve(result);
}
});
convert.stdin.write(pickle);
convert.stdin.end();
});
};
convert.py (node-pickle)
import sys
try:
import simplejson as json
except ImportError:
import json
try:
import cPickle as pickle
except ImportError:
import pickle
import codecs
import jsonpickle
def main(argv):
try:
if argv[0] == '--loads':
buffer = sys.stdin.buffer.read()
decoded = codecs.decode(buffer, 'base64')
d = pickle.loads(decoded, encoding='latin1')
j = jsonpickle.encode(d,False)
sys.stdout.write(j)
elif argv[0] == '--dumps':
d = json.loads(argv[1])
p = pickle.dumps(d)
sys.stdout.write(str(p))
except Exception as e:
print('Error: ' + str(e))
sys.stdout.write('-1')
if __name__ == '__main__':
main(sys.argv[1:])
The error I come up against at the moment is:
invalid load key, '\xef'
EDIT 1:
I am now sending the buffer string representation, instead of the buffer, to Python. I then use stdin to read it in as bytes. I started writing the bytes object to a file to compare to the data received from Node, to the buffer received when I subscribe to the data stream from a Python script. I have found that they seem to be identical, apart from certain \x.. sequences found when subscribing from Python, being represented as \xef\xbf\xbd when subscribing from Node. I assume this has something to do with string encoding?? Some examples of the misrepresented sequences are: \x80 (this is the first sequence after the b'; however \x80 does appear elsewhere), \xe3, and \x85.
EDIT 2:
I have now encoded the string I'm sending to Python as base64, then, in Python, decoding the stdin buffer using codecs.decode. The buffer I'm writing to the file now looks more identical to the Python only stream, with no more \xef\xbf\xbd substitutions. However, I now come up against this error:
'ascii' codec can't decode byte 0xe3 in position 1: ordinal not in range(128)
Also, I found a slight difference when trying to match the last 1000 characters of each stream. The is a section in the Python stream (\x0c,'\x023) that looks like this (\x0c,\'\x023) in the stream from Node. Not sure how much that'll affect things.
EDIT 3 (Success!):
After searching up my new error, I found the last piece of this encoding puzzle. Since I was working in Python 3, and the pickle came from Python 2.x, I needed to specify the encoding for pickle.loads as bytes or latin1(the one I needed). I was then able to make use of the wonderful jsonpickle package to do the work of JSON serializing the dict, changing datetime objects into date strings.
So I was able to get the node-pickle npm package to work. My flow of getting a buffer of pickled data from Node to Python to get back JSON is:
In Node
Encode the buffer as a base64 string
Send the string to the Python child process as a stdin input, not an argument
In Python
Read in the buffer from stdin as bytes
Use codecs to decode it from base64
If using Python 3, specify bytes or latin1 encoding for pickle.loads
Use jsonpickle to serialize python objects in JSON
In Node
Collect the buffer from stdout and JSON.parse it
Related
I'm trying to call a SimpleMotionv2 library from python using ctypes, it appears to make the call, but it always returns -1 when I feed it a valid com port.
I've built the DLL and the function I'm trying to call seems exported properly:
DLL Export Viewer Output
Somehow I'm passing the COM port number incorrectly? I'm on windows and its COM5...
The function I'm calling is smOpenBus() and is listed here:
smbus smOpenBus( const char * devicename )
{
int handle;
//true on first call
if(smInitialized==smfalse)
smBusesInit();
//find free handle
for(handle=0;handle<SM_MAX_BUSES;handle++)
{
if(smBus[handle].opened==smfalse) break;//choose this
}
//all handles in use
if(handle>=SM_MAX_BUSES) return -1;
//open bus device
smBus[handle].bdHandle=smBDOpen(devicename);
if(smBus[handle].bdHandle==-1) return -1;
//success
strncpy( smBus[handle].busDeviceName, devicename, SM_BUSDEVICENAME_LEN );
smBus[handle].busDeviceName[SM_BUSDEVICENAME_LEN-1]=0;//null terminate string
smBus[handle].opened=smtrue;
return handle;
}
And my Python code is:
import ctypes as ct
dll = ct.CDLL(r'./SMv2_x64.dll')
dll.smOpenBus.argtypes = ct.c_char_p,
dll.smOpenBus.restype = ct.c_long
ret_val = dll.smOpenBus(b'COM24')
print(ret_val)
What on Earth am I doing wrong?
I've tried making the ptr of type c_wchar_p and c_char_p, adding NULL terminators and string and not... I expect to get a COM port handle return and I get -1.
It’s argtypes (plural) and is a tuple. c_char_p is appropriate for C char*:
dll.smOpenBus.argtypes = ct.c_char_p, # comma makes a 1-tuple
Then pass a byte string:
ret_val = dll.smOpenBus(b'COM5')
I need to compress a file into a specific format that is required by our country's tax regulation entity and it has to be sent encoded in base64.
I work on Python3 and attempted to do the compression with the following code:
import gzip
# Work file generated before and stored in BytesBuffer
my_file = bytes_buffer.getvalue()
def compress(work_file):
encoded_work_file = base64.b64encode(work_file)
compressed_work_file = gzip.compress(encoded_work_file )
return base64.b64encode(compressed_work_file )
compress(my_file)
Now the tax entity returns an error message about an unknown compression format.
Luckily, they provided us the following Java example code:
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
public class DemoGZIP {
private final byte[] BUFFER = new byte[1024];
/**
* #param work_file File to compress
* The file is compressed over the original file name with the extension .zip
* #return boolean
* TRUE success
* FALSE failure
*/
public boolean compress(File work_file ) {
try (GZIPOutputStream out = new GZIPOutputStream (new FileOutputStream(work_file .getAbsolutePath() + ".zip"));
FileInputStream in = new FileInputStream(work_file )) {
int len;
while ((len = in.read(BUFFER)) != -1) {
out.write(BUFFER, 0, len);
}
out.close();
} catch (IOException ex) {
System.err.println(ex.getMessage());
return false;
}
return true;
}
The problem is that I do not have any experience working on Java and do not understand much of the provided code.
Can someone please help me adapt my code to do what the provided code does in python?
As noted in the comment, the Java code does not do Base64 coding, and names the resulting file incorrectly. It is most definitely not a zip file, it is a gzip file. The suffix should be ".gz". Though I doubt that the name matters to your tax agency.
More importantly, you are encoding with Base64 twice. From your description, you should only do that once, after gzip compression. From the Java code, you shouldn't do Base64 encoding at all! You need to get clarification on that.
I am making a REST API project that gets data from a python script and prints it via node js. The data is sent from the python script to node js with the following code:
json_dump = json.dumps(data)
print(data.encode("utf-8", "replace"))
And js gets the data with the following code:
PythonShell.run('script.py', options, function (err, data) {
if (err) throw err;
res.json(JSON.parse(data));
});
But I get the following error:
Unexpected token b in JSON at position 0
The JSON arrives correctly but starts with a 'b' and many characters are not getting printed or gets printed like this: "\xf0\x9f\xa4\x91". What can I do?
Remove the .encode("utf-8", "replace"). This converts the string to a bytes object (the representation starts with the b"...")
json_dump = json.dumps(data)
print(json_dump)
I am trying to communicate to a python script through actionscript. it gives me error on line :
var stdOut:ByteArray = process.standardOutput;
from the function shown below :
public function onOutputData(event:ProgressEvent):void
{
var stdOut:ByteArray = process.standardOutput; //error
var data:String = stdOut.readUTFBytes(process.standardOutput.bytesAvailable);
trace("Got: ", data);
}
Error is:
Implicit coercion of a value with static type IDataInput to a possibly
unrelated type ByteArray.
I am following the same approach as on Adobe's page. Here is some testable code :
package
{
import flash.display.Sprite;
import flash.desktop.NativeProcessStartupInfo;
import flash.filesystem.File;
import flash.desktop.NativeProcess;
import flash.events.ProgressEvent;
import flash.utils.ByteArray;
public class InstaUtility extends Sprite
{
public var nativeProcessStartupInfo:NativeProcessStartupInfo = new NativeProcessStartupInfo();
public var file:File = new File("C:/Python27/python.exe");
public var process:NativeProcess = new NativeProcess();
public function InstaUtility()
{
nativeProcessStartupInfo.executable = file;
nativeProcessStartupInfo.workingDirectory = File.applicationDirectory.resolvePath(".");
trace("Location " + File.applicationDirectory.resolvePath(".").nativePath);
var processArgs:Vector.<String> = new Vector.<String>();
processArgs[0] = "test.py";
nativeProcessStartupInfo.arguments = processArgs;
var process:NativeProcess = new NativeProcess();
process.addEventListener(ProgressEvent.STANDARD_OUTPUT_DATA, onOutputData);
process.start(nativeProcessStartupInfo);
}
public function onOutputData(event:ProgressEvent):void
{
var stdOut:ByteArray = process.standardOutput; //error
var data:String = stdOut.readUTFBytes(process.standardOutput.bytesAvailable);
trace("Got: ", data);
}
}
}
The NativeProcess could not be started. Not supported in current
profile.
Are you testing in Flash IDE?
Test within IDE : In your AIR Publish Settings make sure you ticked only "extended Desktop" when debugging through IDE. This way you also get traces etc.
Test after Publish : You must tick both "Desktop" and "extended Desktop" and also tick "Windows Installer (.exe)". Install your App using the generated .exe file (not the .air file).
Implicit coercion of a value with static type IDataInput to a possibly
unrelated type ByteArray.
var stdOut:ByteArray = process.standardOutput; //error is not how it's done!! Don't make any var each time the progress event fires up. Each firing holds around 32kb or 64kb of bytes only (can't remember), so if the expected result is larger it will continue to fire in multiple chunks... Use and recycle a single public byteArray to hold all the result data.
Try a setup like below :
//# Declare the public variables
public var stdOut : ByteArray = new ByteArray();
public var data_String : String = "";
Your process also needs a NativeProcessExitEvent.EXIT listener.
process.addEventListener(NativeProcessExitEvent.EXIT, on_Process_Exit );
Before you .start a process, also clear the byteArray ready for new data with stdOut.clear();.
Now your progressEvent can look like this below... (Process puts result data into stdOut bytes).
public function onOutputData (event:ProgressEvent) : void
{
//var stdOut:ByteArray = process.standardOutput; //error
//# Progress could fire many times so keep adding data to build the final result
//# "stdOut.length" will be zero at first but add more data to tail end (ie: length)
process.standardOutput.readBytes( stdOut, stdOut.length, process.standardOutput.bytesAvailable );
//# Below should be in a Process "Exit" listener but might work here too
stdOut.position = 0; //move pointer back before reading bytes
data_String = stdOut.readUTFBytes( stdOut.length );
trace("function onOutputData -- Got : " + data_String );
}
But you really need to add an "onProcessExit" listener and then only check for results when the process itself has completed. (Tracing here is much safer for a guaranteed result).
public function on_Process_Exit (event : NativeProcessExitEvent) : void
{
trace ("PYTHON Process finished : ############# " )
stdOut.position = 0; //# move pointer back before reading bytes
data_String = stdOut.readUTFBytes( stdOut.length );
trace("PYTHON Process Got : " + data_String );
}
Im trying to read in image file from a server , with the code below . It keeps going into the exception. I know the correct number of bytes are being sent as I print them out when received. Im sending the image file from python like so
#open the image file and read it into an object
imgfile = open (marked_image, 'rb')
obj = imgfile.read()
#get the no of bytes in the image and convert it to a string
bytes = str(len(obj))
#send the number of bytes
self.conn.send( bytes + '\n')
if self.conn.sendall(obj) == None:
imgfile.flush()
imgfile.close()
print 'Image Sent'
else:
print 'Error'
Here is the android part , this is where I'm having the problem. Any suggestions on the best way to go about receiving the image and writing it to a file ?
//read the number of bytes in the image
String noOfBytes = in.readLine();
Toast.makeText(this, noOfBytes, 5).show();
byte bytes [] = new byte [Integer.parseInt(noOfBytes)];
//create a file to store the retrieved image
File photo = new File(Environment.getExternalStorageDirectory(), "PostKey.jpg");
DataInputStream dis = new DataInputStream(link.getInputStream());
try{
os =new FileOutputStream(photo);
byte buf[]=new byte[1024];
int len;
while((len=dis.read(buf))>0)
os.write(buf,0,len);
Toast.makeText(this, "File recieved", 5).show();
os.close();
dis.close();
}catch(IOException e){
Toast.makeText(this, "An IO Error Occured", 5).show();
}
EDIT: I still cant seem to get it working. I have been at it since and the result of all my efforts have either resulted in a file that is not the full size or else the app crashing. I know the file is not corrupt before sending server side. As far as I can tell its definitely sending too as the send all method in python sends all or throws an exception in the event of an error and so far it has never thrown an exception. So the client side is messed up . I have to send the file from the server so I cant use the suggestion suggested by Brian .
The best way to get a bitmap from a server is to execute the following.
HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet("http://yoururl");
HttpResponse response = client.execute(get);
InputStream is = response.getEntity().getContent();
Bitmap image = BitmapFactory.decodeStream(is);
That will give you your bitmap, to save it to a file do something like the following.
FileOutputStream fos = new FileOutputStream("yourfilename");
image.compress(CompressFormat.PNG, 1, fos);
fos.close();
You can also combine the two and just write straight to disk
HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet("http://yoururl");
HttpResponse response = client.execute(get);
InputStream is = response.getEntity().getContent();
FileOutputStream fos = new FileOutputStream("yourfilename");
byte[] buffer = new byte[256];
int read = is.read(buffer);
while(read != -1){
fos.write(buffer, 0, read);
read = is.read(buffer);
}
fos.close();
is.close();
Hope this helps;
I'm not sure I understand your code. You are calling dis.readFully(bytes); to write the content of dis into your byte array. But then you don't do anything with the array, and then try to write the content of dis through a buffer into your FileOutputStream.
Try commenting out the line dis.readFully(bytes);.
As a side note, I would write to the log rather than popping up a toast for things like the number of bytes or when an exception occurs:
...
} catch (IOException e) {
Log.e("MyTagName","Exception caught " + e.toString());
e.printStackTrace();
}
You could look at these links for examples of writing a file to the SD card:
Android download binary file problems
Android write to sd card folder
I solved it with the help of a Ubuntu Forums member. It was the reading of the bytes that was the problem . It was cutting some of the bytes from the image. The solution was to just send the image whole and remove the sending of the bytes from the equation all together