We Keep Coding

Send and receive objects through WebRTC data channel

In my project, I am using WebRTC to connect between 2 client using the aiortc package.
I am using this example code and it works, but it seems I can't send non-string data in the data channel.
This is what I send in the data channel (modified the code in the start function in client.js file):
dc.onopen = function() {
dataChannelLog.textContent += '- open\n';
dcInterval = setInterval(function() {
let message = new DataObject(/*All the parameters*/);
dataChannelLog.textContent += '> ' + message + '\n';
dc.send(message);
}, 100);
};
Where DataObject is a class I created that contains the data I want to send.
The Python client receives [object Object] as a string. I expected it will send the bytes representing the object that I can convert back in Python to a normal class.
I know that a workaround for this is to convert the object to a string-format (like JSON), but I prefer not to do it because I am sending the objects very frequently (and every object contains a large array in it) and I am sure it will lead to a performance issues.
So my question is, how can I send the object through the data channel without converting to a string¿
EDIT: If it helps, I can use an array instead of an object to represent my data. But again, it is still sent and received as a string.

You need some sort of serializer function to convert your Javascript object into a stream of bytes. Those bytes don't have to be readable as text. You can't just send a Javascript object.
The built-in robust and secure serializer is, of course JSON.stringify(). As you've pointed out JSON is a verbose format.
To avoid using JSON, you'll need to create your own serializer in Javascript and deserializer in Python. Those will most likely be custom code for your particular object type. For best results, you'll copy the attributes of your object, one by one, into a Uint8Array, then send that.
You didn't tell us anything about your object, so it's hard to help you further.
If this were my project I'd get everything working with JSON and then do the custom serialization as a performance enhancement.

Thanks o-jones for the detailed answer.
In my case it was fairly simple, because I was able to represent all my data as an array.
The main issue I had is that I didn't know the send function has an "overload" that accepts bytes array…
After realizing that, I created a Float32Array in Javascript to hold my data and send it.
And in the Python side, I read this data and converted it to a float array using struct.unpack function.
Something like that:
Javascript side:
dc.onopen = function() {
dataChannelLog.textContent += '- open\n';
dcInterval = setInterval(function() {
let dataObj = new DataObject(/*All the parameters*/);
let data = new Float32Array(3); // Build an array (in my case 3 floats array)
// Insert the data into the array
data[0] = dataObj.value1;
data[1] = dataObj.value2;
data[2] = dataObj.value3;
// Send the data
dc.send(data);
}, 100);
};
Python side:
import struct
def message_received(message: str | bytes) -> None:
if isinstance(message, str):
return # Don't handle string messages
# Read 3 floats from the bytes array we received.
data = struct.unpack('3f', message)

TypeError: Argument 'payload' has incorrect type (expected bytes, got str). How can I fix it? Why it works on Python 2 but not using Python 3?

I am pretty new in Python and I have the following problem using *scapy** library. Here you can find the entire code (but I think that it is not so important because the error is on a specific line: https://github.com/AndreaNobili/replace_download/blob/master/replace_download.py)
Into a Python 2 project I have the following two lines:
modified_packet = set_load(scapy_packet, "HTTP/1.1 301 Moved Permanently\nLocation: https://www.rarlab.com/rar/wrar590.exe\n\n")
# Replace the original packet payload with the packet forget by scapy:
packet.set_payload(str(modified_packet))
This is the code of my set_load() function:
def set_load(packet, load):
#pdb.set_trace()
print("set_load() START")
# When the victim try to download a ".exe" file he\she is redirected to this other ".exe" link:
packet[scapy.Raw].load = load
# The value of the following fields are changed because the file is changed, they will be removed and
# scapy automatically recalculate the values of these fields inserting the correct values:
del packet[scapy.IP].len
del packet[scapy.IP].chksum
del packet[scapy.TCP].chksum
return packet
So basically I am forging a packet using scapy, finally I am setting the payload of the original packet variable with the payload forged by Scapy:
packet.set_payload(str(modified_packet))
NOTE: The packet variable is not a **scapy packet but a packet obtained using netfilterqueue
Running my script with Python 2 it works fine but using Python 3 this last line give me the following error:
TypeError: Argument 'payload' has incorrect type (expected bytes, got str)
> /root/Documents/PycharmWS/replace_download/replace_download.py(61)process_packet()
-> packet.set_payload(str(modified_packet))
So I am converting the scapy packet into a string and then I am setting the payload of the original netfilterqueue packet but it seems that it is expecting a bytes
How can I fix this problem? What am I missing?
Another doubt is: why Python 2 it is working fine? I suspect that the netfilterqueue dependency version used by Python 2 is slightly different from the one used by Python 3 and in the old version expected a string instead a bytes parameter. Is this reasoning correct or am I missing something?

Python 3 uses "bytes" on the wire. Instead of using str(), use bytes(). Have a look at http://python-future.org/compatible_idioms.html#strings-and-bytes for a great comparison of what you should be doing on Py3 vs on Py2.
In your case, just do
packet.set_payload(bytes(modified_packet))

Python Azure Queue, getting error

I am struggling with an encoding issue. I am still trying to figure out the Python3 encoding scheme. I am trying to upload a json object from Python into an Azure Queue. I am using Python3
I make the json object
response = {"UserImageId": 636667744866847370, "OutputImageName": "car-1807177_with_blue-2467336_size_1020_u38fa38.png"}
queue_service.put_message(response_queue, json.dumps(response))
When it gets to the queue, I get the error
{"imgResponse":"The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. ","log":null,"$return":""}
So I have to do something else, because apparently I need to base64 encode my string. So I try
queue_service.put_message(response_queue, base64.b64encode(json.dumps(response).encode('utf-8')))
and I get
TypeError: message should be of type str
From the Azure Storage Queue package. If I check the type of the above statement, it is of type bytes (makes sense).
So my question is, how do I encode my json object into something that the queue service will understand. I would really like to be able to keep the _ and - and . characters in the image name.

If anyone is looking to solve this problem using QueueClient rather than QueueService, here is what worked for me:
import json
from azure.storage.queue import QueueServiceClient, QueueClient, QueueMessage, TextBase64EncodePolicy
conn_string = '[YOUR_CONNECTION_STRING_HERE]'
queue_client = QueueClient.from_connection_string(
conn_string,
'[QUEUE_NAME_HERE]',
message_encode_policy=TextBase64EncodePolicy()
)
queue_client.send_message(json.dumps({'a':'b'}))

this is what I had to do in my code to make it work:
queue_service = QueueService(account_name=os.getenv('storageAccount'), account_key=os.getenv('storageKey'))
queue_service.encode_function = QueueMessageFormat.text_base64encode
after that I could just put messages:
queue_service.put_message('bbbb', message) # 'bbbb' is a queue name

Deserialize list of objects using protobuf

I'm building a C# server and python client app with socket communication. The server sends serialized list of objects to client, but I've got no idea (and couldn't find either) how to deserialize a list in python. Any help would be appreciated.

Allright, I found solution if anyone is interested. The trick is to create a new message type and add original one as repeated. Here's how
message TransactionPackets {
repeated TransactionPacket packet = 1;
}
message TransactionPacket {
required int32 trans_id = 1;
required string user_id = 2;
required int64 date = 3;
}
Now I can simply deserialize a list of objects by calling TransactionPackets.ParseFromString()

Check this:
"The Protocol Buffer wire format is not self-delimiting, so protocol buffer parsers cannot determine where a message ends on their own. The easiest way to solve this problem is to write the size of each message before you write the message itself. When you read the messages back in, you read the size, then read the bytes into a separate buffer, then parse from that buffer."
https://developers.google.com/protocol-buffers/docs/techniques

How to read JSON from socket in python? (Incremental parsing of JSON)

I have a socket opened and I'd like to read some json data from it. The problem is that the json module from standard library can only parse from strings (load only reads the whole file and calls loads inside) It even looks that all the way inside the module it all depends on the parameter being string.
This is a real problem with sockets since you can never read it all to string and you don't know how many bytes to read before you actually parse it.
So my questions are: Is there a (simple and elegant) workaround? Is there another json library that can parse data incrementally? Is it worth writing it myself?
Edit: It is XBMC jsonrpc api. There are no message envelopes, and I have no control over the format. Each message may be on a single line or on several lines.
I could write some simple parser that needs only getc function in some form and feed it using s.recv(1), but this doesn't as a very pythonic solution and I'm a little lazy to do that :-)

Edit: given that you aren't defining the protocol, this isn't useful, but it might be useful in other contexts.
Assuming it's a stream (TCP) socket, you need to implement your own message framing mechanism (or use an existing higher level protocol that does so). One straightforward way is to define each message as a 32-bit integer length field, followed by that many bytes of data.
Sender: take the length of the JSON packet, pack it into 4 bytes with the struct module, send it on the socket, then send the JSON packet.
Receiver: Repeatedly read from the socket until you have at least 4 bytes of data, use struct.unpack to unpack the length. Read from the socket until you have at least that much data and that's your JSON packet; anything left over is the length for the next message.
If at some point you're going to want to send messages that consist of something other than JSON over the same socket, you may want to send a message type code between the length and the data payload; congratulations, you've invented yet another protocol.
Another, slightly more standard, method is DJB's Netstrings protocol; it's very similar to the system proposed above, but with text-encoded lengths instead of binary; it's directly supported by frameworks such as Twisted.

If you're getting the JSON from an HTTP stream, use the Content-Length header to get the length of the JSON data. For example:
import httplib
import json
h = httplib.HTTPConnection('graph.facebook.com')
h.request('GET', '/19292868552')
response = h.getresponse()
content_length = int(response.getheader('Content-Length','0'))
# Read data until we've read Content-Length bytes or the socket is closed
data = ''
while len(data) < content_length or content_length == 0:
s = response.read(content_length - len(data))
if not s:
break
data += s
# We now have the full data -- decode it
j = json.loads(data)
print j

What you want(ed) is ijson, an incremental json parser.
It is available here: https://pypi.python.org/pypi/ijson/ . The usage should be simple as (copying from that page):
import ijson.backends.python as ijson
for item in ijson.items(file_obj):
# ...
(for those who prefer something self-contained - in the sense that it relies only on the standard library: I wrote yesterday a small wrapper around json - but just because I didn't know about ijson. It is probably much less efficient.)
EDIT: since I found out that in fact (a cythonized version of) my approach was much more efficient than ijson, I have packaged it as an independent library - see here also for some rough benchmarks: http://pietrobattiston.it/jsaone

Do you have control over the json? Try writing each object as a single line. Then do a readline call on the socket as described here.
infile = sock.makefile()
while True:
line = infile.readline()
if not line: break
# ...
result = json.loads(line)

Skimming the XBMC JSON RPC docs, I think you want an existing JSON-RPC library - you could take a look at:
http://www.freenet.org.nz/dojo/pyjson/
If that's not suitable for whatever reason, it looks to me like each request and response is contained in a JSON object (rather than a loose JSON primitive that might be a string, array, or number), so the envelope you're looking for is the '{ ... }' that defines a JSON object.
I would, therefore, try something like (pseudocode):
while not dead:
read from the socket and append it to a string buffer
set a depth counter to zero
walk each character in the string buffer:
if you encounter a '{':
increment depth
if you encounter a '}':
decrement depth
if depth is zero:
remove what you have read so far from the buffer
pass that to json.loads()

You may find JSON-RPC useful for this situation. It is a remote procedure call protocol that should allow you to call the methods exposed by the XBMC JSON-RPC. You can find the specification on Trac.

res = str(s.recv(4096), 'utf-8') # Getting a response as string
res_lines = res.splitlines() # Split the string to an array
last_line = res_lines[-1] # Normally, the last one is the json data
pair = json.loads(last_line)
https://github.com/A1vinSmith/arbitrary-python/blob/master/sockets/loopHost.py