We Keep Coding

USB Serial data sending gibberish

i am having an issue i cannot seem to resolve. i am using python on the raspberry pi to read from a usb connection on the pi (that is being converted from serial).
i am able to connect to the usb port and start receiving data with the code
ser = serial.Serial("myUsbPortID", 9600)
bytes = ser.inWaiting()
print ser.read(bytes)
i know that the baudrate is 9600 (hardware manufacturers docs) but for some reason when i try to read the stream of data i get a lot of gibberish in the form of different languages and characters. After i kill the program my screen still replaces my characters with the gibberish data as i type.
i'm sure this isn't the stream of data the hardware is sending. something somewhere is converting things but i have no idea what it may be.
when i boot up the device and it is initializing then i get readable information. but when the device start operating i only get this weird characters
is there a way to convert these characters to the data that it is actually coming in as?
example pic:
output screen

so i was able to solve the issue of the gibberish with this line on python code (for anyone else having this issue).
data = ":".join("{:02x}".format(ord(c)) for c in bytes)
where bytes is the raw data, i'm making it ':' delimited.
seems the connection and baud rate was ok. luckily the data was able to convert to hex ok.

Can't extract individual fields from scapy packet

I've been experimenting with scapy and Python 3 and I want to use the ARP protocol to find mac addresses of computers on the network. This is my code:
>>> packet = srp(Ether(dst="ff:ff:ff:ff:ff:ff")/ARP(pdst=IP_OF_HOST))
Then to extract the data from this packet I used the following line:
>>> packet[0][Ether].src
However for some reason this produces the error:
AttributeError: 'list' object has no attribute 'src'
Every tutorial I read used the method I used for extracting field data, why wouldn't it work for me?

It has to do with the difference between the functions srp(), and srp1() (Those being for network layer 2, for layer 3 you would use sr() and sr1() respectively).
srp() sends the packet, and saves all packets, whether they are answered or not. To get say the source MAC address I'd do this:
answered_packets, unanswered_packets = packet
for packet_i_sent, packet_i_received in answered_packets:
print(packet_i_received.src)
srp1() sends a packet, and waits for one answer, and saves only the reply packet. This means the format is different, since you don't have to deal with the unanswered packets, and my previous method would work:
print(packet.src) #The [Ether] part doesn't matter,
#that's just for looking fancier when you print(packet)
So basically I used the command srp() and tried to decode the reply like I was using srp1()

Python TCP socket for a lot of data

We (as project group) are currently stuck on the issue of how to handle live data to our server.
We are getting updates on data every second, and we would like to insert this into our database (security is currently not an issue, because it is a school project). The problem is here we tried python SockerServer and AsyncIO to create a TCP server to which the data can be sent.
We got this working with different libraries etc. But we are stuck on the fact that if we keep an open connection with the client (in this case hardware which sends data every second) we can't split the different JSON or XML messages. They are all added up together.
We know why because TCP only provides order.
Any thoughts on how to handle this? So that every message sent will get split from the others.
Recreating the socket won't be the right option if I recall correctly.

What you will have to do is ensure that there is a clear delimiter for each message. For example, the first 6 characters of every message could be the length of the message - whatever reads from the socket decodes the length then reads that number of bytes, and sends the data to whatever needs it. Another way would be if there is a character/byte which never appears in the content, send it immediately before a message - for example control-A (binary value 1) could be the leadin character, and send control-B (binary value 2) as the leadout. Again the server looks for these framing a message.

If you can't change the client side (the thing sending the data), then you are going to have to parse the input. You can't just add a delimiter to something that you don't control.
An alternative is to use a header that encodes the size of the message that will be sent. Lets say you use a header of 4 bytes, The client first send the server a header with the size of the message to come. The client then sends the message (up to 4 gigs or there about). The server knows that it must first read 4 bytes (a header). It calculates the size n that the header contained then reads n bytes from the socket buffer. You are guaranteed to have read only your message. Using special delimiters is dangerous as you MUST know all possible values that a client can send.
It really depends on the type of data you are receiving. What type of connection, latency... If you have a pause of 1 second between packets and your connection is consistent, you could probably get away with first reading the entire buffer once to clear it, then as soon as there is data available - read it and clear the buffer it. not a great approach, but it might work for what you need - and no parsing involved.

Encoding for ZMQ

I am currently doing project which requires communication from a PC to the device, so far I've decided on socket comms. and have written some code. I am also using ZMQ for ipc on the device itself.
My script works by sending data as text across. I was trying to encode my data to utf-8 so that it can be easily read on the device and displays in a frame and performs the tasks as needed. However, I can't seem to get the encoding working right, I've tried searching for examples or tutorials online but can't seem to find any.
I've tried using socket.send (msg.encode("UTF-8")) to encode my data, and message = socket.recv() to recv & print the data on the server. This works but the server would print out the exact text data which is not what I wanted. I'm unsure whether this is the correct way, and hope that someone could point me in the correct direction for encoding & printing the encoded data without decoding back to text.

You are receiving the text as encoded UTF8 data. It is all working correctly.
However, if you are printing the data on the receiving end directly to a terminal that happens to be configured to display UTF-8, you won't see any difference.
Print the representation instead:
print repr(message)
to see the string literal representation including any non-printable, non-ASCII bytes displayed as escape strings.

erlang interface to python

I've been expanding on python version of Armstrong's classical example of interfaces. Everything works fine while I communicate bytes. But, I'd like to communicate long integers and floats. Mabye even (oh, no) strings. Here is my code:
http://pastebin.com/epxgDmvu
http://pastebin.com/rD7CWRkz
First of all, all that I know how to send are bytes. Can erlang send anything else to it's inteface? Or do I have to convert float to list of bytes, send it to python and then assemble it back to float in python?
And, the ohter way around: if I pack with 'f' format, erlang recognises that as a list of bytes. Is there a way to persuade erlang to take those bytes as one single float? If not, where can I find out how to convert that erlang list to erlang float?
In case erlang can communicate only bytes, how do you propose that I send a list of integers? If I convert integers to lists of bytes, then I can't send them in one message since reciever won't know where one integer ends and other begins, right? Should I then send integers one by one?
Yes, I am looking into ErlPort and py-interface and some other, but I'd like to start from fundamentals.
Regards,
dijxtra

At a low (from programming point of view) level you always send/receive only a bunch of bytes through different external interfaces (pipes, network, files...) independently from a programming language you're using. For example when you work with pipes (which you got calling open_port/2) you send/receive data as a stream of bytes.
To send any useful information through a stream you need to divide it to chunks which usually named messages. Message format define how messages can be extracted from a stream of bytes. For example using Type-Length-Value (TLV) encoding you can send values of different length marked with a type.
In ErlPort it all works as following:
When you call open_port/2 you must add option {packet, N} so all binary messages sent through this port are preceded by their length, sent in N bytes (Length-Value encoding).
Before to send a term to the port you must use term_to_binary/1 function which encode Erlang term to a bunch of bytes using Erlang external term format.
On Python side when you instantiate erlproto.Port class you must pass the same packet value as you pass to open_port/2 (packet=1 by default).
Port.read method first read the length of the message from the stream and then the message body.
The message body decoded by the erlterms.decode function.
When you send a reply using Port.write method it's first encoded to Erlang external term format by erlterms.encode function.
Then the encoded chunk prepended by its length sent back to Erlang.
On Erlang side binary_to_term/1 is used to decode the data.