We Keep Coding

Using serializing an object in python for use with an XBee

For a project I'm working on, I'm supposed to use XBee radio modules, which isn't super important, except that I have to read and write to their serial port in order to use them. I'm currently working with Python and ROS, so I'm attempting to send TransformStamped messages over the XBees.
My question is, unless I'm misunderstanding how Serial.read() and Serial.write() work, how can I tell how many bytes to read? I was planning on using Pickle to serialize the data into a string, and then sending that over the serial ports. Is there a better way that I've overlooked? Is there some sort of loop that would work to read data until the end of the pickled string is read?

The short answer is, serial.read() cannot tell you how many bytes to read. Either you have some prior knowledge as to how long the message is, or the data you send has some means of denoting the boundaries between messages.
Hint; knowing how long a message is is not enough, you also need to know whereabouts in the received byte stream a message has actually started. You don't know for sure that the bytes received are exactly aligned with the sent bytes: you may not have started the receiver before the transmitter, so they can be out of step.
With any serialisation one has to ask, is it self delimiting, or not? Google Protocol buffers are not. I don't think Pickle is either. ASN.1 BER is, at least to some extent. So is XML.
The point is that XBee modules are (assuming you're using the ones from Digi) just unreliable byte transports, so whatever you put through them has to be delimited in some way so that the receiving end knows when it has a complete message. Thus if you pickle or Google Protocol Buf your message, you need some other way of framing the serialised data so that the receiving end knows it has a complete message (i.e. it's seen the beginning and end). This can be as simple as some byte pattern (e.g. 0xffaaccee00112233) used to denote the end of one message and the beginning of the next, chosen so as to be unlikely to occur in the sent messages themselves. Your code at the receiving end would read and discard data until is saw that pattern, would then read subsequent data into a buffer until it saw that pattern again, and only then would it attempt to de-pickle / de-GPB the data back into an object.
With ASN.1 BER, the data stream itself incorporates effectively the same thing, saving you the effort. It uses tags, values and length fields to tell its decoders about the incoming data, and if the incoming data makes no sense to the decoder in comparison to the original schema, incorrectly framed data is easily ignored.
This kind of problem also exists on tcp sockets, though at least with those delivery is more or less guaranteed (the first bytes you receive are the first bytes sent). A Digimesh connection does not quite reach the same level of guaranteed delivery as a tcp socket, so something else is needed (like a framing byte pattern) for the receiving application to know that it is synchronised with the sender.

python sending TCP/IP data instantly

I am currently sending data over TCP/IP in myserver using something like this
for str in lst:
data = str + "\n"
self._conn.sendall(data)
Now suppose my list has the following two string in it
1-This is statement 1 in list
2-This is statement 2 in list
My client is receiving half of line 2 like this.
This is statement 1 in list
This is
I would like to send line1 and then line 2 in the list individually. I understand that TCP/IP works this way in which it will send the entire data that is available to send. I think I could put a delay in after calling self._conn.sendall(data) but i wanted to know what other options I have. I cannot make changes to the receiver of the data and I can only make changes to the sender. So far my only option is adding a delay after each send.

TCP works with streams of data, not individual packets. It's like reading data from a file. The sender puts data in its send buffer, and TCP can decide for itself when to send it. The timing of the arrival at the receiving application depends on when the data was sent and on (often unpredictable) network conditions.
TCP deliveries can be made more predicable if you use the TCP_NODELAY flag in your socket (something like socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1). This would cause TCP to send out data as soon as it arrives in its buffer. But still, there would be no guarantees as to arrival times. This is why any time based solution would break, at least in some cases.
The solution is to divide the data stream into chunks yourself. There are several ways of doing that. Here are a few:
Use fixed length messages - if all messages have a fixed length, the receiver just has to recv() the right number of bytes, process the message, then wait for the same number of bytes.
Send the length of the message before each message. If you want to send the string "blah", encode it as "0004blah" or something similar. The receiver will (always) read the first four bytes (which are 0004) to figure out the number of remaining bytes to read. It will then read the required number of bytes, process the message, and then wait for the next one. It's a robust solution that's also easy to implement.
Use a delimiter. Lines in text files are divided by newline characters (\n). Similarly, you can add a special delimiter byte (or bytes) between messages. For example, you can define that messages always end with a dollar sign ($). Then all the receiver has to do is read from the socket byte by byte until it receives a dollar sign. Of course if you take this approach, you have to make sure that the body of the messages doesn't contain the delimiter character.

TCP is based on a stream, not individual messages. So you need to parse the end point of each message yourself. One idea in your case would be to read until you get a newline, then process the line. Note that you might read this:
This is statement 1 in list
This is
Then you need to check to see if you got a newline, process the line, then leave your buffer ready to receive the rest, like this:
This is

TCP has a local buffer that is not sent until it's full. You can force flushing of the local buffer so it's sent after every message, but when the other party receives these packages they get stored in another local buffer and your separation may disappear. TCP is a stream, use it as a stream. You have to use separator characters and when the packets are received you have to separate the messages manually. If you want more control, use UDP packets.

pySerial XON/OFF fails on Linux but works on Windows

I have a small loop that is supposed to read some text data of multiple lines and unspecified length (sometimes rather large) from the serial port, and write it to a file:
while True:
data = port.readline()
file.write(data)
if not data:
break
I have the port setup correctly and the whole program works beautifully on a windows machine. When I run it on a Linux box it still receives the data but the software flow control fails to work - instead the control codes for the XON and XOFF (\S and \Q) are actually being written to the file. Causing overrun errors and destroying my data.
I've read and tried everything I can think of - it seems most examples of this are only expecting a few bytes and aren't using any flow control. I have also tried the miniterm that is included with pySerial and it gives the same results.
Any ideas?
Added lines for file and port:
file = open('temp.txt', 'ab') #Open the temporary file to append the incoming data to.
port = serial.Serial('/dev/ttyAMA0', baudrate=9600, bytesize=8, parity='N', stopbits=1, timeout=1, xonxoff=1, rtscts=0, dsrdtr=0)
There isn't much more to it than that except that I write a string to the other device to start sending, which works since I'm seeing the response back with the data. I've also tried printing to the console rather than writing the file. Same thing - overruns and printed control codes.
Update: I have also tried reading a single byte at a time instead of the whole line and have tried other baud rates - all with no change.
Update#2: Have been doing some reading and testing. Tested with minicom. Same results. Manually verified that port was set properly with 'stty -a' (although I don't trust this as it may save and restore the settings according to something I read). It's possibly a problem in termios?

Try removing unprintables characters from your received data.
data = data.rstrip()

perhaps you can try using this serial capture code. it has a Pyserial approach also but it is simplified. There is a split formula for the newline that may help with the flow control if you change it to meet your needs.
https://github.com/DaDaDadeo/Batch-Capture/blob/master/serial_to_tcp.py

How to split a file into smaller pieces before sending

How do I get the following code to break up large files into smaller parts and send those parts, instead of sending the whole file? It fails to send large files (Tested with an ubuntu iso around 600mb)
...some code
# file transfer
with open(sendFile, "rb") as f:
while 1:
fileData = f.read()
if fileData == "": break
# send file
s.sendall(EncodeAES(cipher, fileData))
f.close()
...more code
I tried with f.read(1024), but that didn't work.
Finally, when splitting up the files, I would need to be able to put the parts together again.
I'm also encrypting the files using PyCrypto, if that has any impact on what I'm trying to do. Guess it would be smartest to encrypt the seperate parts, instead of encrypting the whole file and then splitting that into parts.
Hope the above code is enough. If not, I'll update with more code.

I may be wrong, but I'm betting that your actual problem is not what you think it is, and it's the same reason your attempt to fix it by reading 1K at a time didn't help. Apologies if I'm wrong, and you already know this basic stuff.
You're trying to send your cipher text like this:
s.sendall(EncodeAES(cipher, fileData))
There is certainly no length information, no delimiter, etc. within this code. And you can't possibly be sending length data outside this function, because you don't know how long the ciphertext will be before getting to this code.
So, I'm guessing the other side is doing something like this:
data = s.recv(10*1024*1024)
with open(recvFile, "wb") as f:
f.write(DecodeAES(cipher, data))
Since the receiver has no way of knowing where the encrypted file ends and the next encrypted file (or other message) begins, all it can do is try to receive "everything" and then decrypt it. But that could be half the file, or the file plus 6-1/2 other messages, or the leftover part of some previous message plus half the file, etc. TCP sockets are just streams of bytes, not sequences of separate messages. If you want to send messages, you have to build a protocol on top of TCP.
I'm guessing the reason you think it only fails with large files is that you're testing on localhost, or on a simple LAN. In that case, for smallish sends, there's a 99% chance that you will recv exactly as much as you sent. But once you get too big for one of the buffers along the way, it goes from working 99% of the time to 0% of the time, so you assume the problem is that you just can't send big files.
And the reason you think that breaking it into chunks of 1024 bytes gives you gibberish is that it means you're doing a whole bunch of messages in quick succession, making it much less likely that the send and recv calls will match up one-to-one. (Or this one may be even simpler—e.g., you didn't match the changes on the two sides, so you're not decrypting the same way you're encrypting.)
Whenever you're trying to send any kind of messages (files, commands, whatever) over the network, you need a message-based protocol. But TCP/IP is a byte-stream-based protocol. So, how do you handle that? You build a message protocol on top of the stream protocol.
The easiest way to do that is to take a protocol that's already been designed for your purpose, and that already has Python libraries for the client and either Python libraries or a stock daemon that you can just use as-is for the server. Some obvious examples for sending a file are FTP, TFTP, SCP, or HTTP. Or you can use a general-purpose protocol like netstring, JSON-RPC, or HTTP.
If you want to learn to design and implement protocols yourself, there are two basic approaches.
First, you can start with Twisted, monocle, Tulip, or some other framework that's designed to do all the tedious and hard-to-get-right stuff so you only have to write the part you care about: turning bytes into messages and messages into bytes.
Or you can go bottom-up, and build your protocol handler out of basic socket calls (or asyncore or something else similarly low-level). Here's a simple example:
def send_message(sock, msg):
length = len(msg)
if length >= (1 << 32):
raise ValueError('Sorry, {} is too big to fit in a 4GB message'.format(length))
sock.sendall(struct.pack('!I', length))
sock.sendall(msg)
def recv_bytes(sock, length):
buf = ''
while len(buf) < length:
received = sock.recv(4-len(buf))
if not received:
if not buf:
return buf
raise RuntimeError('Socket seems to have closed in mid-message')
buf += received
return buf
def recv_message(sock):
length_buf = recv_bytes(sock, 4)
length = struct.unpack('!I', buf)
msg_buf = recv_bytes(sock, length)
return msg_buf
Of course in real life, you don't want to do tiny little 4-byte reads, which means you need to save up a buffer across multiple calls to recv_bytes. More importantly, you usually want to turn the flow of control around, with a Protocol or Decoder object or callback or coroutine. You feed it with bytes, and it feeds something else with messages. (And likewise for the sending side, but that's always simpler.) By abstracting the protocol away from the socket, you can replace it with a completely different transport—a test driver (almost essential for debugging protocol handlers), a tunneling protocol, a socket tied to a select-style reactor (to handle multiple connections at the same time), etc.

Problems with sending commands over pySerial

I'm trying to talk to a home made card over a serial port, and is therefor using pySerial. In Hyperterminal, everything works fine. I can write:
$ audio on
and the audio is enabled, but if I use
ser = serial.Serial("COM1", 38400)
ser.write("audio on\r\n")
nothing happens. I can read incoming data however, so it's not something wrong with the communication. I doesn't help if I change \r\n to just \n or \r either.
EDIT: Sometime I actually get the feedback: No such command when sending the exact same command as works from HyperTerminal. The setup is also the exact same as in HyperTerminal.
Solved:
To make it work, I had to send one and one character, and ending the transmission with \r.

Get an oscilloscope (you've got one, right?) and watch the serial line. Send one character per second through it and see what comes up on the scope (set it to trigger on the start bit). Serial port bits are in the order: start, LSB .. MSB, parity, stop.
See if there are characters that don't get through, or if there's a pattern. Another possibility is that everything is actually making it out the port, and your board is dropping characters.

Triple check that the baud rate of the device is indeed 38400
Triple check parity, stop bits, etc
Be aware of signal degradation for serial transmissions over long distances (probably not your issue)
If all the above checkout try putting the string into a byte array and sending that through the write command. Just a guess.

Sending characters via Hyperterminal deliver characters at the speed you type them. Sending characters through pyserial they are delivered as a continuous stream. The receiver (especially at high baud rates) could drop them, depending on the nature of the receiver.
Also, when you send commands to an interpreter, you only need the \r terminator (without the \n), (this is all that is sent by hyperterm, normally). HOWEVER, if you are just displaying the values, you may need the \n to generate the new line.