We Keep Coding

Differences between serial communication in matlab vs. python 3.7? Sending int values over 128 via python serial to an arduino

I have some matlab functions which I would like to translate into python 3.7. The functions calculate values for joint angles of a little robot from trossenrobotics and send those values via serial port to the robot which is controlled by an arduino board. The board runs a programm from Trossenrobotics, which interpretes the data send via the serial port and reacts accordingly.
I already managed to translate all the functions and they give the same outputs as the matlab functions, but the serial communication just doesn't work.
In matlab fwrite(s, int_value) and fread(s) are used for the communication. The int_values represent a highbyte and a lowbyte of a joint position(0-1024) and are send seperately.
In python I used pyserial and the functions s.write(byte) and s.read().
I converted the int values into bytes with chr(int).encode().
Since I was struggling with my actual objective, I first wanted to abstract it and make it simpler. Now I am just trying to turn on an LED on the arduino for 2 seconds, when a special byte is received and send the same byte back to python.
I noticed that as long as the value I am sending is smaller that 128 it works just fine, but when it's greater it won't work.
I printed the output of chr(255).encode() which is b'\xc3\xbf', what to me looked like it could be the problem.
The I tried using chr(255).encode('charmap') and printed it which gives back b'\xff', what looks right to me, but it still doesn't work for numbers between 128 and 255.
I also noticed, that when I send the data over the terminal
with s.write(chr(115).encode()) It doesn't return a value, but when I use
s.write(chr(255).encode('charmap')) it returns a 1.
Here's my python pogramm:
python
import serial
from time import sleep
port = 'COM4'
baudrate = 38400
s = serial.Serial(port,baudrate)
sleep(3)
m = 115
s.write(chr(m).encode())
while s.in_waiting == 0:
print('##### waiting #####')
sleep(2)
if s.in_waiting > 0:
r = int.from_bytes(s.read(), byteorder = 'big')
print(r)
s.close()
And here's the arduino programm:
C#
void setup() {
pinMode(13, OUTPUT);
digitalWrite(13,LOW);
Serial.begin(38400);
}
void loop() {
if (Serial.available() > 0)
{
if (Serial.read() == 's')
{
digitalWrite(13, HIGH);
Serial.write('s');
delay(2000);
}
}
else
{
digitalWrite(13, LOW);
}
}
My questions would be:
Regarding my primary problem (sending multiple bytes via matlab, python):
1) Does anybody know if there are any fundamental differences between serial communication in matlab and in python which could cause my problems?
Regarding my abstracted problem (sending one bye via python):
2) How can I send values greater than 128 (up to 255) via the serial port?

There is no fundamental difference between Python and Matlab on this regard.
But in your Matlab code it seems (I'm assuming because what you say):
The int_values represent a highbyte and a lowbyte of a joint position(0-1024) and are send seperately.
that you're sending an int16
to be able to fit up to 1024.
I have no idea what you're trying to do with chr but I have the feeling what you need is replace these lines:
m = 115
s.write(chr(m).encode())
With (on Python 3.x):
m=115
s.write(m.to_bytes(2, byteorder="big"))
That would write: b'\x00s', a mix of hex and ASCII, but you should not worry about that, because that is exactly the same as b'\x00\x73'
And if you do, then you can do: b'\x00s'==b'\x00\x73' and you'll get True.

Thanks for your answer! And sorry for the late reply.
I already tried that right at the beginning but always got an exception. It took me a while to figure out why. It was because I was using numpy.uint8() for my integer values.
After I removed it I didn't get any exception but it didn't work either.
I used chr() because it didn't throw an exception with the numpy.uint8() and honestly because I did not know what else to do...
Today I finally found the solution.
Here is the link to where I found it:
arduino.stackexchange.com
Using s.write(struct.pack('>B', int_value) works and seems to be the equivalent to matlabs fwrite(s, int_value).
Sorry If my question didn't make a lot of sense to you, I am just glad I finally figured it out.

Working with TCP sockets in Python

I need to:
1) Create a socket to a specific IP and Port
2) Send a string of data via the connection
3) Await the response and check it is valid
I have seen a lot of conflicting advice at the moment and so decided to ask a new question to try and clear this up.
I'm trying to use the socket library in Python to achieve this however am facing a couple of issues. I've tried a few different methods, however I am facing an issue sending the string of data. My data is in XML format so getting this as a string is proving difficult for me. I have attempted converting it to binary however would prefer it in plain text format, just that it should be as a string. Any pointers as to whether the code would fulfil the 3 steps I wish to complete would be perfect!
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('109.73.xxx.xxx', 29006))
s.sendall('*XML SHOULD GO HERE*')
data = s.recv(1024)
s.close()
print 'Received', repr(data)
I haven't got a response back from the server at the moment because I suspect of the erroneous format of the string of data sent. This means I haven't been able to check whether the code prints the response data from the server.
The XML data I need to pass as a string is in the following format:
<Message xmlns:xsi="http://www.w3.org/CANNOTPROVIDETHISURL" xmlns:xsd="http://www.w3.org/CANNOTPROVIDETHISURL">
<ClientHeader>
<element1>RC1234</element1>
<element2>1234</element2>
<element3>12345678</element3>
<element4>123456789</element4>
<element5>-1</element5>
<element6>-1</element6>
<element7>A123</element7>
<element8>TMS</element8>
</ClientHeader>
<MsgType>TYPE</MsgType>
<MsgData></MsgData>
</Message>

Python - can't decipher serial port bits

I have a MaxiMet GMX600 weather station.
Here is the manual:
http://gillinstruments.com/data/manuals/maximet-manual-iss1.pdf
I have it connect to my serial port and it's spitting out data every 1 second. Which is great!
Unfortunately, I can't decipher the data...
I should be seeing strings like this:
$WIMWV ,069,R,004.06,N,A*00
$WIMWV,122,T,,N,A*14
$WIXDR,C,+023.2,C,TEMP,P,1.0281,B,PRESS,H,037,P,RH,Y,000.0,M,PRECIP*0A
Followed 1 second later by:
$WIMWV ,238,R,000.46,N,A*06
$WIMWV,303,T,,N,A*15
$WIXDR,C,+023.2,C,TEMP,P,1.0281,B,PRESS,H,037,P,RH,Y,000.0,M,PRECIP*0A
Followed 1 second later by:
$WIMWV ,130,R,000.21,N,A*0C
$WIMWV,205,T,,N,A*12
$WIXDR,C,+023.2,C,TEMP,P,1.0281,B,PRESS,H,038,P,RH,Y,000.0,M,PRECIP*05
Unfortunately, all I'm seeing is gobbledy-gook such as:
�����������������W���������������������������������������������������������������]���}���������������������u���+��+����; ���; �����; k��K; ���K;k��;� ����;���;� ���;���u������������
Followed one second later by:
�叙���������������������W�����������������������������������������������������������������]��哙���������������������W���������������������������������������������������������������]��坙
Followed one second later by:
�����]���y����������������������W�����������������������������������������������������������������]��叙���������������������W�����������������������������������������������������������������]�
Here's my Python code:
import time
import serial
import binascii
import struct
ser=serial.Serial(
port='/dev/ttyAMA0',
baudrate=19200,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS,
timeout=5
)
while 1:
y=0
str=""
while y<200:
#x=binascii.b2a_qp(ser.read()).strip().strip("=")
#x=binascii.hexlify(ser.read()).strip()
x=ser.read().strip()
#x=struct.pack('<h',int(ser.read()))
str=x+str
y=y+1
#ser.flush()
print str
You can see I've been trying to manipulate the bytes, but nothing is happening. I keep getting gobbldy-gook.
If I uncomment the line bin2ascii.hexlify(ser.read()).strip(), I get output like:
9fa78d919fa75dfb00ebe59599f9a79f9f9f9fa78fa3979fa9a795a38d998b9f958b9f9d57999fa59b9da5959d9f9ba79fa39f9f9fa79fa39f9f9f9f9fa79ba38d9f9fa9a79da39f9b9fa9a78d979fa799a39d9d9f9da78d9b9fa79d9fa39f9f9fa7998f9fa75dfb00ebe57d99f9a79f9f9f9fa78fa3979fa9a795a38f998b9f958b9f9d57999fa59b9da5959d9f9ba79fa39f9f9fa79fa39f9f9f9f9fa79ba38d9f9fa9a79da39f9b9fa9a78d979fa799a39d9d9f9da79b999fa79f9fa39f9f9fa7938f9fa75dfb
Followed one second later by:
9f9fa79d9fa39f9f9fa78d959fa75dfb00ebe57999f9a79f9f9f9fa78fa3979fa9a795a39d978b9f958b9f9d57999fa59b9da5959d9f9ba79fa39f9f9fa79fa39f9f9f9f9fa79ba38d9f9fa9a79da39f9b9fa9a78d979fa799a39d9d9f9da79d9599a79d9fa39f9f9fa795979fa75dfb00ebe59b99f9a79f9f9f9fa78fa3979fa9a795a39f978b9f958b9f9d57999fa59b9da5959d9f9ba79fa39f9f9fa79fa39f9f9f9f9fa79ba38d9f9fa9a79da39f9b9fa9a78d979fa799a39d9d9f9da7959b9fa79d9fa39f9f
Followed one second later by:
99a39d9d9f9da79b9d9fa79d9fa39f9f9fa793939fa75dfb00ebe57b99f9a79f9f9f9fa78fa3979fa9a795a399978b9f958b9f9d57999fa59b9da5959d9f9ba79fa39f9f9fa79fa39f9f9f9f9fa79ba38d9f9fa9a79da39f9b9fa9a78d979fa799a39d9d9f9da79f9d9fa79d9fa39f9f9fa797939fa75dfb00ebe59f99f9a79f9f9f9fa78fa3979fa9a795a39b978b9f958b9f9d57999fa59b9da5959d9f9ba79fa39f9f9fa79fa39f9f9f9f9fa79ba38d9f9fa9a79da39f9b9fa9a78d979fa799a39d9d9f9da795
ASCII only goes up to 0x7F, so I guess all these 0x9_ numbers are producing the funny characters?
I'm afraid I'm really stuck. I can't decipher it - my Python skills aren't good enough and I've been at this for hours... Does anyone have any Python suggestions as to how I can fix this output?

Python, serial - changing baudrate, strange behaviour

I am having troubles with changing baudrate while the port is running. All the communication is run at 100k baud, but I also need to send some data at 10k baud. I've read I should use setBaudrate method, so I tried this:
ser = serial.Serial(2, baudrate=BAUD, timeout=TIMEOUT)
def reset(string):
if string:
ser.flushInput() #erase input and output buffers
ser.flushOutput()
ser.setBaudrate(RESET_BAUD) #change baudrate to 10k
ser.write(string)
ser.setBaudrate(BAUD) #go back to 100k
The problem is, it doesn't work right. I don't know what is wrong here, but the string just isn't received properly. But here is interesting part - if I remove the last line (going back to 100k) and run this function from the shell, everything is fine. Then I can just run the last command directly in shell, not inside function.
My question is what exactly happens here and how to avoid it? All I need is a function to send a string with different baudrate and then return to the original baudrate...

You need to wait long enough for the string to be sent before resetting the BAUD rate - otherwise it changes while some of it is still in the serial port (hardware) buffer.
Add time.sleep(0.01*len(string)) before the last line.
BTW try not to use reserved words like string as variable names as it can cause problems.

My guess is that the baud rate is being changed before the data is actually sent. A good bet is to force the data to be sent before trying to change the baud rate.
According to the docs, this is done by calling Serial.flush() (not flushInput() or flushOutput(), as these just discard the buffer contents).

How do I dump the TCP client's buffer in order to accept more data?

I've got a simple TCP server and client. The client receives data:
received = sock.recv(1024)
It seems trivial, but I can't figure out how to recieve data larger than the buffer. I tried chunking my data and sending it multiple times from the server (worked for UDP), but it just told me that my pipe was broken.
Suggestions?

If you have no idea how much data is going to pour over the socket, and you simply want to read everything until the socket closes, then you need to put socket.recv() in a loop:
# Assumes a blocking socket.
while True:
data = sock.recv(4096)
if not data:
break
# Do something with `data` here.

Mike's answer is the one you're looking for, but that's not a situation you want to find yourself in. You should develop an over-the-wire protocol that uses a fixed-length field that describes how much data is going to be sent. It's a Type-Length-Value protocol, which you'll find again and again and again in network protocols. It future-proofs your protocol against unforeseen requirements and helps isolate network transmission problems from programmatic ones.
The sending side becomes something like:
socket.write(struct.pack("B", type) #send a one-byte msg type
socket.write(struct.pack("H", len(data)) #send a two-byte size field
socket.write(data)
And the receiving side something like:
type = socket.read(1) # get the type of msg
dataToRead = struct.unpack("H", socket.read(2))[0] # get the len of the msg
data = socket.read(dataToRead) # read the msg
if TYPE_FOO == type:
handleFoo(data)
elif TYPE_BAR == type:
handleBar(data)
else:
raise UnknownTypeException(type)
You end up with an over-the-wire message format that looks like:
struct {
unsigned char type;
unsigned short length;
void *data;
}

Keep in mind that:
Your operating system has it's own idea of what it's TCP/IP socket buffer size is.
TCP/IP packet maximum size (generally is 1500 bytes)
pydoc for socket suggests that 4096 is a good buffer size
With that said, it'd really be helpful to see the code around that one line. There are a few things that could play into this, if you're using select or just polling, is the socket non-blocking, etc.
It also matters how you're sending the data, if your remote end disconnects. More details.