We Keep Coding

Error status after sending 32768 bytes from serial port in Python

I am making uart communication using serial library. I can send and receive data, but when the number of bytes sent reaches 32768 or more, I get the error;
ValueError: byte must be in range(0,256)
function definition:
def Write_to_serial_port(value):
ser.write([value])
function usage:
#...
for i in data_buf[0:total_len]: # data_buf is a list
Write_to_serial_port(i)
#...
The error message that occurs when the number of bytes sent reaches 32768:
An alternative serial port write function I tested:
def Write_to_serial_port(value):
data = struct.pack('>B', value)
ser.write(data)
Again the error message that occurs when the number of bytes sent reaches 32768:
I also tried periodically flushing the input and output buffers, but it didn't help.
Any ideas on the solution?
EDIT1:
The purpose of the program is to send the bytes in the binary file. While doing this, I send 128 bytes (from the binary file) and 13 bytes of CRC, file size etc information over the serial port in each cycle. data_buff size is 255 bytes but I am using 141 bytes.
function usage(Extended):
# ...
# Some definitions and assignments
# ...
while(bytes_remaining):
if(bytes_remaining >= 128):
len_to_read = 128
else:
len_to_read = bytes_remaining
for x in range(len_to_read):
file_read_value = bin_file.read(1)
data_buf[9+x] = int(file_read_value[0])
data_buf[0] = mem_write_cmd_total_len-1
data_buf[1] = write_cmd
data_buf[2] = word_to_byte(base_mem_address,1,1)
data_buf[3] = word_to_byte(base_mem_address,2,1)
data_buf[4] = word_to_byte(base_mem_address,3,1)
data_buf[5] = word_to_byte(base_mem_address,4,1)
data_buf[6] = gl_bin_file_sector_needed
data_buf[7] = len_to_read
data_buf[8] = send_count
send_count = send_count + 1
crc32 = get_crc(data_buf,mem_write_cmd_total_len - 4)
data_buf[9 +len_to_read] = word_to_byte(crc32,1,1)
data_buf[10+len_to_read] = word_to_byte(crc32,2,1)
data_buf[11+len_to_read] = word_to_byte(crc32,3,1)
data_buf[12+len_to_read] = word_to_byte(crc32,4,1)
for i in data_buf[0:mem_write_cmd_total_len]:
Write_to_serial_port(i)
#...
Error Message
EDIT2: I also tried splitting the 40KB binary file into 128byte chunk files and sending it. But I got the same error on the 256th file. I guess 256*128 = 32768 can't be a coincidence.

How to store multiple values received from socket in a structure in python

I want to receive multiple messages from socket and I would like to store it in an structure like in c++. which will be used for further verification.
Here is my code:
def Receive(event,self):
message = socket.recv(0.1)
message is an object which has attributes like message.arbitration , message.timestamp
I want to store the message.timestamp and other attributes from upcoming recv socket API in structure some how like this:
struct canmessage[0].timestamp[0]=message.timestamp
For example if I receive 100 messages,
I want to store the 100 messages in a structure in python.
How to implement this logic in python.
I have tried list and dict but nothing worked out.
Can anyone please help resolve this issue.

The struct module can convert Python data to and from a byte stream suitable for transmission over a socket. I can be used when you have a fixed format structure being transmitted or received.
In the following example, the sent or received packet follows the format of an network-ordered (big-endian) double followed by a 10-byte UTF-8-encoded string padded with null bytes if necessary. The storage structure uses a datetime object and a Python Unicode str:
from datetime import datetime
import struct
class Message:
def __init__(self, ts, arb):
self.timestamp = ts
self.arbitration = arb
def __repr__(self):
return f'Message(timestamp={self.timestamp!r}, arbitration={self.arbitration!r})'
def serialize(self):
return struct.pack('!d10s', self.timestamp.timestamp(), self.arbitration.encode())
#classmethod
def deserialize(cls, raw):
ts, arb = struct.unpack('!d10s', raw)
return Message(datetime.fromtimestamp(ts), arb.decode().rstrip('\x00'))
x = Message(datetime.now(), 'Yes?')
data = x.serialize()
print(data)
msg = Message.deserialize(data)
print(msg)
print(msg.timestamp.ctime(), msg.arbitration)
Output:
b'A\xd8D\xbb\x15R\xdb4Yes?\x00\x00\x00\x00\x00\x00'
Message(timestamp=datetime.datetime(2021, 8, 10, 14, 15, 1, 294629), arbitration='Yes?')
Tue Aug 10 14:15:01 2021 Yes?
If you don't have a fixed format, JSON is a popular, readable format for transmitting data:
from datetime import datetime
import json
class Message:
def __init__(self, ts, arb):
self.timestamp = ts
self.arbitration = arb
def __repr__(self):
return f'Message(timestamp={self.timestamp!r}, arbitration={self.arbitration!r})'
def json(self):
data = {'timestamp': self.timestamp.timestamp(), 'arbitration': self.arbitration}
return json.dumps(data, ensure_ascii=False).encode()
#classmethod
def from_json(cls, raw):
data = json.loads(raw)
return Message(datetime.fromtimestamp(data['timestamp']), data['arbitration'])
x = Message(datetime.now(), 'Yes?')
data = x.json()
print(data)
msg = Message.from_json(data)
print(msg)
print(msg.timestamp.ctime(), msg.arbitration)
Output:
b'{"timestamp": 1628630908.680338, "arbitration": "Yes?"}'
Message(timestamp=datetime.datetime(2021, 8, 10, 14, 28, 28, 680338), arbitration='Yes?')
Tue Aug 10 14:28:28 2021 Yes?
You can also use the pickle module which is meant for serializing/deserializing arbitrary Python objects, but note you have to trust the data received as unpickling arbitrary objects can be unsafe.
from dataclasses import dataclass
import pickle
#dataclass
class Message:
timestamp: datetime = datetime.now()
arbitration: str = ''
data = pickle.dumps(Message(datetime.now(), 'Yes?'))
print(data)
msg = pickle.loads(data)
print(msg)
Output:
b'\x80\x04\x95j\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x07Message\x94\x93\x94)\x81\x94}\x94(\x8c\ttimestamp\x94\x8c\x08datetime\x94\x8c\x08datetime\x94\x93\x94C\n\x07\xe5\x08\n\x0e\x12#\x0e^\xa4\x94\x85\x94R\x94\x8c\x0barbitration\x94\x8c\x04Yes?\x94ub.'
Message(timestamp=datetime.datetime(2021, 8, 10, 14, 18, 35, 941732), arbitration='Yes?')
Transmitting and receiving the byte data packets is left as an exercise for the OP 🙂. The struct format is fixed-sized (read 18 bytes from the socket). json can use socket.makefile and readline() for reading a line at a time if transmitted in a single newline-terminated line. pickle can read a pickled object(s) directly from a socket wrapped in a makefile.

Maybe this thread here is helpful.
It is suggested to use the dataclass module to get struct like behaviour:
from dataclasses import dataclass
#dataclass
class Message:
timestamp: list = []
arbitration: list = []
message = socket.recv(0.1)
can_messages = [Message()]
can_messages[0].timestamp.append(message.timestamp)
I am not sure if this is exactly what you are looking for, but I hope it can help you.

Sending Real Time images captured using unity camera

server
private void SendImageByte()
{
image_bytes = cm.Capture();
print(image_bytes.Length);
if (connectedTcpClient == null)
{
return;
}
try
{
// Get a stream object for writing.
NetworkStream stream = connectedTcpClient.GetStream();
if (stream.CanWrite)
{
// string serverMessage = "This is a message from your server.";
// Convert string message to byte array.
byte[] serverMessageAsByteArray = Encoding.ASCII.GetBytes(image_bytes.ToString());
// Write byte array to socketConnection stream.
stream.Write(serverMessageAsByteArray, 0, serverMessageAsByteArray.Length);
Debug.Log("Server sent his message - should be received by client");
}
}
catch (SocketException socketException)
{
Debug.Log("Socket exception: " + socketException);
}
}
client
import socket
host = "127.0.0.1"
port = 1755
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))
def receive_image():
data = sock.recv(999999).decode('utf-8')
print(len(data))
while True:
receive_image()
here script capture images from unity camera
public byte[] Capture()
{
if(renderTexture == null)
{
// creates off-screen render texture that can rendered into
rect = new Rect(0, 0, captureWidth, captureHeight);
renderTexture = new RenderTexture(captureWidth, captureHeight, 24);
screenShot = new Texture2D(captureWidth, captureHeight, TextureFormat.RGB24, false);
}
// _camera = GetComponent<Camera>();
_camera.targetTexture = renderTexture;
_camera.Render();
// reset active camera texture and render texture
_camera.targetTexture = null;
RenderTexture.active = null;
// read pixels will read from the currently active render texture so make our offscreen
// render texture active and then read the pixels
RenderTexture.active = renderTexture;
screenShot.ReadPixels(rect, 0, 0);
screenShot.Apply();
byte[] imageBytes = screenShot.EncodeToPNG();
//Object.Destroy(screenShot);
//File.WriteAllBytes(Application.dataPath + "/../"+ imagePath + "/img{counter}.png", bytes);
//counter = counter + 1;
return imageBytes;
}
Am trying to send real-time images on Unity3D from C# to python using socket communication to be processed and return back values to unity, but the problem even the bytes length received on the client is not the same as the server. I send about 400K bytes but I receive only 13
C# is the server and python is the client
or am doing it wrong but the main goal I want to create simulator work as udacity self-driving

Are you sure that image_bytes.ToString() returns what you expect and not maybe just something like "System.Byte[]" => 13 chars => 13 bytes.
In general why would you convert an already byte[] into a string just in order to convert it back into byte[] to send? I'm pretty sure you do not want to transmit binary image data using UTF-8 ... one option might be Base64 string, but still that would be quite inefficient.
Just send the raw bytes like e.g.
stream.Write(image_bytes, 0, image_bytes.Length);
And then receive until you receive that length.
A typical solution is to prepend the length of the message to send and on the receiver side actually wait until you received exactly that amount of bytes like e.g.
var lengthBytes = BitConverter.GetBytes(image_bytes.Length);
stream.Write(lengthBytes, 0, lengthBytes.Length);
stream.Write(image_bytes, 0, image_bytes.Length);
Now you know that on the receiver side you first have to receive exactly 4 bytes (== one int) which will tell you exactly how many bytes to receive for the actual payload.
Now I'm no python expert but after googling around a bit I think something like
def receive_image()
lengthBytes = sock.recv(4)
length = struct.unpack("!i", lengthBytes)[0]
data = sock.recv(length)
Note: After reading John Gordon's comment on the question I guess this still doesn't fully solve the waiting until according buffers are actually filled - as said no python expert - but I hope it gives you a idea where to go ;)

Python UDP SocketServer can't read whole packet

At sender side I have the following code using processing language (portion code):
udp = new UDP( this, 6002 ); // create a new datagram connection on port 6000
//udp.log( true ); // <-- printout the connection activity
udp.listen( true ); // and wait for incoming message
escribeUDPLog3(1,TRANSMIT,getTime()); //call function
int[] getTime(){
int year = year();
int month = month()
int day = day();
int hour = hour();
int minute = minute();
int second = second();
int[] time_constructed = {year, month,day,hour,minute,second};
return time_constructed;
}
void escribeUDPLog3(int pkg_type, int state, int[] time){
short year = (short)(time[0]); //>> 8;
byte year_msb = byte(year >> 8);
byte year_lsb = byte(year & 0x00FF);
byte month = byte(time[1]);
byte day = byte(time[2]);
byte hour = byte(time[3]);
byte minute = byte(time[4]);
byte second = byte(time[5]);
byte[] payload = {byte(pkg_type), byte(state), year_msb, year_lsb, month, day, hour, minute,second};
try {
if (UDP5_flag) {udp.send(payload, UDP5_IP, UDP5_PORT);}
}
catch (Exception e) {
e.printStackTrace();
}
}
At receiver side I'm using SocketServer python structure to set up a server listening for udp datagrams, as following.
from datetime import datetime
import csv
import SocketServer
def nodeStateCheckout(nodeid, state, nodeState):
if (state == ord(nodeState)):
return "OK"
else:
return "FAIL"
def timeConstructor(time):
year = str(ord(time[0]) << 8 | ord(time[1]))
month = str(ord(time[2]))
day = str(ord(time[3]))
hour = str(ord(time[4]))
minute = str(ord(time[5]))
second = str(ord(time[6]))
time_formatted = year + "-" + month + "-" + day \
+ " " + hour + ":" + minute + ":" + second
return time_formatted
class MyUDPHandler(SocketServer.BaseRequestHandler):
"""
This class works similar to the TCP handler class, except that
self.request consists of a pair of data and client socket, and since
there is no connection the client address must be given explicitly
when sending data back via sendto().
"""
def handle(self):
try:
data = self.request[0].strip()
socket = self.request[1]
#print "{} wrote:".format(self.client_address[0])
pkg_type = ord(data[0])
if pkg_type == 1: # log 3
state = ord(data[1])
csvfile = open("log3.csv", "a+")
csvwriter = csv.writer(csvfile, delimiter=',')
time_reconstructed = timeConstructor(data[2:9])
if state == 3:
csvwriter.writerow(["STOP",time_reconstructed])
elif state == 2:
csvwriter.writerow(["START",time_reconstructed])
else:
print "unknown state"
csvfile.close()
else:
print "packet not known"
except IndexError:
print "Bad parsed byte"
if __name__ == "__main__":
HOST, PORT = "localhost", 8892
server = SocketServer.UDPServer((HOST, PORT), MyUDPHandler)
server.serve_forever()
Edited:
I have problem specifically when using timeConstructor(data[2:9]), because I'm accessing out of index data, sometimes (with the help of print) I can't received second byte from data, and one time it get me out of index because I didn't received minute and second. Most of the time the code works well, but this type of error get me curious.
Old:
The problem is when reading the payload, sometimes its seems that some bytes doesn't arrive, even when I captured the whole payload using Wireshark (but Wireshark didn't tell me if this is the sent packet or received packet because I'm using loopback interfaces, maybe duplicated info?). If the datagram has 16 bytes payload long, sometimes I received 15 because when parsing from data I get out of index error.
I think that there are some buffer problems. Isn't it? How to configured it properly? I know that I can get packet loss because of connectionless protocol but I dont think that bytes get lost. It is supposed that "data" has all payload data from one udp datagram.

I believe your problem is that socket.sendto() does not always send all the bytes that you give it. It returns the number of bytes sent and you may have to call it again. You might be better off with opening the socket yourself and calling socket.sendall()

Raw socket python packet sniffer

I have created a simple RAW socket based packet sniffer. But when I run it, it rarely captures up a packet. First I created this to capture packets in 1 second time intervals, but seeing no packets are captured I commented that line. I was connected to internet and a lot of http traffic are going here and there, but I could not capture a one. Is there a problem in this in the code where I created the socket? Please someone give me a solution. I am fairly new to python programming and could not understand how to solve this.
import socket, binascii, struct
import time
sock = socket.socket(socket.PF_PACKET, socket.SOCK_RAW, socket.htons(0x800))
print "Waiting.."
pkt = sock.recv(2048)
print "received"
def processEth(data):
#some code to process source mac and dest. mac
return [smac, dmac]
def processIP(data):
sip = str(binascii.hexlify(data[1]))
dip = str(binascii.hexlify(data[2]))
return [sip, dip]
def processTCP(data):
sport = str(data[0])
dport = str(data[1])
return [sport, dport]
while len(pkt) > 0 :
if(len(pkt)) > 54:
pkt = sock.recv(2048)
ethHeader = pkt[0][0:14]
ipHeader = pkt[0][14:34]
tcpHeader = pkt[0][34:54]
ethH = struct.unpack("!6s6s2s",ethHeader)
ethdata = processEth(ethH)
ipH = struct.unpack("!12s4s4s",ipHeader)
ipdata = processIP(ipH)
tcpH = struct.unpack("!HH16", tcpHeader)
tcpdata = processTCP(tcpH)
print "S.mac "+ethdata[0]+" D.mac "+ethdata[1]+" from: "+ipdata[0]+":"+tcpdata[0]+" to: "+ipdata[1]+":"+tcpdata[1]
#time.sleep(1);
else:
continue

If you showed all the code, you are running into an endless loop.
Whenever a paket is coming in which has not a length greater then 54 bytes, you end up reading the same packet all the time.
Additionally, socket.recv() returns a string/byte sequence; your approach of accessing the data is wrong. pkt[0] returns a string with length 1; pkt[0][x:y] will not return something useful.
I am not familiar with using sockets, but with some changes I got output that might look similar to what you intended (there is something missing in processEth() I think...).
[...]
while len(pkt) > 0:
print "Waiting.."
pkt = sock.recv(2048)
print "received"
if(len(pkt)) > 54:
ethHeader = pkt[0:14]
ipHeader = pkt[14:34]
tcpHeader = pkt[34:38]
ethH = struct.unpack("!6s6s2s",ethHeader)
ethdata = processEth(ethH)
ipH = struct.unpack("!12s4s4s",ipHeader)
ipdata = processIP(ipH)
tcpH = struct.unpack("!HH16", tcpHeader)
tcpdata = processTCP(tcpH)
print "S.mac "+ethdata[0]+" D.mac "+ethdata[1]+" from: "+ipdata[0]+":"+tcpdata[0]+" to: "+ipdata[1]+":"+tcpdata[1]
#time.sleep(1);
else:
continue