I need an application that will do the following:
read UDP message, get data from it - the data is textual, comma separated.
build a new message based on this values
send a new UDP message to other destination.
For the first and the last I'm fine. for the second I have a challenge:
Lets say that the data I've received is Year = 2020. The message on the Wireshark (step 3), I shall see it as 2 bytes valued 07 e4
How can I do it? I've tried couple of ways, none of them provided me with the desired format.
Sample of the code:
data1 = '\xab\xcd\xef\xgh'
...
data, addr = sock.recvfrom (200)
elements = data.decode().split(",")
Date=elements[15].split("/")
Year = int(Date(2))
year = <do something with Year to convert to right format>
newMsg = data1 + year
...
newSock.sendto(newMsg.encode('charmap'), (IP, int(port)))
Python version 3.5
there seem to be a couple of issues here. first, if your data1 is really supposed to be raw bytes then you're better off declaring it as such by prefixing it with a b, something like:
data1 = b'\xab\xcd\xef\xff'
if it comes from somewhere else, then encode it to bytes appropriately.
for answering your main question, the struct module has useful tools for encoding numbers as bytes in the way you want. ctypes can also be useful, but struct tends to be easier to use for these sorts of things:
data, addr = sock.recvfrom (200)
elements = data.decode().split(",")
date = elements[15].split("/")
year = int(date(2))
newMsg = data1 + struct.pack('!h', year)
and then you can send it with:
newSock.sendto(newMsg, (IP, int(port)))
Related
I have the following data from a datagram message received by UDP socket:
b'/13/raw\x00,ffffffffffffff\x00=\xca\x00\x00=[\x00\x00\xbf\x82H\x00B\x14\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00B;\x81\n\xbf\xbat4C-\xfd\xb0B\x87\xac\xd9\x00\x00\x00\x00E\xcd\xc0\x00I\xcc\xc1\xe0'
This is my code:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((IP, p))
print("Listening for UDP packets on {0}:{1} ...".format(IP, p))
while 1:
rawdata, _ = sock.recvfrom(1024)
print(rawdata)
How can I convert rawdata to floats?
This is the data type that I am expecting:
Both are different samples!
Thank you
each float is 4 bytes of data (typically... not always... again you need to consult the docs for whatever p[ayload you need)
but assuming that /13/raw is easy enough for you and the rest is what you need to decode
if we assume that \x00 is some sort of delimiter we are left with the bytestring as follows
msgBytes = b',ffffffffffffff\x00=\xca\x00\x00=[\x00\x00\xbf\x82H\x00B\x14\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00B;\x81\n\xbf\xbat4C-\xfd\xb0B\x87\xac\xd9\x00\x00\x00\x00E\xcd\xc0\x00I\xcc\xc1\xe0'
we can see there are 72 characters which means 18 floats (18*4 === 72)
however your expected output only seems to have 14 floats so that means 16 bytes are some sort of metadata (maybe a checksum, or some payload information (how many floats to expect))
we can just struct unpack it as floats now ... but i dont get your "expected value"
so some of those may be doubles or something other than 4 byte floats
floats = struct.unpack('18f',msgBytes)
now you have a list of 18 floats ... but again not matching your expected output, it may be related to endianness or some other method that you need to do additional processing on (eg maybe it should be ints and they become floats by dividing by 100 or something?)
[edit] investigating further I suspect ffffffffff\x00 is some sort of header or metadata
so if we start with
myBytes = b'=\xca\x00\x00=[\x00\x00\xbf\x82H\x00B\x14\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00B;\x81\n\xbf\xbat4C-\xfd\xb0B\x87\xac\xd9\x00\x00\x00\x00E\xcd\xc0\x00I\xcc\xc1\xe0'
and unpack it with struct.unpack('14f',myBytes) we get 14 floats as expected ... some are zero like in your example ... but the rest dont match
(7.254942539348875e-41, 3.273012823123475e-41, 6.659058584274783e-39, 1.1762210642058864e-38, 0.0, 0.0, 0.0, 1.244453845292514e-32, 2.2792208653754642e-07, -1.84210369180704e-09, -6070301691478016.0, 0.0, 1.7706052095336117e-38, -1.1171693447134314e+20)
so I think we still dont have enough info to actually answer this question, need to know more about the datasource and other stuff
I have an industrial sensor which provides me information via telnet over port 10001.
It has a Data Format as follows:
Also the manual:
All the measuring values are transmitted int32 or uint32 or float depending on the sensors
Code
import telnetlib
import struct
import time
# IP Address, Port, timeout for Telnet
tn = telnetlib.Telnet("169.254.168.150", 10001, 10)
while True:
op = tn.read_eager() # currently read information limit this till preamble
print(op[::-1]) # make little-endian
if not len(op[::-1]) == 0: # initially an empty bit starts (b'')
data = struct.unpack('!4c', op[::-1]) # unpacking `MEAS`
time.sleep(0.1)
my initial attempt:
Connect to the sensor
read data
make it to little-endian
OUTPUT
b''
b'MEAS\x85\x8c\x8c\x07\xa7\x9d\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x86\x8c\x8c\x07\xa7\x9e\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x85\x8c\x8c\x07\xa7\x9f\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa0\x01\x0c'
b'\xa7\xa2\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa1\x01\x0c'
b'\x8c\x07\xa7\xa3\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07'
b'\x88\x8c\x8c\x07\xa7\xa4\x01\x0c\x15\x04\xf6MEAS\x88\x8c'
b'MEAS\x8b\x8c\x8c\x07\xa7\xa5\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xa6\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xa7\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa8\x01\x0c'
b'\x01\x0c\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa9\x01\x0c'
b'\x8c\x07\xa7\xab\x01\x0c\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xaa'
b'\x8c\x8c\x07\xa7\xac\x01\x0c\x15\x04\xf6MEAS\x8c\x8c'
b'AS\x89\x8c\x8c\x07\xa7\xad\x01\x0c\x15\x04\xf6MEAS\x8a'
b'MEAS\x88\x8c\x8c\x07\xa7\xae\x01\x0c\x15\x04\xf6ME'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xaf\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb0\x01\x0c'
b'\x0c\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb1\x01\x0c'
b'\x07\xa7\xb3\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c\x07\xa7\xb2\x01'
b'\x8c\x8c\x07\xa7\xb4\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c'
b'\x85\x8c\x8c\x07\xa7\xb5\x01\x0c\x15\x04\xf6MEAS\x84'
b'MEAS\x87\x8c\x8c\x07\xa7\xb6\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb7\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb8\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb9\x01\x0c'
b'\xa7\xbb\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xba\x01\x0c'
try to unpack the preamble !?
How do I read information like Article number, Serial number, Channel, Status, Measuring Value between the preamble?
The payload size seems to be fixed here for 22 Bytes (via Wireshark)
Parsing the reversed buffer is just weird; please use struct's support for endianess. Using big-endian '!' in a little-endian context is also odd.
The first four bytes are a text constant. Ok, fine perhaps you'll need to reverse those. But just those, please.
After that, use struct.unpack to parse out 'IIQI'. So far, that was kind of working OK with your approach, since all fields consume 4 bytes or a pair of 4 bytes. But finding frame M's length is the fly in the ointment since it is just 2 bytes, so parse it with 'H', giving you a combined 'IIQIH'. After that, you'll need to advance by only that many bytes, and then expect another 'MEAS' text constant once you've exhausted that set of measurements.
I managed to avoid TelnetLib altogether and created a tcp client using python3. I had the payload size already from my wireshark dump (22 Bytes) hence I keep receiving 22 bytes of Information. Apparently the module sends two distinct 22 Bytes payload
First (frame) payload has the preamble, serial, article, channel information
Second (frame) payload has the information like bytes per frame, measuring value counter, measuring value Channel 1, measuring value Channel 2, measuring value Channel 3
The information is in int32 and thus needs a formula to be converted to real readings (mentioned in the instruction manual)
(as mentioned by #J_H the unpacking was as He mentioned in his answer with small changes)
Code
import socket
import time
import struct
DRANGEMIN = 3261
DRANGEMAX = 15853
MEASRANGE = 50
OFFSET = 35
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('169.254.168.150', 10001)
print('connecting to %s port %s' % server_address)
sock.connect(server_address)
def value_mm(raw_val):
return (((raw_val - DRANGEMIN) * MEASRANGE) / (DRANGEMAX - DRANGEMIN) + OFFSET)
if __name__ == '__main__':
while True:
Laser_Value = 0
data = sock.recv(22)
preamble, article, serial, x1, x2 = struct.unpack('<4sIIQH', data)
if not preamble == b'SAEM':
status, bpf, mValCounter, CH1, CH2, CH3 = struct.unpack('<hIIIII',data)
#print(CH1, CH2, CH3)
Laser_Value = CH3
print(str(value_mm(Laser_Value)) + " mm")
#print('RAW: ' + str(len(data)))
print('\n')
#time.sleep(0.1)
Sure enough, this provides me the information that is needed and I compared the information via the propreitary software which the company provides.
I am working on a server engine in Python, for my game made in GameMaker Studio 2. I'm currently having some issues with making and sending a packet.
I've successfully managed to establish a connection and send the first packet, but I can't find a solution for sending data in a sequence of which if the first byte in the packed struct is equal to a value, then unpack other data into a given sequence.
Example:
types = 'hhh' #(message_id, x, y) example
message_id = 0
x = 200
y = 200
buffer = pack(types, 0,x, y)
On the server side:
data = conn.recv(BUFFER_SIZE)
mid = unpack('h', data)[0]
if not data: break
if mid == 0:
sequnce = 'hhh'
x = unpack(sequnce, data)[1]
y = unpack(sequnce, data)[2]
It looks like your subsequent decoding is going to vary based on the message ID?
If so, you will likely want to use unpack_from which allows you to pull only the first member from the data (as written now, your initial unpack call will generate an exception because the buffer you're handing it is not the right size). You can then have code that varies the unpacking format string based on the message ID. That code could look something like this:
from struct import pack, unpack, unpack_from
while True:
data = conn.recv(BUFFER_SIZE)
# End of file, bail out of loop
if not data: break
mid = unpack_from('!h', data)[0]
if mid == 0:
# Message ID 0
types = '!hhh'
_, x, y = unpack(types, data)
# Process message type 0
...
elif mid == 1:
types = '!hIIq'
_, v, w, z = unpack(types, data)
# Process message type 1
...
elif mid == 2:
...
Note that we're unpacking the message ID again in each case along with the ID-specific parameters. You could avoid that if you like by using the optional offset argument to unpack_from:
x, y = unpack_from('!hh', data, offset=2)
One other note of explanation: If you are sending messages between two different machines, you should consider the "endianness" (byte ordering). Not all machines are "little-endian" like x86. Accordingly it's conventional to send integers and other structured numerics in a certain defined byte order - traditionally that has been "network byte order" (which is big-endian) but either is okay as long as you're consistent. You can easily do that by prepending each format string with '!' or '<' as shown above (you'll need to do that for every format string on both sides).
Finally, the above code probably works fine for a simple "toy" application but as your program increases in scope and complexity, you should be aware that there is no guarantee that your single recv call actually receives all the bytes that were sent and no other bytes (such as bytes from a subsequently sent buffer). In other words, it's often necessary to add a buffering layer, or otherwise ensure that you have received and are operating on exactly the number of bytes you intended.
Could you unpack whole data to list, and then check its elements in the loop? What is the reason to unpack it 3 times? I guess, you could unpack it once, and then work with that list - check its length first, if not empty -> check first element -> if equal to special one, continue on list parsing. Did you try like that?
I'm currently working on a project that involves some very remote data gathering. At the end of each day, a very short summary is sent back to the server via a satellite connection.
Because sending data over the satellite is very expensive, I want the data to be as compact as possible. Additionally, the service I'm using only allows for sending data in the following two formats: ASCII, hexadecimal. Most of the data I will be sending consists of floats, where the precision should be as high as possible without taking up too much space.
Below I have a working version of what I'm currently using, but there should be a more efficient way to store the data. Any help would be much appreciated.
import ctypes
#------------------------------------------------------------------
#This part is known to the client as well as the server
class my_struct(ctypes.Structure):
_pack_ = 1
_fields_ = [('someText', ctypes.c_char * 12),
('underRange', ctypes.c_float),
('overRange', ctypes.c_float),
('TXPDO', ctypes.c_float)]
def print_struct(filled_struct):
d = filled_struct
for name,typ in d._fields_:
value = getattr(d, name)
print('{:20} {:20} {}'.format(name, str(value), str(typ)))
#------------------------------------------------------------------
# This part of the code is performed on the client side
#Filling the struct with some random data, real data will come from sensors
s = my_struct()
s.someText = 'Hello World!'.encode()
s.underRange = 4.01234
s.overRange = 4.012345
s.TXPDO = 1.23456789
#Rounding errors occurred when converting to ctypes.c_float
print('Data sent:')
print_struct(s)
data = bytes(s) #Total length is 24 bytes (12 for the string + 3x4 for the floats)
data_hex = data.hex() #Total length is 48 bytes in hexadecimal format
#Now the data is sent over a satellite connection, it should be as small as possible
print('\nLength of sent data: ',len(data_hex),'bytes\n')
#------------------------------------------------------------------
# This part of the code is performed on the server side
def move_bytes_to_struct(struct_to_fill,bytes_to_move):
adr = ctypes.addressof(struct_to_fill)
struct_size = ctypes.sizeof(struct_to_fill)
bytes_to_move = bytes_to_move[0:struct_size]
ctypes.memmove(adr, bytes_to_move, struct_size)
return struct_to_fill
#Data received can be assumed to be the same as data sent
data_hex_received = data_hex
data_bytes = bytes.fromhex(data_hex_received)
data_received = move_bytes_to_struct(my_struct(), data_bytes)
print('Data received:')
print_struct(data_received)
I don't know if you are overcomplicating things a bit.
The struct module will let you do pretty much what you are doing, but simpler:
struct.pack("fffs12", 4.01234, 4.012345, 1.23456789, 'Hello World!'.encode())
This of course depends on you knowing how long your string is, but you could also not care about that:
struct.pack("fff", 4.01234, 4.012345, 1.23456789) + 'Hello World!'.encode()
But, about saving things more efficient:
The more you know about your data, the more shortcuts you can take.
Is the string only ascii? You could squeeze each char into maybe 7 bits, or even 6.
That could bring your 12 bytes string to 9.
If you know the range of your floats, you could perhaps trim that as well.
If you can send larger batches, compression might help as well.
I have revised my question in response to being put on hold. Hopefully this will better match SO standards.
The purpose of this program is to build and send UDP packets which use Alternating Bit Protocol as a simple resending mechanism. I have confirmed already that the packets can be sent and received correctly. The issue is with the ABP bit and its flipping.
The problem facing me now is that despite trying multiple different methods, I cannot flip the ABP bit used for confirming that a packet or ack received is the correct numbered one. I start out by sending a packet with ABP bit=0, and in response, the receiving process should see this and send back an ack with ABP bit=0. Upon receiving that, the sender program flips its ABP bit to 1 and sends a new packet with this ABP bit value. The receiver will get that, send a matching ack with ABP bit=1, and the sender will receive, flip its bit back to 0, and continue the cycle until the program has finished sending information.
Code below, sorry for length, but it is complete and ready to run. The program takes four command line arguments, here is the command I have been using:
% python ftpc.py 164.107.112.71 4000 8000 manygettysburgs.txt
where ftpc.py is the name of the sender program, 164.107.112.71 is an IP address, 4000 and 8000 are port numbers, and manygettysburgs.txt is a text file I have been sending. It should not make a difference if a different .txt is used, but for full accuracy use a file with a length of between 8000 and 9000 characters.
import sys
import socket
import struct
import os
import select
def flipBit(val): #flip ABP bit from 0 to 1 and vice versa
foo = 1 - val
return foo
def buildPacketHeader(IP, Port, Flag, ABP):
#pack IP for transport
#split into four 1-byte values
SplitIP = IP.split('.')
#create a 4-byte struct to pack IP, and pack it in remoteIP
GB = struct.Struct("4B")
remoteIP = GB.pack(int(SplitIP[0]),int(SplitIP[1]),int(SplitIP[2]),int(SplitIP[3]))
#remoteIP is now a 4-byte string of packed IP values
#pack Port for transport
#create a 2-byte struct to pack port, and pack it in remotePort
GBC = struct.Struct("H")
remotePort = GBC.pack(int(Port)) #needs another byte
#remotePort is now a 2-byte string
#add flag
flag = bytearray(1)
flag = str(Flag)
#add ABP
ABP = flipBit(ABP)
abp = str(ABP)
#create header and join the four parts together
Header = ''.join(remoteIP)
Header += remotePort
Header += flag
Header += abp
return Header
#assign arguments to local values
IP = sys.argv[1]
PORT = sys.argv[2]
TPORT = sys.argv[3]
localfile = sys.argv[4]
#declare the socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
#create destination information arrays
remoteIP = bytearray(4)
remotePort = bytearray(2)
#create flag array
flag = bytearray(1)
#create ABP bit
bit = 1
print bit
#send file size packet
#get size of file from os command
filesize = bytearray(4)
#save as 4-byte string
filesize = str(os.stat(localfile).st_size)
#package the filesize string
filesizestr = buildPacketHeader(IP,PORT,1,bit) #build header
print bit
filesizestr += filesize #complete the packet
s.sendto(filesizestr, ('127.0.0.1', int(TPORT))) #send packet
# end of send file name packet
#begin listening for responses
read, write, err = select.select([s], [], [], 1) #timeout set to 1 seconds
if len(read) > 0:
#data received
data = read[0].recv(1200)
if data[7] != bit:
print "failed ack"
#resend packet
else:
print "Timeout."
#resend packet
#send next data packet
#get filename as string (from arg4 diredtly)
filename = bytearray(20)
#save as 20-byte string
filename = localfile
#package the filename string
filenamestr = buildPacketHeader(IP,PORT,2,bit) #build header
print bit
filenamestr += filename #complete the packet
s.sendto(filenamestr, ('127.0.0.1', int(TPORT))) #send packet
#end of send file name packet
#send file content packet
#reading while loop goes here
with open(localfile, 'rb', 0) as f: #open the file
while True:
fstr = f.read(1000)
if not fstr:
print "NOTHING"
break
#put together the main packet base
filecontentstr = buildPacketHeader(IP,PORT,3,bit)
print bit
filecontentbytearray = bytearray(1000) #create ytear array
filecontentbytearray = fstr #assign fstr to byte array
filecontentsend = ''.join(filecontentstr) #copy filecontentstr to new string since we will be using filecontentstr again in the future for other packets
filecontentsend += filecontentbytearray #append read data to be sent
s.sendto(filecontentsend, ('127.0.0.1', int(TPORT))) #send the file content packet
#end of send file content packet
s.close()
In this code, every time that buildPacketHeader is called, it performs flipBit as part of its operations. flipBit is supposed to flip the bit's value for ABP. I have prints set up to print out the new value of bit after all calls to buildPacketHeader, so as to track the value. Whenever I run the program, however, I always see the same value for the ABP bit.
I've tried several methods, including changing to a bool. Here are some changes to flipBit I have tried:
def flipBit(val): #flip ABP bit from 0 to 1 and vice versa
if val == 0:
val = 1
else:
val = 0
return val
and some with bools instead:
def flipBit(val):
val = not val
return val
def flipBit(val):
val = (True, False)[val]
return val
I figure that many of these methods are in fact working options due to past experience. That said, I am completely baffled as to why it is not working as expected in this program. I would assume that it is my inexperience with python that is at fault, for despite having now used it for a decent amount of time, there still are peculiarities which escape me. Any help is greatly appreciated.
I don't understand what your objection is to Python ints, but the ctypes module provides a world of low-level mutable objects; e.g.,
>>> import ctypes
>>> i = ctypes.c_ushort(12) # 2-byte unsigned integer, initial value 12
>>> i
c_ushort(12)
>>> i.value += 0xffff - 12
>>> hex(i.value)
'0xffff'
>>> i.value += 1
>>> i.value # silently overflowed to 0
0
This is my first time ever answering my own SO question, so hopefully this turns out right. If anyone has additions or further answers, then feel free to answer as well, or comment on this one.
I ended up solving the problem by adding another return value to the return statement of buildPacketHeader, so that in addition to returning a string I also return the new value of bit as well. I confirmed that it was working by setting up the following prints inside of buildPacketHeader:
#add ABP
print "before:",ABP #test line for flipBit
ABP = flipBit(ABP)
abp = str(ABP)
print "after:",ABP #test line for flipBit
The output of which is shown here (I ended it early but the proof of functionality is still visible)
% python ftpc.py 164.107.112.70 4000 8000 manygettysburgs.txt
before: 1
after: 0
Timeout, resending packet...
before: 0
after: 1
Timeout, resending packet...
before: 1
after: 0
As can be seen, the before of the second packet is the after of the first packet, and the before of the third packet is the after of the second packet. Through this, you can see that the program is now flipping bits correctly.
The change made to buildPacketHeader is shown below:
return Header
becomes
return Header, ABP
and calls to buildPacketHeader:
filesizestr = buildPacketHeader(IP,PORT,1,bit)
become
filesizestr, bit = buildPacketHeader(IP,PORT,1,bit)
Very simple for such a bother. Make sure you return values if you want to make them change.