We Keep Coding

python integer format conversion

I need an application that will do the following:
read UDP message, get data from it - the data is textual, comma separated.
build a new message based on this values
send a new UDP message to other destination.
For the first and the last I'm fine. for the second I have a challenge:
Lets say that the data I've received is Year = 2020. The message on the Wireshark (step 3), I shall see it as 2 bytes valued 07 e4
How can I do it? I've tried couple of ways, none of them provided me with the desired format.
Sample of the code:
data1 = '\xab\xcd\xef\xgh'
...
data, addr = sock.recvfrom (200)
elements = data.decode().split(",")
Date=elements[15].split("/")
Year = int(Date(2))
year = <do something with Year to convert to right format>
newMsg = data1 + year
...
newSock.sendto(newMsg.encode('charmap'), (IP, int(port)))
Python version 3.5

there seem to be a couple of issues here. first, if your data1 is really supposed to be raw bytes then you're better off declaring it as such by prefixing it with a b, something like:
data1 = b'\xab\xcd\xef\xff'
if it comes from somewhere else, then encode it to bytes appropriately.
for answering your main question, the struct module has useful tools for encoding numbers as bytes in the way you want. ctypes can also be useful, but struct tends to be easier to use for these sorts of things:
data, addr = sock.recvfrom (200)
elements = data.decode().split(",")
date = elements[15].split("/")
year = int(date(2))
newMsg = data1 + struct.pack('!h', year)
and then you can send it with:
newSock.sendto(newMsg, (IP, int(port)))

How do I decode data when I don't know the specific encoding

I'm receiving data using an OSC server and the data looks like this:
b'Muse-C46F/elements/alpha_absolute\x00\x00\x00,ffff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
b'Muse-C46F/elements/alpha_relative\x00\x00\x00,ffff\x00\x00\x00\x7f\xc0\x00\x00\x7f\xc0\x00\x00\x7f\xc0\x00\x00\x7f\xc0\x00\x00'
b'Muse-C46F/elements/alpha_session_score\x00\x00,ffff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
b'Muse-C46F/elements/alpha_absolute\x00\x00\x00,ffff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
I'm trying to decode 4 floats from those encoded sequences.
This is my whole code:
print('Program Initiated')
UDP_IP = "192.168.2.57"
UDP_PORT = 4000
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
if 'alpha' in str(data):
print(struct.unpack('>32s8sffff', data))
I saw a similar question and used the print(struct.unpack('>32s8sffff', data)), but I get unpack requires a buffer of 56 bytesas error.

These OSC messages consists of three parts:
An address (which ought to start with b'/')
A type tag string, which starts with b',', and defines the type and number of arguments that follow
The messages arguments, as defined by the type tag string.
In the data provided in the question, each message consists of 36 or 40 bytes of address, then 5 bytes of tag type string. Disregarding the initial comma, the tag type string consists of four 'f's, so we expect the arguments to consist of four floats, consisting of 16 bytes (4 per float).
Each message has some extra bytes after the 16 required for the four floats; let's assume* that these are padding that can be discarded.
Therefore the struct format is going to be: a variable number of address characters, five tag type characters and four floats.
Then the code required to extract the data looks like this:
$ cat osc.py
import struct
data = [
b"Muse-C46F/elements/alpha_absolute\x00\x00\x00,ffff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x
b"Muse-C46F/elements/alpha_relative\x00\x00\x00,ffff\x00\x00\x00\x7f\xc0\x00\x00\x7f\xc0\x00\x00\x7f\xc0\x00\x00\x
b"Muse-C46F/elements/alpha_session_score\x00\x00,ffff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
b"Muse-C46F/elements/alpha_absolute\x00\x00\x00,ffff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x
]
if __name__ == "__main__":
for msg in data:
num_address_bytes = msg.index(b",")
num_argument_bytes = len(msg) - (num_address_bytes + 5)
num_extra_bytes = num_argument_bytes - 16
address, type_, *floats = struct.unpack(
">{}s5s4f".format(num_address_bytes), msg[:-num_extra_bytes]
)
print(address, type_, floats)
This output is generated:
b'Muse-C46F/elements/alpha_absolute\x00\x00\x00' b',ffff' [0.0, 0.0, 0.0, 0.0]
b'Muse-C46F/elements/alpha_relative\x00\x00\x00' b',ffff' [1.7796490496925177e-43, -2.000030279159546, -2.000030279159546, -2.000030279159546]
b'Muse-C46F/elements/alpha_session_score\x00\x00' b',ffff' [0.0, 0.0, 0.0, 0.0]
b'Muse-C46F/elements/alpha_absolute\x00\x00\x00' b',ffff' [0.0, 0.0, 0.0, 0.0]
* It's generally a bad idea to make assumptions. If the OSC server has documentation, check whether its formats deviate from the OSC spec.

unable to unpack information between custom Preamble in Python and telnetlib

I have an industrial sensor which provides me information via telnet over port 10001.
It has a Data Format as follows:
Also the manual:
All the measuring values are transmitted int32 or uint32 or float depending on the sensors
Code
import telnetlib
import struct
import time
# IP Address, Port, timeout for Telnet
tn = telnetlib.Telnet("169.254.168.150", 10001, 10)
while True:
op = tn.read_eager() # currently read information limit this till preamble
print(op[::-1]) # make little-endian
if not len(op[::-1]) == 0: # initially an empty bit starts (b'')
data = struct.unpack('!4c', op[::-1]) # unpacking `MEAS`
time.sleep(0.1)
my initial attempt:
Connect to the sensor
read data
make it to little-endian
OUTPUT
b''
b'MEAS\x85\x8c\x8c\x07\xa7\x9d\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x86\x8c\x8c\x07\xa7\x9e\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x85\x8c\x8c\x07\xa7\x9f\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa0\x01\x0c'
b'\xa7\xa2\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa1\x01\x0c'
b'\x8c\x07\xa7\xa3\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07'
b'\x88\x8c\x8c\x07\xa7\xa4\x01\x0c\x15\x04\xf6MEAS\x88\x8c'
b'MEAS\x8b\x8c\x8c\x07\xa7\xa5\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xa6\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xa7\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa8\x01\x0c'
b'\x01\x0c\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa9\x01\x0c'
b'\x8c\x07\xa7\xab\x01\x0c\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xaa'
b'\x8c\x8c\x07\xa7\xac\x01\x0c\x15\x04\xf6MEAS\x8c\x8c'
b'AS\x89\x8c\x8c\x07\xa7\xad\x01\x0c\x15\x04\xf6MEAS\x8a'
b'MEAS\x88\x8c\x8c\x07\xa7\xae\x01\x0c\x15\x04\xf6ME'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xaf\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb0\x01\x0c'
b'\x0c\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb1\x01\x0c'
b'\x07\xa7\xb3\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c\x07\xa7\xb2\x01'
b'\x8c\x8c\x07\xa7\xb4\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c'
b'\x85\x8c\x8c\x07\xa7\xb5\x01\x0c\x15\x04\xf6MEAS\x84'
b'MEAS\x87\x8c\x8c\x07\xa7\xb6\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb7\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb8\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb9\x01\x0c'
b'\xa7\xbb\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xba\x01\x0c'
try to unpack the preamble !?
How do I read information like Article number, Serial number, Channel, Status, Measuring Value between the preamble?
The payload size seems to be fixed here for 22 Bytes (via Wireshark)

Parsing the reversed buffer is just weird; please use struct's support for endianess. Using big-endian '!' in a little-endian context is also odd.
The first four bytes are a text constant. Ok, fine perhaps you'll need to reverse those. But just those, please.
After that, use struct.unpack to parse out 'IIQI'. So far, that was kind of working OK with your approach, since all fields consume 4 bytes or a pair of 4 bytes. But finding frame M's length is the fly in the ointment since it is just 2 bytes, so parse it with 'H', giving you a combined 'IIQIH'. After that, you'll need to advance by only that many bytes, and then expect another 'MEAS' text constant once you've exhausted that set of measurements.

I managed to avoid TelnetLib altogether and created a tcp client using python3. I had the payload size already from my wireshark dump (22 Bytes) hence I keep receiving 22 bytes of Information. Apparently the module sends two distinct 22 Bytes payload
First (frame) payload has the preamble, serial, article, channel information
Second (frame) payload has the information like bytes per frame, measuring value counter, measuring value Channel 1, measuring value Channel 2, measuring value Channel 3
The information is in int32 and thus needs a formula to be converted to real readings (mentioned in the instruction manual)
(as mentioned by #J_H the unpacking was as He mentioned in his answer with small changes)
Code
import socket
import time
import struct
DRANGEMIN = 3261
DRANGEMAX = 15853
MEASRANGE = 50
OFFSET = 35
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('169.254.168.150', 10001)
print('connecting to %s port %s' % server_address)
sock.connect(server_address)
def value_mm(raw_val):
return (((raw_val - DRANGEMIN) * MEASRANGE) / (DRANGEMAX - DRANGEMIN) + OFFSET)
if __name__ == '__main__':
while True:
Laser_Value = 0
data = sock.recv(22)
preamble, article, serial, x1, x2 = struct.unpack('<4sIIQH', data)
if not preamble == b'SAEM':
status, bpf, mValCounter, CH1, CH2, CH3 = struct.unpack('<hIIIII',data)
#print(CH1, CH2, CH3)
Laser_Value = CH3
print(str(value_mm(Laser_Value)) + " mm")
#print('RAW: ' + str(len(data)))
print('\n')
#time.sleep(0.1)
Sure enough, this provides me the information that is needed and I compared the information via the propreitary software which the company provides.

Unspecified byte lengths in Python

I'm writing a client for a P2P application at the minute and the spec for the protocol says that the header for each packet should have each field with a particular byte length like so:
Version: 1 Byte
Type: 1 Byte
Length: 2 Bytes
And then the data
I've got the way of packing and unpacking the header fields (I think) like this:
packed = struct.pack('cch' , '1' , '1' , 26)
This constructs a header for a packet with a data length of 26, but when it comes to unpacking the data I'm unsure how to go about getting the rest of the data afterwards. To unpack we need to know the size of all the fields, unless I'm missing something? I guess to pack the data I'd use a format indicator 'cch26s' meaning:
1 Byte char
1 Byte char
2 Byte short
26 Byte char array
But how do I unpack the data when I don't know how much data will be included in the packet first?

The way you're describing the protocol, you should unpack the first four bytes first, and extract Length (a 16-bit int). This tells you how many bytes to unpack in a second step.
version, type, length = struct.unpack("cch", packed[:4])
content, = struct.unpack("%ds" % length, packed[4:])
This is if everything checks out. unpack() requires that the packed buffer contain exactly as much data as you unpack. Also, check whether the 4 header bytes are included in the length count.

You can surmise the number of characters to unpack by inspecting len(data).
Here is a helper function which does this for you:
def unpack(fmt, astr):
"""
Return struct.unpack(fmt, astr) with the optional single * in fmt replaced with
the appropriate number, given the length of astr.
"""
# http://stackoverflow.com/a/7867892/190597
try:
return struct.unpack(fmt, astr)
except struct.error:
flen = struct.calcsize(fmt.replace('*', ''))
alen = len(astr)
idx = fmt.find('*')
before_char = fmt[idx-1]
n = (alen-flen)/struct.calcsize(before_char)+1
fmt = ''.join((fmt[:idx-1], str(n), before_char, fmt[idx+1:]))
return struct.unpack(fmt, astr)
You can use it like this:
unpack('cchs*', data)

Getting TCP packet payload from Python and impacket

I've been able to find packets of interest using code based on this example:
How can I filter a pcap file by specific protocol using python?
The next child from the TCP packet is the actual data:
if isinstance(child1, TCP):
if child1.get_th_dport() == 80:
x = child1.child()
print x
This prints out the packet data like wire shark and shows hex and ascii versions. However I have been unable so far to find a way to simply get the hex contents. I know I can manipulate the printable output but I figured there must be a way to get the data in the hex form...
I've looked through the samples but none seem to do this. Anybody know the right way?

You can use packet.get_data_as_string() to get the raw bytes, and then display it however you like. I've replicated the "hex column" output produced by print child. Should be easy to tweak to produce ASCII columns as well:
def display_hex(pkt, cols=8):
size = cols * 4
data = ''.join('%02x' % ord(b) for b in pkt.get_data_as_string())
for i in range(0, len(data), size):
for j in range(0, size, 4):
print data[i+j:i+j+4],
print
if isinstance(child, TCP):
display_hex(child)
Output:
1703 0103 b0b1 9387 be4e fe00 9230 6192
e3bb 217e c1cb 8511 556f f986 4f31 542d
15c6 f42e f3bb 93d5 cf33 f126 c174 dbc4
... snip ...
8b1d 8707 96d6 7a18 2aab fd0b 48ee c4eb
b7d8 a67f 8bc0 597d 1044 a076 1a9e 24ba
959b fda3 1adb 2384 669c e6c8 c3b5 bef4
1189 eda8 3e