I came across this peice of code here: to compute checksum.
As far as I understand in order to segregate the binary data structure into 16 bit words as required for TCP checksum:
I recon the value of w should be dirieved as w= ord(msg[i]) << 8 + ord(msg[i+1]) unless, the byte order has to be changed. I am not too sure as to why w would be assigned value as w = ord(msg[i]) + ord(msg[i+1]) << 8. Is there anything specific I am missing here?
def checksum(msg):
s = 0
# loop taking 2 characters at a time
for i in range(0, len(msg), 2):
w = ord(msg[i]) + (ord(msg[i+1]) << 8 )
s = s + w
s = (s>>16) + (s & 0xffff);
s = s + (s >> 16);
#complement and mask to 4 byte short
s = ~s & 0xffff
return s
In this case I think "network order", "big endian" and "little endian" are being mixed with the TCP Checksum calculation.
The TCP Checksum calculation is defined in RFC 1071: https://www.rfc-editor.org/rfc/rfc1071
At the beginning of page 2:
Using the notation [a,b] for the 16-bit integer a*256+b, where a and b are bytes,
The bytes in the Pseudo Header and partially filled TCP Header are just "bytes" and no implication is made as to what they mean (they must already be in "network order")
The formula used by the author is just following RFC 1071
Related
I am trying to read a binary message from an ESP32 using a broker; i wrote a phyton script where I subscribe the topic. the message that i actually receive is:
b'\x00\x00\x00?'
this is a float binary little endian message but I don't the key to decode it. Is there a way to find the decode key based on this data?
This is my python code:
import paho.mqtt.client as mqtt
def on_connect1(client1, userdata1, flags1, rc1):
client1.subscribe("ESP32DevKit123/mytopic")
def on_message1(client1, userdata1, msg1):
print(msg1.topic+" "+ "TESTENZA: "+str(msg1.payload))
client1 = mqtt.Client()
client1.username_pw_set(username="myuser",password="mypassword")
client1.on_connect = on_connect1
client1.on_message = on_message1
client1.connect("linkclient", portnumber, 60)
def twosComplement_hex(hexval):
bits = 16 # Number of bits in a hexadecimal number format
on_message1 = int(hexval, bits)
if on_message1 & (1 << (bits-1)):
on_message1 -= 1 << bits
return on_message1
client1.loop_forever()
It also gives me an error in the line on_message1 -= 1 << bits; the error says: Expected intended block pylance. Any solutions?
The data you provided is b'\x00\x00\x00?' - I'm going to assume that this is 0000003f (please output hex with msg1.payload.hex()).
I'll also assume that by "float binary little endian" you mean a big endian floating point (IEE754) - note that this does not match up with the algorithm you are using (twos compliment). Plugging this input into an online tool indicates that the expected result ("Float - Big Endian (ABCD)") is 8.82818e-44 (it's worth checking with this tool; sometimes the encoding may not be what you think it is!).
Lets unpack this using python (see the struct docs for more information):
>>> from struct import unpack
>>> unpack('>f', b'\x00\x00\x00\x3f')[0]
8.828180325246348e-44
Notes:
The [0] is there because unpack returns an array (you can unpack more than one item from the input)
>f - the > means big-endian and the f float (standard size = 4 bytes)
The reason your original code gives the error "Expected intended block" is due to the lack of indentation in the line on_message1 -= 1 << bits (as it follows an if it needs to be indented). The algorithm does not appear relevant to your task (but there may be details I'm missing).
I'm a heavy LabVIEW user who is just starting to learn Python. I work with industrial and aerospace equipment a lot and something I need to do very often is process some data, then export it over some communications protocol in binary. For example, let's say I have a packet that contains a struct/cluster/other-complex-data-element that has the following underlying data elements:
sync - unsigned 32-bit integer
time - 64-bit double
payload ID - 16-bit signed integer
source - 16-but signed integer
destination - 16-bit signed integer
payload length - 32 bit signed integer
data 1 - 8-bit unsigned integer
data 2 - 8-bit unsigned integer
data 3 - 8-bit unsigned integer
data 4 - 8-bit unsigned integer
data 4 - 64-bit double
data 5 - 32-bit single
data 6 - 16 bit unsigned integer
crc - 32-bit unsigned integer
(This frame should be 42 bytes long)
I call this a frame, where there is some header information, a payload, then a crc, I think that's a common term for what I'm creating. The data types, and their location in the byte stream is critical. Any extraneous or missing bytes breaks the data transfer protocol and the data cannot be tolerated.
My question is this:
How do you achieve this easily in Python? In LabVIEW (and probably other languages), there are good, built in functions and methods to clearly define the data types, then flatten them to a string of bytes that is very efficient. It seems that with picking, there are things going on that I don't understand.
In my example code, I have a simple function to get some memory information, then serialize it. I would expect the integer version to have 88 bytes and the float version to have 172 bytes, but I get 87 and 115 respectively. Here is the code, thanks for your help!
import psutil
import time
import pickle
def getMemoryInfo():
while True:
virtual_memory = psutil.virtual_memory()
swap_memory = psutil.swap_memory()
memoryInfo = list(virtual_memory+swap_memory)
# memoryInfo = [float(x) for x in memoryInfo]
time.sleep(1.000)
print(memoryInfo)
string = pickle.dumps(memoryInfo)
print(string)
print(len(memoryInfo))
print(len(string))
getMemoryInfo()
The struct module worked for me. It was a little more tedious than I would have hoped, but it worked just fine. Here is the code I ended up using:
def build_frame(payload, payload_class, payload_id, source, destination):
# form frame header and payload
frame = {"sync": int(0x64617665),
"absolute_time": time.time(),
"relative_time": time.monotonic(),
"source": int(source),
"destination": int(destination),
"counter": 0,
"payload_class": int(payload_class),
"payload_id": int(payload_id),
"payload_length": int(len(payload)),
"payload": payload}
# form bytearray to crc
sync = (struct.pack('<I', frame['sync']))
absolute_time = (struct.pack('<d', frame['absolute_time']))
relative_time = (struct.pack('<d', frame['relative_time']))
source = (struct.pack('i', frame['source']))
destination = (struct.pack('i', frame['destination']))
counter = (struct.pack('I', frame['counter']))
payload_class = (struct.pack('i', frame['payload_class']))
payload_id = (struct.pack('i', frame['payload_id']))
payload_length = (struct.pack('i', frame['payload_length']))
payload_bytes = frame['payload']
crc_bytes = sync + absolute_time + relative_time + source + destination + counter + payload_class + payload_id + payload_length + payload_bytes
# crc bytes and add to frame
frame['crc'] = binascii.crc32(crc_bytes)
return frame
I'm working on a communication command protocol between a PLC and a 3rd party device.
The manufacturer has provided me with the following information for calculating the CRC values that will change depending on the address of the device I wish to read information from.
A CRC is performed on a block of data, for example the first seven bytes of all transmissions are followed by a two byte CRC for that data. This CRC will be virtually unique for that particular combination of bytes. The process of calculating the CRC follows:
Inputs:
N.BYTES = Number or data bytes to CRC ( maximum 64 bytes )
DATA() = An array of data bytes of quantity N.BYTES
CRC.MASK = 0xC9DA a hexadecimal constant used in the process
Outputs:
CRC = two byte code redundancy check made up of CRC1 (High byte) and CRC2 (Low byte)
Process:
START
CRC = 0xFFFF
FOR N = 1 TO N.BYTES
CRC = CRC XOR ( DATA(N) AND 0xFF )
FOR I = 1 TO 8
IF ( CRC AND 0x0001 ) = 0 THEN LSR CRC
ELSE LSR CRC ; CRC = CRC XOR CRC.MASK
NEXT I
NEXT N
X = CRC1 ; Change the two bytes in CRC around
CRC1 = CRC2
CRC2 = X
END
They also provided me with a couple of complete command strings for the first few device addresses.
RTU #1
05-64-00-02-10-01-00-6C-4B-53-45-EB-F7
RTU #2
05-64-00-02-10-02-00-1C-AE-53-45-EB-F7
RTU #3
05-64-00-02-10-03-00-CC-F2-53-45-EB-F7
The header CRC bytes in the previous three commands are 6C-4B, 1C-AE, and CC-F2 respectively.
I calculated out a the first few lines by hand to have something to compare against when I wrote out the following code in Python.
byte1 = 05
byte2 = 100
byte3 = 00
byte4 = 02
byte5 = 16
byte6 = 01
byte7 = 00
byte8 = 00
mask = 51674
hexarray = [byte1, byte2, byte3, byte4, byte5, byte6, byte7, byte8]
#print hexarray
CRCdata = 65535
for n in hexarray:
CRCdata = CRCdata ^ (n & 255)
print(n, CRCdata)
for i in range(1,8):
if (CRCdata & 1) == 0:
CRCdata = CRCdata >> 1
# print 'if'
else:
CRCdata = CRCdata >> 1
CRCdata = CRCdata ^ mask
# print 'else'
print(i, CRCdata)
print CRCdata
I added byte8 due to some research I did mentioning that an extra byte of 0s needs to be added to the end of the CRC array for calculations. I converted the final result and did the byte swap manually. The problem I've been running into, is that my CRC calculations, whether I keep byte8 or not, are not matching up with any of the three examples that have been provided.
I'm not quite sure where I am going wrong on this and any help would be greatly appreciated.
I was able to solve the issue by updating the code to range(0,8) and dropping byte8.
I want to send long values using python to an arduino board which runs c++. The serial communication breaks the 4 byte numbers up and sends them byte by byte. When I try to reassemble them on the back end, I only get a valid number for 2 bytes instead of the four bytes I sent.
Here is the python code sending instructions.
pos1 = int(input("pos1: "))
pos2 = int(input("pos2: "))
data = struct.pack('<ll', pos1, pos2)
ser.write(data)
Here is the arduino code to parse the bytes that it reads.
if(Serial.available()>0){
size_t numbytes = Serial.readBytes(data, 8);
for(int i=0; i<8; i++){
Serial.println(data[i], HEX);
}
pos1 = readfourbytes(data[0], data[1], data[2], data[3]);
pos2 = readfourbytes(data[4], data[5], data[6], data[7]);
Serial.println(pos1);
Serial.println(pos2);
}
long readfourbytes(byte fourthbyte, byte thirdbyte, byte thirdbyte, byte firstbyte){
long result = (firstbyte << 24) + (secondbyte << 16) + (thirdbyte << 8) + fourthbyte;
return result;
}
I guess this means the arduino is little endian? My problem is the second position value that is read is completely off. The python code seems to be the problem however I don't know why. when I send the int values of 100 for both, I get an output of
b'd\x00\x00\x00d\x00\x00\x00'
from the python code as the binary being sent in the data variable. But from the arduino, I recieve:
64
0
0
0
6D
2
0
0
100
621
So there is a disconnect between what I am sending and what I am recieving. The baudrates are the same and there is no other obvious fault that I am aware of.
All the expressions (<any>byte << <bits>) are evaluated as int that seems to be 16 bits on the arduino. Cast <any>byte into long, and you're done.
long readfourbytes(byte fourthbyte, byte thirdbyte, byte thirdbyte, byte firstbyte){
long result = ((long)firstbyte << 24) + ((long)secondbyte << 16) + (thirdbyte << 8) + fourthbyte;
return result;
}
I try to create a python client for bacula, but I have some problem with the authentication.
The algorithm is :
import hmac
import base64
import re
...
challenge = re.search("auth cram-md5 ()", data)
#exemple ''
passwd = 'b489c90f3ee5b3ca86365e1bae27186e'
hm = hmac.new(passwd, challenge).digest()
rep = base64.b64encode(hm).strp().rstrip('=')
#result with python : 9zKE3VzYQ1oIDTpBuMMowQ
#result with bacula client : 9z+E3V/YQ1oIDTpBu8MowB'
There's a way more simple than port the bacula's implemenation of base 64?
int
bin_to_base64(char *buf, int buflen, char *bin, int binlen, int compatible)
{
uint32_t reg, save, mask;
int rem, i;
int j = 0;
reg = 0;
rem = 0;
buflen--; /* allow for storing EOS */
for (i=0; i >= (rem - 6);
if (j
To verify your CRAM-MD5 implementation, it is best to use some simple test vectors and check combinations of (challenge, password, username) inputs against the expected output.
Here's one example (from http://blog.susam.in/2009/02/auth-cram-md5.html):
import hmac
username = 'foo#susam.in'
passwd = 'drowssap'
encoded_challenge = 'PDc0NTYuMTIzMzU5ODUzM0BzZGNsaW51eDIucmRzaW5kaWEuY29tPg=='
challenge = encoded_challenge.decode('base64')
digest = hmac.new(passwd, challenge).hexdigest()
response = username + ' ' + digest
encoded_response = response.encode('base64')
print encoded_response
# Zm9vQHN1c2FtLmluIDY2N2U5ZmE0NDcwZGZmM2RhOWQ2MjFmZTQwNjc2NzIy
That said, I've certainly found examples on the net where the response generated by the above code differs from the expected response stated on the relevant site, so I'm still not entirely clear as to what is happening in those cases.
I HAVE CRACKED THIS.
I ran into exactly the same problem you did, and have just spent about 4 hours identifying the problem, and reimplementing it.
The problem is the Bacula's base64 is BROKEN, AND WRONG!
There are two problems with it:
The first is that the incoming bytes are treated as signed, not unsigned. The effect of this is that, if a byte has the highest bit set (>127), then it is treated as a negative number; when it is combined with the "left over" bits from previous bytes are all set to (binary 1).
The second is that, after b64 has processed all the full 6-bit output blocks, there may be 0, 2 or 4 bits left over (depending on input block modulus 3). The standard Base64 way to handle this is to multiply the remaining bits, so they are the HIGHEST bits in the last 6-bit block, and process them - Bacula leaves them as the LOWEST bits.
Note that some versions of Bacula may accept both the "Bacula broken base64 encoding" and the standard ones, for incoming authentication; they seem to use the broken one for their authentication.
def bacula_broken_base64(binarystring):
b64_chars="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
remaining_bit_count=0
remaining_bits=0
output=""
for inputbyte in binarystring:
inputbyte=ord(inputbyte)
if inputbyte>127:
# REPRODUCING A BUG! set all the "remaining bits" to 1.
remaining_bits=(1 << remaining_bit_count) - 1
remaining_bits=(remaining_bits<<8)+inputbyte
remaining_bit_count+=8
while remaining_bit_count>=6:
# clean up:
remaining_bit_count-=6
new64=(remaining_bits>>remaining_bit_count) & 63 # 6 highest bits
output+=b64_chars[new64]
remaining_bits&=(1 << remaining_bit_count) - 1
if remaining_bit_count>0:
output+=b64_chars[remaining_bits]
return output
I realize it's been 6 years since you asked, but perhaps someone else will find this useful.