I am sure this will be simple for most experts on here. I have a few binary strings, the last of which is a checksum. I am trying to reproduce the calc to get the checksum, but I am unable to get there. I have tried and looked at many of the posts relating to checksum on here as well.
Here is the bitstring, spaced in groups of 8: 01010000 10000000 01100111 11110110
(11110110) should be the checksum. For testing I have done this:
def checksum(bytes):
return (sum(bytes) & 0xFF)
def bitstring_to_bytes(s):
v = int(s, 2)
b = bytearray()
while v:
b.append(v & 0xff)
v >>= 8
return bytes(b[::-1])
t= ('010100001000000001100111') #left out the checksum as expected
bytes = bitstring_to_bytes(t)
print('bytes = %s' % bytes)
checksum_bytes = checksum(bytes)
checksum_bytes = checksum_bytes | 0xf0
checksum_bytes = bin(checksum_bytes)
print('checksum =%s' % checksum_bytes)
The above gets close with 11110111 vice 11110110. The last bit is off.
Related
I am trying to write a code for crc16 in python but i am not getting correct remainder. I send hex_string as input
and also try for byte array and binary but still not getting proper output.what should be the input ??
import sys
import time
class CRC:
def crc16ibm(msg1):
#msg11=hex(msg1)
msg11 = list(msg1)
buflen = len(msg1)
print("The message is ZZZ : ", msg1)
#print("Hex value : " + hex(msg1))
print("The length is QQQ : ", len(msg1))
preset, offset, polynom =0, 0,0xA001
preset = preset & 0xFFFF
polynom = polynom & 0xFFFF
buflen_i: int = buflen
code_move = 0xFFFF
crc = preset
for i in range(buflen):
#data = msg11[(i+offset)% buflen_i] & code_move
data = (msg1[(i+offset)% buflen_i]) & code_move
#print(type(data))
crc ^= data
for j in range(8):
if((crc & 0x0001) !=0):
crc = (crc>>1)^polynom
else:
crc = crc>>1
return crc&0xFFFF
# Test data for the crc to run
print("First set of data : ")
#msg1 = 0x000000000000003608010000016B40D8EA30010000000000000000000000000000000105021503010101425E0F01F10000601A014E0000000000000000010000C7CF
#crc1 = 0x0000C7CF
msg1 = 0x000000000000004A8E010000016B412CEE000100000000000000000000000000000000010005000100010100010011001D00010010015E2C880002000B000000003544C87A000E000000001DD7E06A00000100002994
crc1 = 0x00002994
#msg1 = 0x000000000000008c08010000013feb55ff74000f0ea850209a690000940000120000001e09010002000300040016014703f0001504c8000c0900730a00460b00501300464306d7440000b5000bb60007422e9f180000cd0386ce000107c700000000f10000601a46000001344800000bb84900000bb84a00000bb84c00000000024e0000000000000000cf00000000000000000100003fca
#msg1 = 0xFFFF
#crc1 = 0xFFFF
print("The hex message is : " + hex(msg1))
print("The length of hex msg is :", len(hex(msg1)))
print("The CRC is ", hex(crc1))
print(" and lenght is ", len(hex(crc1)))
zero, m1, kind =0, len(hex(msg1)), 'b'
msg_a = f'{msg1:{zero}>{m1}{kind}}'
#print(type(msg_a))
msgInBin =bin(int(msg_a, 16)).zfill(8)
crcchecksum = crc16ibm(msgInBin)
print(" the msg_a is PPP : ", msg_a)
print(" The lenght of EEE: ", len(msg_a))
if (crcchecksum == crc1):
print("Checksum is saying the whole package is received")
else:
print("The ", hex(crc1), " is not matching is ", hex(crcchecksum))
The input would normally be bytes, which your examples suggest would be converted from hexadecimal to bytes.
Writing that hexadecimal input as a giant integer is not what you want to do. That throws away, for example, all of the leading zeros which appear to be part of the message. Your attempt to then use list() to get bytes out of that doesn't work. What you want is a character string of hexadecimal input, and to use bytearray.fromhex() to convert that to bytes.
Your examples have the CRC at the end. It would appear that you need to calculate the CRC on the bytes that precede the CRC, not including the CRC. You would then compare that to the CRC. (Those CRC bytes are not appended in the correct order to be able to compute the CRC on the entire message and expect to get zero.)
Where are your examples from? Can you link to the source?
Here is an example for your CRC parameters (known as CRC-16/ARC). msg should be a bytearray:
def crc16arc(msg):
crc = 0
for b in msg:
crc ^= b
for _ in range(8):
crc = (crc >> 1) ^ 0xa001 if crc & 1 else crc >> 1
return crc
print(hex(crc16arc(b'123456789')))
print(hex(crc16arc(bytearray.fromhex("313233343536373839"))))
print(hex(crc16arc(bytearray.fromhex("3132333435363738393dbb"))))
I want to determine the 8 bit modulo 256 checksum of an ASCII string. I know the formula is:
checksum = (sum of bytes)%256
How can I do this in python (manipulate bytes)? If I start with a string: '1c03e8', I should output 0x94. The main problem is I'm not sure as to how to find the sum of the bytes of an ASCII string. Here is the main idea of what I'm looking for :
https://www.scadacore.com/tools/programming-calculators/online-checksum-calculator/
It has CheckSum8 Modulo 256
I have tried:
component = ('1c03e8')
for i in range(len(component)):
checksum.append(int(float(component[i].encode("hex"))))
print checksum
print hex(int(sum(checksum)%256))
this although gives me 0x52
You to need to encode the string as ASCII, because as you said, it's an ASCII string.
Example, quick-and-dirty solution:
print(hex(sum('1c03e8'.encode('ascii')) % 256))
def calc_checksum(s):
sum = 0
for c in s:
sum += ord(c)
sum = sum % 256
return '%2X' % (sum & 0xFF)
print calc_checksum('1c03e8'.encode('ascii'))
Try the checksum-calculator package.
Step 1:
pip install checksum-calculator
Step 2:
from checksum_calculator import *
inputString = "your_string"
outputString = inputString.encode('utf-8').hex()
print(compute_checksum8_xor(outputString))
print(compute_checksum8_mod256(outputString))
print(compute_checksum8_2s_complement(outputString))
crc_table = None
def make_crc_table():
global crc_table
crc_table = [0] * 256
for n in xrange(256):
c = n
for k in xrange(8):
if c & 1:
c = 0xedb88320L ^ (c >> 1)
else:
c = c >> 1
crc_table[n] = c
make_crc_table()
"""
/* Update a running CRC with the bytes buf[0..len-1]--the CRC
should be initialized to all 1's, and the transmitted value
is the 1's complement of the final running CRC (see the
crc() routine below)). */
"""
def update_crc(crc, buf):
c = crc
for byte in buf:
c = crc_table[int((c ^ ord(byte)) & 0xff)] ^ (c >> 8)
return c
# /* Return the CRC of the bytes buf[0..len-1]. */
def crc(buf):
return update_crc(0xffffffffL, buf) ^ 0xffffffffL
I used this code to calculate png crc value
My IHDR chunk data is 000008A0 000002FA 08020000 00 and the result of that code was 0xa1565b1L
However real crc was 0x84E42B87. I checked this value with well known png checker tool and correct crc was 0x84E42B87.
I can't understand how this value is calculated and correct value.
The CRC is calculated over the chunk type and the data, not just the data. So those bytes would be preceded by the four bytes IHDR. Then you get the correct CRC.
As an aside, I have no idea how you got 0xa1565b1L from 000008A0 000002FA 08020000 00. I get 0xa500050a as the CRC of those bytes. There must be something else that you're doing wrong as well. You would need to provide a complete example for us to be able to tell.
I have been trying to get my head around CRC32 calculations without much success, the values that I seem to get do not match what I should get.
I am aware that Python has libraries that are capable of generating these checksums (namely zlib and binascii) but I do not have the luxury of being able to use them as the CRC functionality do not exist on the micropython.
So far I have the following code:
import binascii
import zlib
from array import array
poly = 0xEDB88320
table = array('L')
for byte in range(256):
crc = 0
for bit in range(8):
if (byte ^ crc) & 1:
crc = (crc >> 1) ^ poly
else:
crc >>= 1
byte >>= 1
table.append(crc)
def crc32(string):
value = 0xffffffffL
for ch in string:
value = table[(ord(ch) ^ value) & 0x000000ffL] ^ (value >> 8)
return value
teststring = "test"
print "binascii calc: 0x%08x" % (binascii.crc32(teststring) & 0xffffffff)
print "zlib calc: 0x%08x" % (zlib.crc32(teststring) & 0xffffffff)
print "my calc: 0x%08x" % (crc32(teststring))
Then I get the following output:
binascii calc: 0xd87f7e0c
zlib calc: 0xd87f7e0c
my calc: 0x2780810c
The binascii and zlib calculations agree where as my one doesn't. I believe the calculated table of bytes is correct as I have compared it to examples available on the net. So the issue must be the routine where each byte is calculated, could anyone point me in the correct direction?
Thanks in advance!
I haven't looked closely at your code, so I can't pinpoint the exact source of the error, but you can easily tweak it to get the desired output:
import binascii
from array import array
poly = 0xEDB88320
table = array('L')
for byte in range(256):
crc = 0
for bit in range(8):
if (byte ^ crc) & 1:
crc = (crc >> 1) ^ poly
else:
crc >>= 1
byte >>= 1
table.append(crc)
def crc32(string):
value = 0xffffffffL
for ch in string:
value = table[(ord(ch) ^ value) & 0xff] ^ (value >> 8)
return -1 - value
# test
data = (
'',
'test',
'hello world',
'1234',
'A long string to test CRC32 functions',
)
for s in data:
print repr(s)
a = binascii.crc32(s)
print '%08x' % (a & 0xffffffffL)
b = crc32(s)
print '%08x' % (b & 0xffffffffL)
print
output
''
00000000
00000000
'test'
d87f7e0c
d87f7e0c
'hello world'
0d4a1185
0d4a1185
'1234'
9be3e0a3
9be3e0a3
'A long string to test CRC32 functions'
d2d10e28
d2d10e28
Here are a couple more tests that verify that the tweaked crc32 gives the same result as binascii.crc32.
from random import seed, randrange
print 'Single byte tests...',
for i in range(256):
s = chr(i)
a = binascii.crc32(s) & 0xffffffffL
b = crc32(s) & 0xffffffffL
assert a == b, (repr(s), a, b)
print('ok')
seed(42)
print 'Multi-byte tests...'
for width in range(2, 20):
print 'Width', width
r = range(width)
for n in range(1000):
s = ''.join([chr(randrange(256)) for i in r])
a = binascii.crc32(s) & 0xffffffffL
b = crc32(s) & 0xffffffffL
assert a == b, (repr(s), a, b)
print('ok')
output
Single byte tests... ok
Multi-byte tests...
Width 2
Width 3
Width 4
Width 5
Width 6
Width 7
Width 8
Width 9
Width 10
Width 11
Width 12
Width 13
Width 14
Width 15
Width 16
Width 17
Width 18
Width 19
ok
As discussed in the comments, the source of the error in the original code is that this CRC-32 algorithm inverts the initial crc buffer, and then inverts the final buffer contents. So value is initialised to 0xffffffff instead of zero, and we need to return value ^ 0xffffffff, which can also be written as ~value & 0xffffffff, i.e. invert value and then select the low-order 32 bits of the result.
If using binary data where the crc is chained over multiple buffers I used the following (using the OPs table):
def crc32(data, crc=0xffffffff):
for b in data:
crc = table[(b ^ crc) & 0xff] ^ (crc >> 8)
return crc
One can XOR the final result with -1 to agree with the online calculators.
crc = crc32(b'test')
print('0x{:08x}'.format(crc))
crc = crc32(b'te')
crc = crc32(b'st', crc)
print('0x{:08x}'.format(crc))
print('xor: 0x{:08x}'.format(crc ^ 0xffffffff))
output
0x278081f3
0x278081f3
xor: 0xd87f7e0c
I have a string that is packed such that each character was originally an unsigned byte but is stored as 7 bits and then packed into an unsigned byte array. I'm trying to find a quick way to unpack this string in Python but the function I wrote that uses the bitstring module works well but is very slow. It seems like something like this should not be so slow but I'm probably doing it very inefficiently...
This seems like something that is probably trivial but I just don't know what to use, maybe there is already a function that will unpack the string?
from bitstring import BitArray
def unpackString(raw):
msg = ''
bits = BitArray(bytes=raw)
mask = BitArray('0b01111111')
i = 0
while 1:
try:
iByte = (bits[i:i + 8] & mask).int
# value of 0 denotes a line break
if iByte == 0:
msg += '\n'
elif iByte >= 32 and iByte <= 126:
msg += chr(iByte)
i += 7
except:
break
return msg
This took me a while to figure out, as your solution seems to ignore the first bit of data. Given the input byte of 129 (0b10000001) I would expect to see 64 '1000000' printed by the following, but your code produces 1 '0000001' -- ignoring the first bit.
bs = b'\x81' # one byte string, whose value is 129 (0x81)
arr = BitArray(bs)
mask = BitArray('0b01111111')
byte = (arr[0:8] & mask).int
print(byte, repr("{:07b}".format(byte)))
Simplest solution would be to modify your solution to use bitstring.ConstBitStream -- I got an order of magnitude speed increase with the following.
from bitstring import ConstBitStream
def unpack_bitstream(raw):
num_bytes, remainder = divmod(len(raw) * 8 - 1, 7)
bitstream = ConstBitStream(bytes=raw, offset=1) # use offset to ignore leading bit
msg = b''
for _ in range(num_bytes):
byte = bitstream.read("uint:7")
if not byte:
msg += b'\n'
elif 32 <= byte <= 126:
msg += bytes((byte,))
# msg += chr(byte) # python 2
return msg
However, this can be done quite easily using only the standard library. This makes the solution more portable and, in the instances I tried, faster by another order of magnitude (I didn't try the cythonised version of bitstring).
def unpack_bytes(raw, zero_replacement=ord("\n")):
# use - 1 to ignore leading bit
num_bytes, remainder = divmod(len(raw) * 8 - 1, 7)
i = int.from_bytes(raw, byteorder="big")
# i = int(raw.encode("hex"), 16) # python 2
if remainder:
# remainder means there are unused trailing bits, so remove these
i >>= remainder
msg = []
for _ in range(num_bytes):
byte = i & 127
if not byte:
msg.append(zero_replacement)
elif 32 <= byte <= 126:
msg.append(byte)
i >>= 7
msg.reverse()
return bytes(msg)
# return b"".join(chr(c) for c in msg) # python 2
I've used python 3 to create these methods. If you're using python 2 then there are a number of adjustments you'll need to make. I've added these as comments after the line they are intended to replace and marked them python 2.