I need to use an embedded system running Python 1.5.2+ (!!!) with very few modules.
And there is no "struct" module usable...
Here is the list of usable modules :
marshal
imp
_main_
_builtin_
sys
md5
binascii
Yes that's it, no struct module...
So, I need to create a 4 bytes representation of an unsigned short integer to send to serial...
With struct :
date = day + month * 32 + (year - 2000) * 512
time = 100 * hour + minute
data = struct.pack(b'HH', date, time)
date on 2 bytes time on 2 bytes and everybody's happy...
But without using 'struct' module, how can I do that?
You can do something like this:
x = 0xabcd
packed_string = chr((x & 0xff00) >> 8) + chr(x & 0x00ff)
Here is a complete translation for you
Before
>>> import struct
>>> day = 1; month = 2; year = 2003
>>> hour = 4; minute = 5
>>> date = day + month * 32 + (year - 2000) * 512
>>> time = 100 * hour + minute
>>> data = struct.pack(b'HH', date, time)
>>> data
'A\x06\x95\x01'
>>> data.encode("hex")
'41069501'
And after
>>> data2 = chr(date & 0xFF) + chr((date >> 8) & 0xFF) + chr(time & 0xFF) + chr((time >> 8) & 0xFF)
>>> data2
'A\x06\x95\x01'
>>> data2.encode("hex")
'41069501'
>>>
I was able to do it by passing a list of the bytes to bytes():
data=bytes([date%256,date//256,time%256,time//256])
Related
I have been trying to get my head around CRC32 calculations without much success, the values that I seem to get do not match what I should get.
I am aware that Python has libraries that are capable of generating these checksums (namely zlib and binascii) but I do not have the luxury of being able to use them as the CRC functionality do not exist on the micropython.
So far I have the following code:
import binascii
import zlib
from array import array
poly = 0xEDB88320
table = array('L')
for byte in range(256):
crc = 0
for bit in range(8):
if (byte ^ crc) & 1:
crc = (crc >> 1) ^ poly
else:
crc >>= 1
byte >>= 1
table.append(crc)
def crc32(string):
value = 0xffffffffL
for ch in string:
value = table[(ord(ch) ^ value) & 0x000000ffL] ^ (value >> 8)
return value
teststring = "test"
print "binascii calc: 0x%08x" % (binascii.crc32(teststring) & 0xffffffff)
print "zlib calc: 0x%08x" % (zlib.crc32(teststring) & 0xffffffff)
print "my calc: 0x%08x" % (crc32(teststring))
Then I get the following output:
binascii calc: 0xd87f7e0c
zlib calc: 0xd87f7e0c
my calc: 0x2780810c
The binascii and zlib calculations agree where as my one doesn't. I believe the calculated table of bytes is correct as I have compared it to examples available on the net. So the issue must be the routine where each byte is calculated, could anyone point me in the correct direction?
Thanks in advance!
I haven't looked closely at your code, so I can't pinpoint the exact source of the error, but you can easily tweak it to get the desired output:
import binascii
from array import array
poly = 0xEDB88320
table = array('L')
for byte in range(256):
crc = 0
for bit in range(8):
if (byte ^ crc) & 1:
crc = (crc >> 1) ^ poly
else:
crc >>= 1
byte >>= 1
table.append(crc)
def crc32(string):
value = 0xffffffffL
for ch in string:
value = table[(ord(ch) ^ value) & 0xff] ^ (value >> 8)
return -1 - value
# test
data = (
'',
'test',
'hello world',
'1234',
'A long string to test CRC32 functions',
)
for s in data:
print repr(s)
a = binascii.crc32(s)
print '%08x' % (a & 0xffffffffL)
b = crc32(s)
print '%08x' % (b & 0xffffffffL)
print
output
''
00000000
00000000
'test'
d87f7e0c
d87f7e0c
'hello world'
0d4a1185
0d4a1185
'1234'
9be3e0a3
9be3e0a3
'A long string to test CRC32 functions'
d2d10e28
d2d10e28
Here are a couple more tests that verify that the tweaked crc32 gives the same result as binascii.crc32.
from random import seed, randrange
print 'Single byte tests...',
for i in range(256):
s = chr(i)
a = binascii.crc32(s) & 0xffffffffL
b = crc32(s) & 0xffffffffL
assert a == b, (repr(s), a, b)
print('ok')
seed(42)
print 'Multi-byte tests...'
for width in range(2, 20):
print 'Width', width
r = range(width)
for n in range(1000):
s = ''.join([chr(randrange(256)) for i in r])
a = binascii.crc32(s) & 0xffffffffL
b = crc32(s) & 0xffffffffL
assert a == b, (repr(s), a, b)
print('ok')
output
Single byte tests... ok
Multi-byte tests...
Width 2
Width 3
Width 4
Width 5
Width 6
Width 7
Width 8
Width 9
Width 10
Width 11
Width 12
Width 13
Width 14
Width 15
Width 16
Width 17
Width 18
Width 19
ok
As discussed in the comments, the source of the error in the original code is that this CRC-32 algorithm inverts the initial crc buffer, and then inverts the final buffer contents. So value is initialised to 0xffffffff instead of zero, and we need to return value ^ 0xffffffff, which can also be written as ~value & 0xffffffff, i.e. invert value and then select the low-order 32 bits of the result.
If using binary data where the crc is chained over multiple buffers I used the following (using the OPs table):
def crc32(data, crc=0xffffffff):
for b in data:
crc = table[(b ^ crc) & 0xff] ^ (crc >> 8)
return crc
One can XOR the final result with -1 to agree with the online calculators.
crc = crc32(b'test')
print('0x{:08x}'.format(crc))
crc = crc32(b'te')
crc = crc32(b'st', crc)
print('0x{:08x}'.format(crc))
print('xor: 0x{:08x}'.format(crc ^ 0xffffffff))
output
0x278081f3
0x278081f3
xor: 0xd87f7e0c
For a personal project, I'm working on implementing SHA-256 in Python 3, without using the hashlib module (since that would defeat the purpose of learning how SHA-256 works). I've been working from the Wikipedia pseudocode, but my code gives incorrect output (compared to the hashlib output). I've been staring at the code for an hour, and besides a headache, I've made no headway on figuring out what I've done wrong.
The code:
#!/usr/bin/env python3
import hashlib
import sys
# ror function taken from http://stackoverflow.com/a/27229191/2508324
def ror(val, r_bits, max_bits=32):
return ((val & (2**max_bits-1)) >> r_bits%max_bits)|(val << (max_bits-(r_bits%max_bits)) & (2**max_bits-1))
h = [0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19]
k = [0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2]
s = sys.stdin.read().encode()
msg = [int(x,2) for c in s for x in '{:08b}'.format(c)]
msg.append(1)
while len(msg) % 512 != 448:
msg.append(0)
msg.extend([int(x,2) for x in '{:064b}'.format(len(s))])
for i in range(len(msg)//512):
chunk = msg[512*i:512*(i+1)] # sloth love chunk
w = [0 for _ in range(64)]
for j in range(16):
w[j] = int(''.join(str(x) for x in chunk[32*j:32*(j+1)]),2)
for j in range(16, 64):
s0 = ror(w[j-15], 7) ^ ror(w[j-15], 18) ^ (w[j-15] >> 3)
s1 = ror(w[j-2], 17) ^ ror(w[j-2], 19) ^ (w[j-2] >> 10)
w[j] = (w[j-16] + s0 + w[j-7] + s1) % 2**32
work = h[:]
for j in range(64):
S1 = ror(work[4], 6) ^ ror(work[4], 11) ^ ror(work[4], 25)
ch = (work[4] & work[5]) ^ (~work[4] & work[6])
temp1 = (work[7] + S1 + ch + k[j] + w[j]) % 2**32
S0 = ror(work[0], 2) ^ ror(work[0], 13) ^ ror(work[0], 22)
maj = (work[0] & work[1]) ^ (work[0] & work[2]) ^ (work[1] & work[2])
temp2 = (S0 + maj) % 2**32
work = [(temp1 + temp2) % 2**32] + work[:-1]
work[4] = (work[4] + temp1) % 2**32
h = [(H+W)%2**32 for H,W in zip(h,work)]
print(''.join('{:08x}'.format(H) for H in h))
print(hashlib.sha256(s).hexdigest())
If the implementation was correct, the two outputs would match. Instead, I get this (with input abc):
$ echo -n abc | ./sha256.py
203b1d9016060802fe5ef80436611159de1868b58d44940e3d3979eab5f4d193
ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad
I have thoroughly examined the code, but I do not see any differences between it and the Wikipedia pseudocode. I suspect the error is in the compression loop (for j in range(64):). I've manually debugged and reviewed the state of the program up through initializing the first 16 words of the w array, and it all checks out.
Any help would be greatly appreciated!
SHA1 works on bits, not bytes. Therefore, the 64 bit length at the end of the padding is expressed in bits as well; the mistake is in the line
msg.extend([int(x,2) for x in '{:064b}'.format(len(s))])
which should be
msg.extend([int(x,2) for x in '{:064b}'.format(len(s) * 8)])
I have a number like 0x5423 where I want to extract 4 values:
a = 0x5 # 15 downto 12
b = 0x42 # 11 downto 3
c = 0x3 # 3 downto 2
d = 0x00 # 1 downto 0
I discovered the module bitstrings that looks great. Unfortunately, for an unknown reason, the bits are numbered from the right.
This is bad because if a add some upper bits like 0xA5423 my extraction won't work anymore:
field = bitstrings.BitArray('0x5423')
a = field[0:4].uint
b = field[4:12].uint
c = field[12:14].uint
d = field[14:16].uint
How can I properly extract my bitfields without complex arithmetic manipulations such as:
b = (a >> 4) & 0xFF
Ideally I would have:
b = field.range(11, 4)
Convert the string to 0x#### format before pass to bitstring.BitArray:
>>> n = '0xA5423'
>>> n = '0x{:04x}'.format(int(n, 16) & 0xffff) # => '0x5423'
>>> field = bitstring.BitArray(n)
>>> field[0:4].uint
5
>>> field[4:12].uint # 0x42 == 66
66
>>> field[12:14].uint
0
>>> field[14:16].uint
3
UPDATE another solution that does not depend on bitstring, and count from left(according to OP):
Convert the number into binary format:
>>> n = '0xA5423'
>>> n = format(int(n, 16), '016b')[::-1] # reversed
>>> n
'11000100001010100101'
>>> int(n[0:2][::-1], 2) # need to reverse again to get proper value
3
>>> int(n[2:4][::-1], 2)
0
>>> int(n[4:12][::-1], 2)
66
>>> int(n[12:16][::-1], 2)
5
I need to compile a program that can convert a Gregorian date to a Mayan one. I also need to use 01/01/1970 as a reference date.
The following auxiliary functions work accordingly and have no mistakes.
# turns dates into tuples
def dmj(date):
"""
>>> dmj('01/01/1970')
(1, 1, 1970)
>>> dmj('00012+00012+02012')
(12, 12, 2012)
"""
tup = ()
for i in date:
if i.isdigit() == False and i != ' ':
date = date.replace(i, ' ')
number_str = ''
for i in date:
number_str += i
if i == ' ':
number = int(number_str)
tup += (number,)
number_str = ''
tup += (int(number_str),)
return tup
# counts days that have passed since 01/01/1970
def daysPassed(date):
"""
>>> daysPassed('01/01/1970')
0
>>> daysPassed('20-7-1988')
6775
>>> daysPassed('00012+00012+02012')
15686
"""
from datetime import date
tup = dmj(date)
begin = date(1970, 1, 1)
end = date(tup[2], tup[1], tup[0])
passed = abs(end - begin)
return passed.days
My idea was to first calculate how many days have passed since the beginning of this pictun (20 baktuns long) in 01/01/1970 and then added the days passed since then according to the given date.
In the Mayan calendar a day is refered to as a kin. Their periods (within one pictun) are as follows:
20 kin = 1 uinal; 18 uinal = 1 tun; 20 tun = 1 katun; 20 katun = 1 baktun
In the long count notation the Mayan date for 01/01/1970 is '12.17.16.7.5'. Baktun are written first, then katuns, etc... Mayan dates start from 0. Basically the first kin of a uinal is number zero and the last one number 19, twenty in total.
I've first compiled the following:
def mayanDate(date, separation='')
"""
>>> mayanDate('01/01/1970')
'12/17/16/7/5'
>>> mayaDate('20-7-1988', separator='/')
'12/18/15/4/0'
>>> mayaDate('00012+00012+02012', separator='-')
'12-19-19-17-11'
>>> mayaDate('21 12 2012', separator='+')
'13+0+0+0+0'
>>> mayaDate('26.03.2407')
'14.0.0.0.0'
"""
days = daysPassed(date) + 13 * 144000 + 18 * 7200\
+ 17 * 400 + 8 * 20 + 6
baktun = str((days //((20 **3) * 18)) - 1)
days = days % ((20 **3) * 18)
katun = str((days //((20 **2) * 18)) - 1)
days = days % ((20 **2) * 18)
tun = str((days // (20 **2)) - 1)
days = days % (20 **2)
uinal = str((days // 20) - 1)
days = days % 20 - 1
kin = str(days)
mayanlist = [baktun, katun, tun, uinal, kin]
for i in date:
if i.isdigit() == False and separator == '':
separator = i
break
mayandate = separator.join(mayanlist)
return mayandate
For some strange reason only 01/01/1970 returns the correct Mayan long notation, despite counting from the beginning of an entire pictun (7,900 years in length!). For all other dates it seems to advance too fast through the calendar, despite my second auxiliary function returning the correct values (even for millennia into the future).
I wonder where's the mistake. For instance mayanDate('20-7-1988') returns '12-18-15-6-0' instead of '12-18-15-4-0' and mayanDate('21 12 2012') returns '13 0 1 12 0' instead of '13 0 0 0 0'.
The issue you're seeing with the negative 1 kin for date '15/01/1970' is due to the removal of one from every date ordinal during the calculation. Taking x%20 will always return a value between 0 and 19 inclusive. Taking one from the result necessarily shifts this range to -1 to 18 inclusive.
The number added to the result of daysPassed(date) seems to be a conversion of the long form of 1/1/1970, with one added to each digit. I'm assuming this has been done to counter the fact that the Mayan calendar starts counting at zero, but is unnecessary. The Mayan date 0.0.0.0.1.5 counts 25, not 151646. This doesn't appear to be the source of the error though, since removing this issue from my own code, I still get the same results as described for the 20-7-1988 and 21-12-2012.
I finally rooted out the error when I went back and switched out all the magic numbers in my code for named constants (it makes the code far easier to debug, read and maintain). You state there are 18 uinal in a tun, and 20 tun in a katun, but these numbers are reversed in the code.
Here's my code:
def mayanDate(date_str, seperation=','):
days_in_kin = 1
kin_in_uinal = 20
uinal_in_tun = 18
tun_in_katun = 20
katun_in_baktun = 20
days_in_uinal = days_in_kin * kin_in_uinal
days_in_tun = days_in_uinal * uinal_in_tun
days_in_katun = days_in_tun * tun_in_katun
days_in_baktun = days_in_katun * katun_in_baktun
days_1970 = 12 * days_in_baktun \
+ 17 * days_in_katun\
+ 16 * days_in_tun\
+ 7 * days_in_uinal\
+ 5 * days_in_kin
total_days = daysPassed(date_str) + days_1970
baktun = total_days // days_in_baktun
total_days -= baktun * days_in_baktun
katun = total_days // days_in_katun
total_days -= katun * days_in_katun
tun = total_days // days_in_tun
total_days -= tun * days_in_tun
uinal = total_days // days_in_uinal
total_days -= uinal * days_in_uinal
kin = total_days // days_in_kin
print seperation.join(map(str, (baktun, katun, tun, uinal, kin)))
(I subtracted the previous calculations from total days, rather than using a modulo operator, since I feel it's cleaner. I guess it's a matter of personal preference.)
I may have found something.
>>>mayanDate('15/01/1970')
'12/17/16/8/-1'
Obviously not possible. -1 has to be 19 here and 8 has to be 7. It seems to turn month too early. Still not out why 01/01/1970 remains correct here. No idea what's so special about that date.
I want to realize IDEA algorithm in Python. In Python we have no limits for variable size, but I need limit bit number in the integer number, for example, to do cyclic left shift. What do you advise?
One way is to use the BitVector library.
Example of use:
>>> from BitVector import BitVector
>>> bv = BitVector(intVal = 0x13A5, size = 32)
>>> print bv
00000000000000000001001110100101
>>> bv << 6 #does a cyclic left shift
>>> print bv
00000000000001001110100101000000
>>> bv[0] = 1
>>> print bv
10000000000001001110100101000000
>>> bv << 3 #cyclic shift again, should be more apparent
>>> print bv
00000000001001110100101000000100
An 8-bit mask with a cyclic left shift:
shifted = number << 1
overflowed = (number & 0x100) >> 8
shifted &= 0xFF
result = overflowed | shifted
You should be able to make a class that does this for you. With a bit more of the same, it can shift an arbitrary amount out of an arbitrary sized value.
The bitstring module might be of help (documentation here). This example creates a 22 bit bitstring and rotates the bits 3 to the right:
>>> from bitstring import BitArray
>>> a = BitArray(22) # creates 22-bit zeroed bitstring
>>> a.uint = 12345 # set the bits with an unsigned integer
>>> a.bin # view the binary representation
'0b0000000011000000111001'
>>> a.ror(3) # rotate to the right
>>> a.bin
'0b0010000000011000000111'
>>> a.uint # and back to the integer representation
525831
If you want a the low 32 bits of a number, you can use binary-and like so:
>>> low32 = (1 << 32) - 1
>>> n = 0x12345678
>>> m = ((n << 20) | (n >> 12)) & low32
>>> "0x%x" % m
'0x67812345'