Related
I have been tasked to take a hex text and convert it base64 system. My problem is I am getting an incorrect output from my code.
Input given:
49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d
Output expected:
SSdtIGtpbGxpbmcgeW91ciBicmFpbiBsaWtlIGEgcG9pc29ub3VzIG11c2hyb29t
Output from my program:
kk7aga2lY2NL5nIHLe6uiBicMLS3IGxp1spAhIHBe0ub9ub3rmQNXVzaOTe3
How I expect my code to work :
NOTE: I understand that Python has built in ways to convert something to base64, as well as using int() to convert any base to decimal, I have opted to use my own as a way to understand the problem more.
Take a string of hexadecimal text and convert it to a decimal number.
Convert the decimal number to a binary number.
Separate the binary number into chunks of 24 bits.
Split the chunk of 24 bits into 4 sections of 6 bits each.
Convert each 6 bits into a decimal number.
Convert the decimal number into the base64-encoded letter/number. Starting from "A" (index of 0) to "/" (index of 63)
My code:
def convertToDecimal(text, original_base):
'''
Program assumes user is using it correctly. It does not bother checking for weird cases like a base being 0 or a negative.
'''
decimal = 0 # used for converting to decimal base first
exp = len(text) - 1 # starting exponent of the base (8^1, 8^0, etc)
hex_nums = {"a": 10, "b": 11, "c": 12, "d": 13, "e": 14, "f": 15}
# convert original_base to a decimal number
for val in text:
if val in hex_nums: # have to worry about letters
val = hex_nums[val]
decimal += int(val) * (original_base ** exp)
exp -= 1
return decimal
def base64(text, base):
letters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
bin_num = bin(convertToDecimal(text, base))[2:]
base64_val = ""
# used for indexing each chunk of binary values
i = 0
j = 25
# used for each chunk of six bits to be converted into a number
six_bit_chunk = ""
while True:
if j > len(bin_num):
# prevents index out of bounds error
break
bin_chunk = bin_num[i:j] # take a chunk of 24 bits
for x in range(0, 24, 6):
six_bit_chunk += bin_chunk[x:x + 6] # take a smaller chunk of 6 bits
index = convertToDecimal(six_bit_chunk, 2) # use the 6 bits to create a decimal value from 0-63
base64_val += letters[index]
six_bit_chunk = ""
i += 25
j += 25
return base64_val
new_text = base64("49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d",
16)
print(new_text)
My attempts to solve this problem have been to look at what the chunks of six bits produce, and they are creating the correct decimal number and the correct letter/number in base64.
Can't you just iterate over the binary representation in groups of 6 directly instead of grouping by 24 and separating into 4 groups again?
But it still has to be left-padded with zeros to a multiple of 24 before splitting into groups of 6.
My function:
def base64_v2(text, base):
base64_letters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
n1 = 24
n2 = 6
dec_num = int(text, base)
bin_str = bin(dec_num)[2:]
# pad with zeros to the left to get multiple of 'n1'
remainder = divmod(len(bin_str), n1)[1]
if remainder > 0:
additional_zeros = n1 - remainder
bin_str = ('0' * additional_zeros) + bin_str
return ''.join(
base64_letters[int(bin_str[i:i+n2], 2)]
for i in range(0, len(bin_str), n2))
My function passes all the following test cases:
sample_data = [
(
'49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d',
'SSdtIGtpbGxpbmcgeW91ciBicmFpbiBsaWtlIGEgcG9pc29ub3VzIG11c2hyb29t',
),
(
'24e1f218372064987d5457b639819715bddb51558cc4acb3ce6eeacd7afa2751bac0a5c7c3c636f24babff4ad68473db',
'JOHyGDcgZJh9VFe2OYGXFb3bUVWMxKyzzm7qzXr6J1G6wKXHw8Y28kur/0rWhHPb',
),
(
'79d6e3a9d5bfba9902274e1b1c2d956fa92d16ba9a972825eb0b32aecd2401653624f900fe79b4550c24aae136ae7260',
'edbjqdW/upkCJ04bHC2Vb6ktFrqalygl6wsyrs0kAWU2JPkA/nm0VQwkquE2rnJg',
),
(
'd251b89c473d6336dfeb3714d92c6b2080c863624ca2746e1380bd0642893ba64ebdae23bb7ad9a1f3ad9efbdc2f18e6',
'0lG4nEc9Yzbf6zcU2SxrIIDIY2JMonRuE4C9BkKJO6ZOva4ju3rZofOtnvvcLxjm',
),
(
'5c2b578064178ad329f37041b063ec05c3ce8f202bb44e9a1260c6ded11ddd91d25ac83bba31bac7987e2da3a188c23d',
'XCtXgGQXitMp83BBsGPsBcPOjyArtE6aEmDG3tEd3ZHSWsg7ujG6x5h+LaOhiMI9',
),
(
'018b79f5c3c1a4f59d12cda25c5ca2a29c4c1fdd1cfdf3f0a4faf350fe384d21bcfd83a8350b49231cf8536595f2a43a',
'AYt59cPBpPWdEs2iXFyiopxMH90c/fPwpPrzUP44TSG8/YOoNQtJIxz4U2WV8qQ6',
),
(
'a617cfbe469cecd19f5ac75303a3049319ffb03d9a757c690d7c09d94dbabd6dce2314e1f409e6285fc0a0220eb803fe',
'phfPvkac7NGfWsdTA6MEkxn/sD2adXxpDXwJ2U26vW3OIxTh9AnmKF/AoCIOuAP+',
),
(
'2bc40041dbe6937e1113b191fd136bcdd741169e9e81809e83ad3104d447d700ed9d1ab5cfbc113c0731b855fde98f87',
'K8QAQdvmk34RE7GR/RNrzddBFp6egYCeg60xBNRH1wDtnRq1z7wRPAcxuFX96Y+H',
),
(
'1904435f5964861c5284b08e0ebf3d201da3ec795a3d9b0f7d5056c4369daed59d42cac72c897356a46305f7aac5fcb4',
'GQRDX1lkhhxShLCODr89IB2j7HlaPZsPfVBWxDadrtWdQsrHLIlzVqRjBfeqxfy0',
),
]
for i, (input_str, expected_output) in enumerate(sample_data):
print('i ', i)
print('input_str ', input_str)
print('expected_output', expected_output)
function_output = base64_v2(input_str, 16)
print('function_output', function_output)
assert expected_output == function_output
I'm not entirely sure if this is correct, but with a few changes I achieved that the output of the function matches the expected output you provided.
The changes (also taking into account the comment from user #JamesKPolt):
everywhere you had 25 y used 24 instead
add zero padding to the beginning of bin_num to complete the length to a multiple of 24
The modified code:
def base64(text, base):
letters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
bin_num = bin(convertToDecimal(text, base))[2:]
# pad with zeros to the left to get multiple of 24
remainder = divmod(len(bin_num), 24)[1]
if remainder > 0:
additional_zeros = 24 - remainder
bin_num = ('0' * additional_zeros) + bin_num
base64_val = ""
# used for indexing each chunk of binary values
i = 0
j = 24
# used for each chunk of six bits to be converted into a number
six_bit_chunk = ""
while True:
if j > len(bin_num):
# prevents index out of bounds error
break
bin_chunk = bin_num[i:j] # take a chunk of 24 bits
for x in range(0, 24, 6):
six_bit_chunk += bin_chunk[x:x + 6] # take a smaller chunk of 6 bits
index = convertToDecimal(six_bit_chunk, 2) # use the 6 bits to create a decimal value from 0-63
base64_val += letters[index]
six_bit_chunk = ""
i += 24
j += 24
return base64_val
This modified function passes all of the following test cases:
sample_data = [
(
'49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d',
'SSdtIGtpbGxpbmcgeW91ciBicmFpbiBsaWtlIGEgcG9pc29ub3VzIG11c2hyb29t',
),
(
'24e1f218372064987d5457b639819715bddb51558cc4acb3ce6eeacd7afa2751bac0a5c7c3c636f24babff4ad68473db',
'JOHyGDcgZJh9VFe2OYGXFb3bUVWMxKyzzm7qzXr6J1G6wKXHw8Y28kur/0rWhHPb',
),
(
'79d6e3a9d5bfba9902274e1b1c2d956fa92d16ba9a972825eb0b32aecd2401653624f900fe79b4550c24aae136ae7260',
'edbjqdW/upkCJ04bHC2Vb6ktFrqalygl6wsyrs0kAWU2JPkA/nm0VQwkquE2rnJg',
),
(
'd251b89c473d6336dfeb3714d92c6b2080c863624ca2746e1380bd0642893ba64ebdae23bb7ad9a1f3ad9efbdc2f18e6',
'0lG4nEc9Yzbf6zcU2SxrIIDIY2JMonRuE4C9BkKJO6ZOva4ju3rZofOtnvvcLxjm',
),
(
'5c2b578064178ad329f37041b063ec05c3ce8f202bb44e9a1260c6ded11ddd91d25ac83bba31bac7987e2da3a188c23d',
'XCtXgGQXitMp83BBsGPsBcPOjyArtE6aEmDG3tEd3ZHSWsg7ujG6x5h+LaOhiMI9',
),
(
'018b79f5c3c1a4f59d12cda25c5ca2a29c4c1fdd1cfdf3f0a4faf350fe384d21bcfd83a8350b49231cf8536595f2a43a',
'AYt59cPBpPWdEs2iXFyiopxMH90c/fPwpPrzUP44TSG8/YOoNQtJIxz4U2WV8qQ6',
),
(
'a617cfbe469cecd19f5ac75303a3049319ffb03d9a757c690d7c09d94dbabd6dce2314e1f409e6285fc0a0220eb803fe',
'phfPvkac7NGfWsdTA6MEkxn/sD2adXxpDXwJ2U26vW3OIxTh9AnmKF/AoCIOuAP+',
),
(
'2bc40041dbe6937e1113b191fd136bcdd741169e9e81809e83ad3104d447d700ed9d1ab5cfbc113c0731b855fde98f87',
'K8QAQdvmk34RE7GR/RNrzddBFp6egYCeg60xBNRH1wDtnRq1z7wRPAcxuFX96Y+H',
),
(
'1904435f5964861c5284b08e0ebf3d201da3ec795a3d9b0f7d5056c4369daed59d42cac72c897356a46305f7aac5fcb4',
'GQRDX1lkhhxShLCODr89IB2j7HlaPZsPfVBWxDadrtWdQsrHLIlzVqRjBfeqxfy0',
),
]
for i, (input_str, expected_output) in enumerate(sample_data):
print('i ', i)
print('input_str ', input_str)
print('expected_output', expected_output)
function_output = base64(input_str, 16)
print('function_output', function_output)
assert expected_output == function_output
I have a binary file containing a stream of 10-bit integers. I want to read it and store the values in a list.
It is working with the following code, which reads my_file and fills pixels with integer values:
file = open("my_file", "rb")
pixels = []
new10bitsByte = ""
try:
byte = file.read(1)
while byte:
bits = bin(ord(byte))[2:].rjust(8, '0')
for bit in reversed(bits):
new10bitsByte += bit
if len(new10bitsByte) == 10:
pixels.append(int(new10bitsByte[::-1], 2))
new10bitsByte = ""
byte = file.read(1)
finally:
file.close()
It doesn't seem very elegant to read the bytes into bits, and read it back into "10-bit" bytes. Is there a better way to do it?
With 8 or 16 bit integers I could just use file.read(size) and convert the result to an int directly. But here, as each value is stored in 1.25 bytes, I would need something like file.read(1.25)...
Here's a generator that does the bit operations without using text string conversions. Hopefully, it's a little more efficient. :)
To test it, I write all the numbers in range(1024) to a BytesIO stream, which behaves like a binary file.
from io import BytesIO
def tenbitread(f):
''' Generate 10 bit (unsigned) integers from a binary file '''
while True:
b = f.read(5)
if len(b) == 0:
break
n = int.from_bytes(b, 'big')
#Split n into 4 10 bit integers
t = []
for i in range(4):
t.append(n & 0x3ff)
n >>= 10
yield from reversed(t)
# Make some test data: all the integers in range(1024),
# and save it to a byte stream
buff = BytesIO()
maxi = 1024
n = 0
for i in range(maxi):
n = (n << 10) | i
#Convert the 40 bit integer to 5 bytes & write them
if i % 4 == 3:
buff.write(n.to_bytes(5, 'big'))
n = 0
# Rewind the stream so we can read from it
buff.seek(0)
# Read the data in 10 bit chunks
a = list(tenbitread(buff))
# Check it
print(a == list(range(maxi)))
output
True
Doing list(tenbitread(buff)) is the simplest way to turn the generator output into a list, but you can easily iterate over the values instead, eg
for v in tenbitread(buff):
or
for i, v in enumerate(tenbitread(buff)):
if you want indices as well as the data values.
Here's a little-endian version of the generator which gives the same results as your code.
def tenbitread(f):
''' Generate 10 bit (unsigned) integers from a binary file '''
while True:
b = f.read(5)
if not len(b):
break
n = int.from_bytes(b, 'little')
#Split n into 4 10 bit integers
for i in range(4):
yield n & 0x3ff
n >>= 10
We can improve this version slightly by "un-rolling" that for loop, which lets us get rid of the final masking and shifting operations.
def tenbitread(f):
''' Generate 10 bit (unsigned) integers from a binary file '''
while True:
b = f.read(5)
if not len(b):
break
n = int.from_bytes(b, 'little')
#Split n into 4 10 bit integers
yield n & 0x3ff
n >>= 10
yield n & 0x3ff
n >>= 10
yield n & 0x3ff
n >>= 10
yield n
This should give a little more speed...
As there is no direct way to read a file x-bit by x-bit in Python, we have to read it byte by byte. Following MisterMiyagi and PM 2Ring's suggestions I modified my code to read the file by 5 byte chunks (i.e. 40 bits) and then split the resulting string into 4 10-bit numbers, instead of looping over the bits individually. It turned out to be twice as fast as my previous code.
file = open("my_file", "rb")
pixels = []
exit_loop = False
try:
while not exit_loop:
# Read 5 consecutive bytes into fiveBytesString
fiveBytesString = ""
for i in range(5):
byte = file.read(1)
if not byte:
exit_loop = True
break
byteString = format(ord(byte), '08b')
fiveBytesString += byteString[::-1]
# Split fiveBytesString into 4 10-bit numbers, and add them to pixels
pixels.extend([int(fiveBytesString[i:i+10][::-1], 2) for i in range(0, 40, 10) if len(fiveBytesString[i:i+10]) > 0])
finally:
file.close()
Adding a Numpy based solution suitable for unpacking large 10-bit packed byte buffers like the ones you might receive from AVT and FLIR cameras.
This is a 10-bit version of #cyrilgaudefroy's answer to a similar question; there you can also find a Numba alternative capable of yielding an additional speed increase.
import numpy as np
def read_uint10(byte_buf):
data = np.frombuffer(byte_buf, dtype=np.uint8)
# 5 bytes contain 4 10-bit pixels (5x8 == 4x10)
b1, b2, b3, b4, b5 = np.reshape(data, (data.shape[0]//5, 5)).astype(np.uint16).T
o1 = (b1 << 2) + (b2 >> 6)
o2 = ((b2 % 64) << 4) + (b3 >> 4)
o3 = ((b3 % 16) << 6) + (b4 >> 2)
o4 = ((b4 % 4) << 8) + b5
unpacked = np.reshape(np.concatenate((o1[:, None], o2[:, None], o3[:, None], o4[:, None]), axis=1), 4*o1.shape[0])
return unpacked
Reshape can be omitted if returning a buffer instead of a Numpy array:
unpacked = np.concatenate((o1[:, None], o2[:, None], o3[:, None], o4[:, None]), axis=1).tobytes()
Or if image dimensions are known it can be reshaped directly, e.g.:
unpacked = np.reshape(np.concatenate((o1[:, None], o2[:, None], o3[:, None], o4[:, None]), axis=1), (1024, 1024))
If the use of the modulus operator appears confusing, try playing around with:
np.unpackbits(np.array([255%64], dtype=np.uint8))
Edit: It turns out that the Allied Vision Mako-U cameras employ a different ordering than the one I originally suggested above:
o1 = ((b2 % 4) << 8) + b1
o2 = ((b3 % 16) << 6) + (b2 >> 2)
o3 = ((b4 % 64) << 4) + (b3 >> 4)
o4 = (b5 << 2) + (b4 >> 6)
So you might have to test different orders if images come out looking wonky initially for your specific setup.
I have a string that is packed such that each character was originally an unsigned byte but is stored as 7 bits and then packed into an unsigned byte array. I'm trying to find a quick way to unpack this string in Python but the function I wrote that uses the bitstring module works well but is very slow. It seems like something like this should not be so slow but I'm probably doing it very inefficiently...
This seems like something that is probably trivial but I just don't know what to use, maybe there is already a function that will unpack the string?
from bitstring import BitArray
def unpackString(raw):
msg = ''
bits = BitArray(bytes=raw)
mask = BitArray('0b01111111')
i = 0
while 1:
try:
iByte = (bits[i:i + 8] & mask).int
# value of 0 denotes a line break
if iByte == 0:
msg += '\n'
elif iByte >= 32 and iByte <= 126:
msg += chr(iByte)
i += 7
except:
break
return msg
This took me a while to figure out, as your solution seems to ignore the first bit of data. Given the input byte of 129 (0b10000001) I would expect to see 64 '1000000' printed by the following, but your code produces 1 '0000001' -- ignoring the first bit.
bs = b'\x81' # one byte string, whose value is 129 (0x81)
arr = BitArray(bs)
mask = BitArray('0b01111111')
byte = (arr[0:8] & mask).int
print(byte, repr("{:07b}".format(byte)))
Simplest solution would be to modify your solution to use bitstring.ConstBitStream -- I got an order of magnitude speed increase with the following.
from bitstring import ConstBitStream
def unpack_bitstream(raw):
num_bytes, remainder = divmod(len(raw) * 8 - 1, 7)
bitstream = ConstBitStream(bytes=raw, offset=1) # use offset to ignore leading bit
msg = b''
for _ in range(num_bytes):
byte = bitstream.read("uint:7")
if not byte:
msg += b'\n'
elif 32 <= byte <= 126:
msg += bytes((byte,))
# msg += chr(byte) # python 2
return msg
However, this can be done quite easily using only the standard library. This makes the solution more portable and, in the instances I tried, faster by another order of magnitude (I didn't try the cythonised version of bitstring).
def unpack_bytes(raw, zero_replacement=ord("\n")):
# use - 1 to ignore leading bit
num_bytes, remainder = divmod(len(raw) * 8 - 1, 7)
i = int.from_bytes(raw, byteorder="big")
# i = int(raw.encode("hex"), 16) # python 2
if remainder:
# remainder means there are unused trailing bits, so remove these
i >>= remainder
msg = []
for _ in range(num_bytes):
byte = i & 127
if not byte:
msg.append(zero_replacement)
elif 32 <= byte <= 126:
msg.append(byte)
i >>= 7
msg.reverse()
return bytes(msg)
# return b"".join(chr(c) for c in msg) # python 2
I've used python 3 to create these methods. If you're using python 2 then there are a number of adjustments you'll need to make. I've added these as comments after the line they are intended to replace and marked them python 2.
How do I convert a hex string to a signed int in Python 3?
The best I can come up with is
h = '9DA92DAB'
b = bytes(h, 'utf-8')
ba = binascii.a2b_hex(b)
print(int.from_bytes(ba, byteorder='big', signed=True))
Is there a simpler way? Unsigned is so much easier: int(h, 16)
BTW, the origin of the question is itunes persistent id - music library xml version and iTunes hex version
In n-bit two's complement, bits have value:
bit 0 = 20
bit 1 = 21
bit n-2 = 2n-2
bit n-1 = -2n-1
But bit n-1 has value 2n-1 when unsigned, so the number is 2n too high. Subtract 2n if bit n-1 is set:
def twos_complement(hexstr, bits):
value = int(hexstr, 16)
if value & (1 << (bits - 1)):
value -= 1 << bits
return value
print(twos_complement('FFFE', 16))
print(twos_complement('7FFF', 16))
print(twos_complement('7F', 8))
print(twos_complement('FF', 8))
Output:
-2
32767
127
-1
import struct
For Python 3 (with comments' help):
h = '9DA92DAB'
struct.unpack('>i', bytes.fromhex(h))
For Python 2:
h = '9DA92DAB'
struct.unpack('>i', h.decode('hex'))
or if it is little endian:
h = '9DA92DAB'
struct.unpack('<i', h.decode('hex'))
Here's a general function you can use for hex of any size:
import math
# hex string to signed integer
def htosi(val):
uintval = int(val,16)
bits = 4 * (len(val) - 2)
if uintval >= math.pow(2,bits-1):
uintval = int(0 - (math.pow(2,bits) - uintval))
return uintval
And to use it:
h = str(hex(-5))
h2 = str(hex(-13589))
x = htosi(h)
x2 = htosi(h2)
This works for 16 bit signed ints, you can extend for 32 bit ints. It uses the basic definition of 2's complement signed numbers. Also note xor with 1 is the same as a binary negate.
# convert to unsigned
x = int('ffbf', 16) # example (-65)
# check sign bit
if (x & 0x8000) == 0x8000:
# if set, invert and add one to get the negative value, then add the negative sign
x = -( (x ^ 0xffff) + 1)
It's a very late answer, but here's a function to do the above. This will extend for whatever length you provide. Credit for portions of this to another SO answer (I lost the link, so please provide it if you find it).
def hex_to_signed(source):
"""Convert a string hex value to a signed hexidecimal value.
This assumes that source is the proper length, and the sign bit
is the first bit in the first byte of the correct length.
hex_to_signed("F") should return -1.
hex_to_signed("0F") should return 15.
"""
if not isinstance(source, str):
raise ValueError("string type required")
if 0 == len(source):
raise valueError("string is empty")
sign_bit_mask = 1 << (len(source)*4-1)
other_bits_mask = sign_bit_mask - 1
value = int(source, 16)
return -(value & sign_bit_mask) | (value & other_bits_mask)
I have this Python code to do this:
from struct import pack as _pack
def packl(lnum, pad = 1):
if lnum < 0:
raise RangeError("Cannot use packl to convert a negative integer "
"to a string.")
count = 0
l = []
while lnum > 0:
l.append(lnum & 0xffffffffffffffffL)
count += 1
lnum >>= 64
if count <= 0:
return '\0' * pad
elif pad >= 8:
lens = 8 * count % pad
pad = ((lens != 0) and (pad - lens)) or 0
l.append('>' + 'x' * pad + 'Q' * count)
l.reverse()
return _pack(*l)
else:
l.append('>' + 'Q' * count)
l.reverse()
s = _pack(*l).lstrip('\0')
lens = len(s)
if (lens % pad) != 0:
return '\0' * (pad - lens % pad) + s
else:
return s
This takes approximately 174 usec to convert 2**9700 - 1 to a string of bytes on my machine. If I'm willing to use the Python 2.7 and Python 3.x specific bit_length method, I can shorten that to 159 usecs by pre-allocating the l array to be the exact right size at the very beginning and using l[something] = syntax instead of l.append.
Is there anything I can do that will make this faster? This will be used to convert large prime numbers used in cryptography as well as some (but not many) smaller numbers.
Edit
This is currently the fastest option in Python < 3.2, it takes about half the time either direction as the accepted answer:
def packl(lnum, padmultiple=1):
"""Packs the lnum (which must be convertable to a long) into a
byte string 0 padded to a multiple of padmultiple bytes in size. 0
means no padding whatsoever, so that packing 0 result in an empty
string. The resulting byte string is the big-endian two's
complement representation of the passed in long."""
if lnum == 0:
return b'\0' * padmultiple
elif lnum < 0:
raise ValueError("Can only convert non-negative numbers.")
s = hex(lnum)[2:]
s = s.rstrip('L')
if len(s) & 1:
s = '0' + s
s = binascii.unhexlify(s)
if (padmultiple != 1) and (padmultiple != 0):
filled_so_far = len(s) % padmultiple
if filled_so_far != 0:
s = b'\0' * (padmultiple - filled_so_far) + s
return s
def unpackl(bytestr):
"""Treats a byte string as a sequence of base 256 digits
representing an unsigned integer in big-endian format and converts
that representation into a Python integer."""
return int(binascii.hexlify(bytestr), 16) if len(bytestr) > 0 else 0
In Python 3.2 the int class has to_bytes and from_bytes functions that can accomplish this much more quickly that the method given above.
Here is a solution calling the Python/C API via ctypes. Currently, it uses NumPy, but if NumPy is not an option, it could be done purely with ctypes.
import numpy
import ctypes
PyLong_AsByteArray = ctypes.pythonapi._PyLong_AsByteArray
PyLong_AsByteArray.argtypes = [ctypes.py_object,
numpy.ctypeslib.ndpointer(numpy.uint8),
ctypes.c_size_t,
ctypes.c_int,
ctypes.c_int]
def packl_ctypes_numpy(lnum):
a = numpy.zeros(lnum.bit_length()//8 + 1, dtype=numpy.uint8)
PyLong_AsByteArray(lnum, a, a.size, 0, 1)
return a
On my machine, this is 15 times faster than your approach.
Edit: Here is the same code using ctypes only and returning a string instead of a NumPy array:
import ctypes
PyLong_AsByteArray = ctypes.pythonapi._PyLong_AsByteArray
PyLong_AsByteArray.argtypes = [ctypes.py_object,
ctypes.c_char_p,
ctypes.c_size_t,
ctypes.c_int,
ctypes.c_int]
def packl_ctypes(lnum):
a = ctypes.create_string_buffer(lnum.bit_length()//8 + 1)
PyLong_AsByteArray(lnum, a, len(a), 0, 1)
return a.raw
This is another two times faster, totalling to a speed-up factor of 30 on my machine.
For completeness and for future readers of this question:
Starting in Python 3.2, there are functions int.from_bytes() and int.to_bytes() that perform the conversion between bytes and int objects in a choice of byte orders.
I suppose you really should just be using numpy, which I'm sure has something or other built in for this. It might also be faster to hack around with the array module. But I'll take a stab at it anyway.
IMX, creating a generator and using a list comprehension and/or built-in summation is faster than a loop that appends to a list, because the appending can be done internally. Oh, and 'lstrip' on a large string has got to be costly.
Also, some style points: special cases aren't special enough; and you appear not to have gotten the memo about the new x if y else z construct. :) Although we don't need it anyway. ;)
from struct import pack as _pack
Q_size = 64
Q_bitmask = (1L << Q_size) - 1L
def quads_gen(a_long):
while a_long:
yield a_long & Q_bitmask
a_long >>= Q_size
def pack_long_big_endian(a_long, pad = 1):
if lnum < 0:
raise RangeError("Cannot use packl to convert a negative integer "
"to a string.")
qs = list(reversed(quads_gen(a_long)))
# Pack the first one separately so we can lstrip nicely.
first = _pack('>Q', qs[0]).lstrip('\x00')
rest = _pack('>%sQ' % len(qs) - 1, *qs[1:])
count = len(first) + len(rest)
# A little math trick that depends on Python's behaviour of modulus
# for negative numbers - but it's well-defined and documented
return '\x00' * (-count % pad) + first + rest
Just wanted to post a follow-up to Sven's answer (which works great). The opposite operation - going from arbitrarily long bytes object to Python Integer object requires the following (because there is no PyLong_FromByteArray() C API function that I can find):
import binascii
def unpack_bytes(stringbytes):
#binascii.hexlify will be obsolete in python3 soon
#They will add a .tohex() method to bytes class
#Issue 3532 bugs.python.org
return int(binascii.hexlify(stringbytes), 16)