We Keep Coding

unpacking a struct

I have a list of arithmetic expressions that I encode and pack into a struct, as follows:
expressions = ["1+12", "16-5", "1-3+4", "12-5", "16+4"] # 5 expressions; but could be any number
s = struct.pack('h', len(expressions)) # Prepend the number of expressions to the struct
for e in expressions:
s += struct.pack(f'h{len(e)}s', len(e), e.encode('utf-8'))
Later, I want to undo this process, and unpack them two bytes a time. But I can't seem to get this right. Here's my unpacking and decoding code:
d = []
for n in range(0, len(expressions), 2): # Iterate over the struct
s = struct.unpack('h', expressions[n])
s = s.decode('utf-8')
d.append(s)
Python is giving me the following error at the unpack call:
TypeError: a bytes-like object is required, not 'int'
I don't understand the reason for the error. I thought that I'd already encoded the elements into Unicode, so they should be bytes-like. My goal is to build up the original list of expressions again. How can I do this?

How to pack a character and a number correctly?

I'm learning about client-server communication in python, and I want to send some packed structures.I want to pack a mathematical sign and a number. I tried like this:
idx = 50
value1 = "<"
value2 = idx
packer = struct.Struct('1s I')
packed_data = packer.pack(*value1, *value2)
But I got the error:
packed_data = packer.pack(*value1, *value2)
TypeError: 'int' object is not iterable
or this error:
packed_data = packer.pack(*value1, *value2)
struct.error: argument for 's' must be a bytes object
If I try like this:
value2 = [idx]
I don't know how to do this correctly.

The first problem is that you are unnecessarily trying to (sequence-)unpack your arguments. The Struct format expects a bytes and an int, and you (almost) already have them.
The second problem is that "<" is a Unicode string, and pack expects bytes instead. You need to properly encode the string first.
packed_data = packer.pack(value1.encode('utf-8'), value2)
The particular encoding you use doesn't matter, as long as you use the same one to unpack the data.
Note that if you did have a Unicode character that couldn't be encoded in one byte, your string format would be wrong. The struct module doesn't handle variable-length strings by itself, so it would probably be simpler to just encode the int by itself and concatenated that with your encoded string.
value =
packed_data = value1.encode('utf-8') + struct.pack("I", value2)

ord function in python2.7 and python 3.4 are different?

I have been running a script where I use the ord() function and for whatever the reason in python 2.7, it accepts the unicode string character just as it requires and outputs an integer.
In python 3.4, this is not so much the case. This is the output of error that is being produced :
Traceback (most recent call last):
File "udpTransfer.py", line 38, in <module>
buf.append(ord(c))
TypeError: ord() expected string of length 1, but int found
When I look in both documentations, the ord function is explained to be doing the same exact thing.
This is the code that I am using for both python versions:
import socket,sys, ast , os, struct
from time import ctime
import time
import csv
# creating the udo socket necessary to receive data
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
ip = '192.168.10.101' #i.p. of our computer
port = 20000 # socket port opened to connect from the matlab udp send data stream
server_address = (ip, port)
sock.bind(server_address) # bind socket
sock.settimeout(2) # sock configuration
sock.setblocking(1)
print('able to bind')
ii = 0
shotNummer = 0
client = ''
Array = []
byte = 8192
filename = time.strftime("%d_%m_%Y_%H-%M-%S")
filename = filename + '.csv'
try :
with open(filename,'wb') as csvfile :
spamwriter = csv.writer(csvfile, delimiter=',',quotechar='|', quoting=csv.QUOTE_MINIMAL)
# spamwriter.writerow((titles))
# as long as data comes in, well take it
while True:
data,client = sock.recvfrom(byte)
buf = []
values = []
for c in data:
# print(type(c))
buf.append(ord(c))
if len(buf) == 4 :
###
Can anyone explain why python3.4 it says that c is an integer, rather than in Python 2.7 where it is actually a string, just as the ord() function requires?

You are passing in an integer to ord() in Python 3. That's because you are iterating over a bytes object in Python 3 (the first element in the tuple return value from socket.recvfrom()):
>>> for byte in b'abc':
... print(byte)
...
97
98
99
From the bytes type documentation:
While bytes literals and representations are based on ASCII text, bytes objects actually behave like immutable sequences of integers[.]
and
Since bytes objects are sequences of integers (akin to a tuple), for a bytes object b, b[0] will be an integer [...].
In Python 2, socket.recvfrom() produces a str object instead, and iteration over such an object gives new one-character string objects, which indeed need to be passed to ord() to be converted to an integer.
You could instead use a bytearray() here to get the same integer sequence in both Python 2 and 3:
for c in bytearray(data):
# c is now in integer in both Python 2 and 3
You don't need to use ord() at all in that case.

I think the difference is that in Python 3 the sock.recvfrom(...) call returns bytes while Python 2.7 recvfrom returns a string. So ord did not change but what is being passed to ord has changed.
Python 2.7 recvfrom
Python 3.5 recvfrom

How to convert a sha256 object to integer and pack it to bytearray in python?

I want to convert a hash256 object to a 32-byte integer first, and then pack it into a bytearray.
>>> import hashlib
>>> hashobj = hashlib.sha256('something')
>>> val_hex = hashobj.hexdigest()
>>> print val_hex
3fc9b689459d738f8c88a3a48aa9e33542016b7a4052e001aaa536fca74813cb
>>> print len(val_hex)
64
The hex string is 64-byte instead of 32-byte, which is not what I want.
>>> val = hashobj.digest()
>>> print val
?ɶ?E?s????????5Bkz#R???6??H?
>>> print len(val)
32
This is a 32-byte string and I want to convert it to a 32-byte integer.
It gave me an error message when I try:
>>> val_int = int(val, 10)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '?\xc9\xb6\x89E\x9ds\x8f\x8c\x88\xa3\xa4\x8a\xa9\xe35B\x01kz#R\xe0\x01\xaa\xa56\xfc\xa7H\x13\xcb'
What should I do to get my int_val?
And how can I use struct to pack it (32-byte) to a bytearray? I found the longest format in python struct document is 'Q' which is only 8-byte.
Thank you very much.

The simplest way in Python 2 to get the integer value of the SHA-256 digest is via the hexdigest. Alternatively, you can loop over the bytearray constructed from the binary digest. Both methods are illustrated below.
import hashlib
hashobj = hashlib.sha256('something')
val_hex = hashobj.hexdigest()
print val_hex
# Build bytearray from binary digest
val_bytes = bytearray(hashobj.digest())
print ''.join(['%02x' % byte for byte in val_bytes])
# Get integer value of digest from the hexdigest
val_int = int(val_hex, 16)
print '%064x' % val_int
# Get integer value of digest from the bytearray
n = 0
for byte in val_bytes:
n = n<<8 | byte
print '%064x' % n
output
3fc9b689459d738f8c88a3a48aa9e33542016b7a4052e001aaa536fca74813cb
3fc9b689459d738f8c88a3a48aa9e33542016b7a4052e001aaa536fca74813cb
3fc9b689459d738f8c88a3a48aa9e33542016b7a4052e001aaa536fca74813cb
3fc9b689459d738f8c88a3a48aa9e33542016b7a4052e001aaa536fca74813cb
In Python 3, we can't pass a plain text string to the hashlib hash function, we must pass a bytes string or a bytearray, eg
b'something'
or
'something'.encode('utf-8')
or
bytearray('something', 'utf-8')
We can simplify the second version to
'something'.encode()
since UTF-8 is the default encoding for str.encode (and bytes.decode()).
To perform the conversion to int, any of the above techniques can be used, but we also have an additional option: the int.from_bytes method. To get the correct integer we need to tell it to interpret the bytes as a big-endian number:
import hashlib
hashobj = hashlib.sha256(b'something')
val = int.from_bytes(hashobj.digest(), 'big')
print('%064x' % val)
output
3fc9b689459d738f8c88a3a48aa9e33542016b7a4052e001aaa536fca74813cb

The point of a bytearray is not to fit the whole content in a single cell. That's why cells are only 1 byte big.
And .digest() returns a byte string, so you are fine just using it immediately:
>>> import hashlib
>>> hashobj = hashlib.sha256('something')
>>> val = hashobj.digest()
>>> print bytearray(val)
?ɶ�E�s������5Bkz#R���6��H�
>>> print repr(bytearray(val))
bytearray(b'?\xc9\xb6\x89E\x9ds\x8f\x8c\x88\xa3\xa4\x8a\xa9\xe35B\x01kz#R\xe0\x01\xaa\xa56\xfc\xa7H\x13\xcb')

I did it this way
import hashlib
x = 'input'
hash = int.from_bytes(hashlib.sha256(x.encode('utf-8')).digest(), 'big')
print(my_hash)
# 91106456816457796232999629894661022820411437165637657988648530670402435361824
lets check the size of the hash
print(len("{0:b}".format(my_hash)))
# 256
perfect!

Python how to shorten a uuuid and decode?

I'm trying to create a shortened ID for one of my models using the following method:
_char_map = string.ascii_letters+string.digits
def index_to_char(sequence):
return "".join([_char_map[x] for x in sequence])
def make_short_id(self):
_id = self.id
digits = []
while _id > 0:
rem = _id % 62
digits.append(rem)
_id /= 62
digits.reverse()
return index_to_char(digits)
#staticmethod
def decode_id(string):
i = 0
for c in string:
i = i * 64 + _char_map.index(c)
return i
Where self.id is a uuid i.e. 1c7a2bc6-ca2d-47ab-9808-1820241cf4d4, but I get the following error:
rem = _id % 62
TypeError: not all arguments converted during string formatting
This method only seems to work when the id is an int.
How can I modify the method to shorten a uuuid and decode?
UPDATE:
Thank you for the help. I was trying to find a way create an encode and decode method that took a string, made it shorter then decode it back again. The methods above can never work with a string (uuid) as pointed out,

The % operator is the string formatting or interpolation operator and does not return the remainder in Python when used with strings. It will try to return a formatted string instead.
I'm not sure what your input is, but try converting it using int so you can get the remainder of it.
Edit: I see your input now, not sure why I missed it. Here's one method of converting a UUID to a number:
import uuid
input = "1c7a2bc6-ca2d-47ab-9808-1820241cf4d4"
id = uuid.UUID(input)
id.int
# returns 37852731992078740357317306657835644116L
Not sure what you mean by "shorten", but it looks like you are trying to "base 62 encode" the UUID. If you use the function from this question you will end up with the following:
uuid62 = base62_encode(id.int)
# uuid62 contains 'RJChvUCPWDvJ7BdQKOw7i'
To get the original UUID back:
# Create a UUID again
id = uuid.UUID(int=base62_decode(uuid62))
id.hex
# returns '1c7a2bc6ca2d47ab98081820241cf4d4'
str(id)
# returns '1c7a2bc6-ca2d-47ab-9808-1820241cf4d4'

_id is string
>>> 11 % 2
1
>>> "11" % 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: not all arguments converted during string formatting

I would suggest using base64.urlsafe_b64encode() from the standard library, rather than rolling your own base62_encode() function.
You first need to convert your hex string to a binary string:
binary_id = id.replace("-", "").decode("hex")
This binary string can the be encoded using the afore-mentioned function:
shortened_id = base64.urlsafe_b64encode(binary_id)