We Keep Coding

Converting lower and upper 8 bits into one value with python

I have pairs like these: (-102,-56), (123, -56). First value from the pairs represents the lower 8 bits and the second value represents the upper 8 bits, both are in signed decimal form. I need to convert these pairs into a single 16 bit values.
I think I was able to convert (-102,-56) pair by:
l = bin(-102 & 0b1111111111111111)[-8:]
u = bin(-56 & 0b1111111111111111)[-8:]
int(u+l,2)
But when I try to do the same with (123, -56) pair I get the following error:
ValueError: invalid literal for int() with base 2: '11001000b1111011'.
I understand that it's due to the different lengths for different values and I need to fill them up to 8 bits.
Am I approaching this completely wrong? What's the best way to do this so it works both on negative and positive values?
UPDATE:
I was able to solve this by:
low_int = 123
up_int = -56
(low_int & 0xFF) | ((up_int & 0xFF) << 8)

You can try to shift the first value 8 bits: try to use the logic described here https://stackoverflow.com/a/1857965/8947333

Just guessing
l, u = -102 & 255, -56 & 255
# shift 8 bits to left
u << 8 + l

Bitwise operations are fine, but not strictly required.
In the most common 2's complement representation for 8 bits:
-1 signed == 255 unsigned
-2 signed == 254 unsigned
...
-127 signed = 129 usigned
-128 signed = 128 usigned
simply the two absolute values always give the sum 256.
Use this to convert negative values:
if b < 0:
b += 256
and then combine the high and low byte:
value = 256 * hi8 + lo8

~ operator in python 3 ouput a amount that I did not expect. What is the reason for this? [duplicate]

I'm a little confused by the ~ operator. Code goes below:
a = 1
~a #-2
b = 15
~b #-16
How does ~ do work?
I thought, ~a would be something like:
0001 = a
1110 = ~a
why not?

You are exactly right. It's an artifact of two's complement integer representation.
In 16 bits, 1 is represented as 0000 0000 0000 0001. Inverted, you get 1111 1111 1111 1110, which is -2. Similarly, 15 is 0000 0000 0000 1111. Inverted, you get 1111 1111 1111 0000, which is -16.
In general, ~n = -n - 1

The '~' operator is defined as:
"The bit-wise inversion of x is defined as -(x+1). It only applies to integral numbers."Python Doc - 5.5
The important part of this sentence is that this is related to 'integral numbers' (also called integers). Your example represents a 4 bit number.
'0001' = 1
The integer range of a 4 bit number is '-8..0..7'. On the other hand you could use 'unsigned integers', that do not include negative number and the range for your 4 bit number would be '0..15'.
Since Python operates on integers the behavior you described is expected. Integers are represented using two's complement. In case of a 4 bit number this looks like the following.
7 = '0111'
0 = '0000'
-1 = '1111'
-8 = '1000'
Python uses 32bit for integer representation in case you have a 32-bit OS. You can check the largest integer with:
sys.maxint # (2^31)-1 for my system
In case you would like an unsigned integer returned for you 4 bit number you have to mask.
'0001' = a # unsigned '1' / integer '1'
'1110' = ~a # unsigned '14' / integer -2
(~a & 0xF) # returns 14
If you want to get an unsigned 8 bit number range (0..255) instead just use:
(~a & 0xFF) # returns 254

It looks like I found simpler solution that does what is desired:
uint8: x ^ 0xFF
uint16: x ^ 0xFFFF
uint32: x ^ 0xFFFFFFFF
uint64: x ^ 0xFFFFFFFFFFFFFFFF

You could also use unsigned ints (for example from the numpy package) to achieve the expected behaviour.
>>> import numpy as np
>>> bin( ~ np.uint8(1))
'0b11111110'

The problem is that the number represented by the result of applying ~ is not well defined as it depends on the number of bits used to represent the original value. For instance:
5 = 101
~5 = 010 = 2
5 = 0101
~5 = 1010 = 10
5 = 00101
~5 = 11010 = 26
However, the two's complement of ~5 is the same in all cases:
two_complement(~101) = 2^3 - 2 = 6
two_complement(~0101) = 2^4 - 10 = 6
two_complement(~00101) = 2^5 - 26 = 6
And given that the two's complement is used to represent negative values, it makes sense to consider ~5 as the negative value, -6, of its complement.
So, more formally, to arrive at this result we have:
flipped zeros and ones (that's equivalent to taking the ones' complement)
taken two's complement
applied negative sign
and if x is a n-digit number:
~x = - two_complement(one_complement(x)) = - two_complement(2^n - 1 - x) = - (2^n - (2^n - 1 - x)) = - (x + 1)

XTEA testvectors in Python

I am trying to test this implementation of the xtea algorithm in Python. The only testvectors I have found are these.
How can I test the output of the algorithm so that I can compare it bytewise?
Which password/key should I choose? Which endian would be best?
(I am on 64 bit xubuntu/x86/little endian)
XTEA
# 64 bit block of data to encrypt
v0, v1 = struct.unpack(endian + "2L", block)
# 128 bit key
k = struct.unpack(endian + "4L", key)
sum, delta, mask = 0L, 0x9e3779b9L, 0xffffffffL
for round in range(n):
v0 = (v0 + (((v1<<4 ^ v1>>5) + v1) ^ (sum + k[sum & 3]))) & mask
sum = (sum + delta) & mask
v1 = (v1 + (((v0<<4 ^ v0>>5) + v0) ^ (sum + k[sum>>11 & 3]))) & mask)
return struct.pack(endian + "2L", v0, v1)
Initial 64 bit test input
# pack 000000 in 64 bit string
byte_string = ''
for c in range(56, -8, -8):
byte_string += chr(000000 >> c & 0xff)
Testvectors (copied from here)
tean values
These are made by starting with a vector of 6 zeroes,
data followed by key, and coding with one cycle then
moving the six cyclically so that n becomes n-1 modulo 6.
We repeat with 2-64 cycles printing at powers of 2 in
hexadecimal. The process is reversed decoding back
to the original zeroes which are printed.
1 0 9e3779b9 0 0 0 0
2 ec01a1de aaa0256d 0 0 0 0
4 bc3a7de2 4e238eb9 0 0 ec01a1de 114f6d74
8 31c5fa6c 241756d6 bc3a7de2 845846cf 2794a127 6b8ea8b8
16 1d8e6992 9a478905 6a1d78c8 8c86d67 2a65bfbe b4bd6e46
32 d26428af a202283 27f917b1 c1da8993 60e2acaa a6eb923d
64 7a01cbc9 b03d6068 62ee209f 69b7afc 376a8936 cdc9e923
1 0 0 0 0 0 0

The C code you linked to seems to assume that a long has 32 bits -- XTEA uses a 64-bit block made of two uint32; the code uses a couple of long and doesn't do anything to handle the overflow which happens when you sum/leftshift (and propagates into later computations).
The python code lets you choose endianness, while the C code treats those numbers as... well, numbers, so if you want to compare them, you need to pick endianness (or if you're lazy, try both and see if one matches :)
Regarding the key, I'm not sure what your problem is, so I'll guess: in case you're not a C programmer, the line static long pz[1024], n, m; is a static declaration, meaning that all those values are implicitly initialized to zero.
Anything else I missed?

Hex string to signed int in Python

How do I convert a hex string to a signed int in Python 3?
The best I can come up with is
h = '9DA92DAB'
b = bytes(h, 'utf-8')
ba = binascii.a2b_hex(b)
print(int.from_bytes(ba, byteorder='big', signed=True))
Is there a simpler way? Unsigned is so much easier: int(h, 16)
BTW, the origin of the question is itunes persistent id - music library xml version and iTunes hex version

In n-bit two's complement, bits have value:
bit 0 = 20
bit 1 = 21
bit n-2 = 2n-2
bit n-1 = -2n-1
But bit n-1 has value 2n-1 when unsigned, so the number is 2n too high. Subtract 2n if bit n-1 is set:
def twos_complement(hexstr, bits):
value = int(hexstr, 16)
if value & (1 << (bits - 1)):
value -= 1 << bits
return value
print(twos_complement('FFFE', 16))
print(twos_complement('7FFF', 16))
print(twos_complement('7F', 8))
print(twos_complement('FF', 8))
Output:
-2
32767
127
-1

import struct
For Python 3 (with comments' help):
h = '9DA92DAB'
struct.unpack('>i', bytes.fromhex(h))
For Python 2:
h = '9DA92DAB'
struct.unpack('>i', h.decode('hex'))
or if it is little endian:
h = '9DA92DAB'
struct.unpack('<i', h.decode('hex'))

Here's a general function you can use for hex of any size:
import math
# hex string to signed integer
def htosi(val):
uintval = int(val,16)
bits = 4 * (len(val) - 2)
if uintval >= math.pow(2,bits-1):
uintval = int(0 - (math.pow(2,bits) - uintval))
return uintval
And to use it:
h = str(hex(-5))
h2 = str(hex(-13589))
x = htosi(h)
x2 = htosi(h2)

This works for 16 bit signed ints, you can extend for 32 bit ints. It uses the basic definition of 2's complement signed numbers. Also note xor with 1 is the same as a binary negate.
# convert to unsigned
x = int('ffbf', 16) # example (-65)
# check sign bit
if (x & 0x8000) == 0x8000:
# if set, invert and add one to get the negative value, then add the negative sign
x = -( (x ^ 0xffff) + 1)

It's a very late answer, but here's a function to do the above. This will extend for whatever length you provide. Credit for portions of this to another SO answer (I lost the link, so please provide it if you find it).
def hex_to_signed(source):
"""Convert a string hex value to a signed hexidecimal value.
This assumes that source is the proper length, and the sign bit
is the first bit in the first byte of the correct length.
hex_to_signed("F") should return -1.
hex_to_signed("0F") should return 15.
"""
if not isinstance(source, str):
raise ValueError("string type required")
if 0 == len(source):
raise valueError("string is empty")
sign_bit_mask = 1 << (len(source)*4-1)
other_bits_mask = sign_bit_mask - 1
value = int(source, 16)
return -(value & sign_bit_mask) | (value & other_bits_mask)

Sum of Digits, properties, hint please

This is the problem:
How many integers 0 ≤ n < 10^18 have the property that the sum of the digits of n equals the sum of digits of 137n?
This solution is grossly inefficient. What am I missing?
#!/usr/bin/env python
#coding: utf-8
import time
from timestrings import *
start = time.clock()
maxpower = 18
count = 0
for i in range(0, 10 ** maxpower - 1):
if i % 9 == 0:
result1 = list(str(i))
result2 = list(str(137 * i))
sum1 = 0
for j in result1:
sum1 += int(j)
sum2 = 0
for j in result2:
sum2 += int(j)
if sum1 == sum2:
print (i, sum1)
count += 1
finish = time.clock()
print ("Project Euler, Project 290")
print ()
print ("Answer:", count)
print ("Time:", stringifytime(finish - start))

First of all, you are to count, not to show the hits.
That is very important. All you have to do is to device an efficient way to count it. Like Jon Bentley wrote in Programming Pearls: "Any methond that considers all permutations of letters for a word is doomed to failure". In fact, I tried in python, as soon as "i" hit 10^9, the system already freezed. 1.5 G memory was consumed. Let alone 10^18. And this also tells us, cite Bentley again, "Defining the problem was about ninety percent of this battle."
And to solve this problem, I can't see a way without dynamic programming (dp). In fact, most of those ridiculously huge Euler problems all require some sort of dp. The theory of dp itself is rather academic and dry, but to implement the idea of dp to solve real problems is not, in fact, the practice is fun and colorful.
One solution to the problem is, we go from 0-9 then 10-99 then 100-999 and so on and extract the signatures of the numbers, summarize numbers with the same signature and deal with all of them as a piece, thus save space and time.
Observation:
3 * 137 = 411 and 13 * 137 = 1781. Let's break the the first result "411" down into two parts: the first two digits "41" and the last digit "1". The "1" is staying, but the "41" part is going to be "carried" to further calculations. Let's call "41" the carry, the first element of the signature. The "1" will stay as the rightest digit as we go on calculating 13 * 137, 23 * 137, 33 * 137 or 43 * 137. All these *3 numbers have a "3" as their rightest digit and the last digit of 137*n is always 1. That is, the difference between this "3" and "1" is +2, call this +2 the "diff" as the second element of the signature.
OK, if we are gonna find a two-digit number with 3 as its last digit, we have to find a digit "m" that satisfies
diff_of_digitsum (m, 137*m+carry) = -2 (1)
to neutralize our +2 diff accumulated earlier. If m could do that, then you know m * 10 + 3, on the paper you write: "m3", is a hit.
For example, in our case we tried digit 1. diff_of_digitsum (digit, 137*digit+carry) = diff_of_digitsum (1, 137*1+41) = -15. Which is not -2, so 13 is not a hit.
Let's see 99. 9 * 137 = 1233. The "diff" is 9 - 3 = +6. "Carry" is 123. In the second iteration when we try to add a digit 9 to 9 and make it 99, we have diff_of_digitsum (digit, 137*digit+carry) = diff_of_digitsum (9, 137*9+123) = diff_of_digitsum (9, 1356) = -6 and it neutralizes our surplus 6. So 99 is a hit!
In code, we just need 18 iteration. In the first round, we deal with the single digit numbers, 2nd round the 2-digit numbers, then 3-digit ... until we get to 18-digit numbers. Make a table before the iterations that with a structure like this:
table[(diff, carry)] = amount_of_numbers_with_the_same_diff_and_carry
Then the iteration begins, you need to keep updating the table as you go. Add new entries if you encounter a new signature, and always update amount_of_numbers_with_the_same_diff_and_carry. First round, the single digits, populate the table:
0: 0 * 137 = 0, diff: 0; carry: 0. table[(0, 0)] = 1
1: 1 * 137 = 137. diff: 1 - 7 = -6; carry: 13. table[(-6, 13)] = 1
2: 2 * 137 = 274. diff: 2 - 7 = -5; carry: 27. table[(-5, 27)] = 1
And so on.
Second iteration, the "10"th digit, we will go over the digit 0-9 as your "m" and use it in (1) to see if it can produce a result that neutralizes the "diff". If yes, it means this m is going to make all those amount_of_numbers_with_the_same_diff_and_carry into hits. Hence counting not showing. And then we can calculate the new diff and carry with this digit added, like in the example 9 has diff 6 and carry 123 but 99 has the diff 9 - 6 ( last digit from 1356) = 3 and carry 135, replace the old table using the new info.
Last comment, be careful the digit 0. It will appear a lot of times in the iteration and don't over count it because 0009 = 009 = 09 = 9. If you use c++, make sure the sum is in unsigned long long and that sort because it is big. Good luck.

You are trying to solve a Project Euler problem by brute force. That may work for the first few problems, but for most problems you need think of a more sophisticated approach.
Since it is IMHO not OK to give advice specific to this problem, take a look at the general advice in this answer.

This brute force Python solution of 7 digits ran for 19 seconds for me:
print sum(sum(map(int, str(n))) == sum(map(int, str(137 * n)))
for n in xrange(0, 10 ** 7, 9))
On the same machine, single core, same Python interpreter, same code, would take about 3170 years to compute for 18 digits (as the problem asked).
See dgg32's answer for an inspiration of a faster counting.