Numpy bitwise xor on signed int - python

I am reading in some binary data that is in offset binary format. The signed integers in numpy are in twos compliment so the values are incorrect. To fix the data I need to flip the most significant bit. However, I am getting some unexpected results from the bitwise xor and not entirely sure what is going on.
Example:
data = np.array([1, 7, -1, -8], dtype='i1')
mask = 0b10000000
def print_bin(data, out):
for d, o in zip(data, out):
bin_data =np.binary_repr(d, d.dtype.itemsize*8)
bin_out = np.binary_repr(o, o.dtype.itemsize*8)
print(f'{bin_data} to {bin_out}')
print_bin(data, data ^ mask)
Output:
00000001 to 0000000010000001
00000111 to 0000000010000111
11111111 to 1111111101111111
11111000 to 1111111101111000
It seems numpy is maybe doing some kind of casting before the xor since the output is 16-bit integer. If I use the functional form np.bitwise_xor instead of ^ I get the same results. Interestingly if I do the xor with -mask the values come out correct. Or if I specify the dtype in the functional form.
print_bin(data, np.bitwise_xor(data, mask, dtype='i1'))
print_bin(data, data ^ -mask)
Output:
00000001 to 10000001
00000111 to 10000111
11111111 to 01111111
11111000 to 01111000
Can anyone explain exactly what is happening in the first case?

Your mask = 0b10000000 is an unsigned integer representation:
>>> mask
... 128
This would need 16 bits to represent as a signed integer, hence numpy casts all the ints to 16 bits to accommodate this operation. You are looking for the signed integer that has the binary representation '10000000'. This corresponds to the integer -128.
So mask = -128 should solve your problem! This is also why negating the mask works (-mask = -128 = '10000000' as a signed int.
Casting the mask to a int8 numpy integer will also do the trick:
>>> np.array(0b10000000, dtype='i1')
... array(-128, dtype=int8)

Related

Python NAND function

How can I do a logical NAND on two numbers in python? Simple example. Let's say I have a number (0xFF) and I want a logical NAND with a mask value of 0x5.
number = 0xFF = 0b1111 1111
mask = 0x05 = 0b0000 0101
---------------------------
desired= 0xFA = 0b1111 1010
I'm not reinventing the wheel here, this seems like it should be easily accomplished, but I'm stumped and I cannot find any solutions online. I can loop through the number and do a "not (number & mask)" at each bit position and reassemble the value I want, but that seems like more work than is needed here.
Python integers have arbitrary length, so taking their NAND doesn't really make sense where they have implicit leading 0s. Plus, they're signed, so bitwise NOT (~) returns the two's complement.
What you can do instead is use a NumPy unsigned type, assuming the numbers fit. In this case, uint8:
import numpy as np
number = np.uint8(0xff)
mask = np.uint8(0x05)
result = ~(number & mask)
for n in number, mask, result:
print(f'0x{n:02X} 0b{n:08b}')
Output:
0xFF 0b11111111
0x05 0b00000101
0xFA 0b11111010
Other NumPy sized data types are listed here.

Inverting bits in python

I am doing some calculations and I was wondering how you can invert a byte in python.
For example:
0b11111111 should be flipped to 0b00000000.
I have tried turning into a string and flipping it then but I cant somehow turn it back into a Integer.
Also, I want to add that the ~ does not work, as it simply just makes the same number but negative
bin(0b11010111 ^ 0b11111111) // bit subtraction
# Result
# bin(0b11010111 ^ 0b11111111)
#'0b101000'
# for each bit subtraction with 1 occur
To explain why ~ "dit not work": you want to invert all bits in a byte (8-bits), but the 0b.... value is not a byte, but an int which has more bits than just 8.
~var inverts all bits in an integer. You can clear all bits except the lowest 8 with a mask to get the result you expect:
MASK8 = 0b11111111 # or 255 decimal
v = 0b11101110
inv = ~v & MASK8 # 0b10001

How to transform a list of byte to list of signed short

I tried something like :
import array
data = self._data.read(size)
samples = array.array('h')
it = iter(data)
grouped_data = zip(it, it)
for d in grouped_data:
samples.append(d[0] | d[1] << 8)
return samples
data is a list[] where values are between 0 and 255.
I need to transform so that it becomes a list of signed short.
I tried with an array of signed short type but sometimes it tries to insert values that are "bigger" than signed short because it's not handling the case where the number is supposed to be negative.
Maybe numpy has something that could help here?
What you almost certainly wanted here is frombytes:
>>> b = b'\1\2\3\4\xff\xff'
>>> a = array.array('h')
>>> a.frombytes(b)
>>> a
array('h', [513, 1027, -1])
Or, if that's the wrong endianness?
>>> a.byteswap()
>>> a
array('h', [258, 772, -1])
You could fix your bit-twiddling math to do it manually, but why do it the hard way? (If you already know bit-twiddling instinctively because you've been coding in C since you were in kindergarten, then it's not the hard way—but then you wouldn't be asking this question.)1
However, since you mentioned numpy, you probably don't want to use array at all:
>>> np.frombuffer(b, dtype=np.int16)
array([ 513, 1027, -1], dtype=int16)
This is doing the exact same thing as above, and even the storage ends up byte for byte equivalent, but you now have a numpy array instead of an array array, and a numpy array is a lot more useful.
1. If you're curious, the simplest way to think about it is to just be explicit: To reinterpret an unsigned int as a (2's complement) signed int, you just reinterpret the MSB as signed and leave the other bytes alone. To reinterpret an unsigned byte as a (2's complement) signed byte, you just subtract 0x100 if it's >=0x80. So d[1]-256 if d[1]>128 else d[1] would do it. Not the shortest or most efficient code, but probably the easiest to understand.
I just found out that I can use
numpy.int16
So I transformed the code to look like :
import array
data = self._data.read(size)
samples = []
it = iter(data)
grouped_data = zip(it, it)
for d in grouped_data:
samples.append(numpy.int16(d[0] | d[1] << 8))
return samples

In Python, how can I convert a RGB array to binary and get the most significant bit?

Im using scipy to read a image and extract the RGB array like this
img_b = misc.imread('google.png')
img_b_blue = img_b[:, :, 0]
img_b_blue = img_b_blue.ravel()
Now I need to convert the img_b_blue array to binary and get the most significant bit.
I can convert using map:
img_b_blue_bin = map(bin, img_b_blue)
But it comes as string in the format '0b11110001'.
Theres a way to using map and convert to binary without the 'b'? And how can I get the most significant bit?
You can get the most significant bit by right-shifting 7 bits. The result is an integer array.
img_b_blue_bin = img_b_blue >> 7
Alternatively, probably clearer in your use case, is compare with 128. Higher = white, lower = black. The result is a boolean array.
img_b_blue_bin = img_b_blue >= 128
Similarly, the (n+1)-th least significant bit can be found using (x >> n) & 1, e.g. the 2nd most significant bit:
img_b_blue_2nd_msb = (img_b_blue >> 6) & 1
the least significant bit:
img_b_blue_lsb = img_b_blue & 1
See How am I getting a single bit from an int? for how this works for a single integers.

How do I do a bitwise Not operation in Python?

In order to test building an Xor operation with more basic building blocks (using Nand, Or, and And in my case) I need to be able to do a Not operation. The built-in not only seems to do this with single bits. If I do:
x = 0b1100
x = not x
I should get 0b0011 but instead I just get 0b0. What am I doing wrong? Or is Python just missing this basic functionality?
I know that Python has a built-in Xor function but I've been using Python to test things for an HDL project/course where I need to build an Xor gate. I wanted to test this in Python but I can't without an equivalent to a Not gate.
The problem with using ~ in Python, is that it works with signed integers. This is also the only way that really makes sense unless you limit yourself to a particular number of bits. It will work ok with bitwise math, but it can make it hard to interpret the intermediate results.
For 4 bit logic, you should just subtract from 0b1111
0b1111 - 0b1100 # == 0b0011
For 8 bit logic, subtract from 0b11111111 etc.
The general form is
def bit_not(n, numbits=8):
return (1 << numbits) - 1 - n
Another way to achieve this, is to assign a mask like this (should be all 1's):
mask = 0b1111
Then xor it with your number like this:
number = 0b1100
mask = 0b1111
print(bin(number ^ mask))
You can refer the xor truth table to know why it works.
Python bitwise ~ operator invert all bits of integer but we can't see native result because all integers in Python has signed representation.
Indirectly we can examine that:
>>> a = 65
>>> a ^ ~a
-1
Or the same:
>>> a + ~a
-1
Ther result -1 means all bits are set. But the minus sign ahead don't allow us to directly examine this fact:
>>> bin(-1)
'-0b1'
The solution is simple: we must use unsigned integers.
First way is to import numpy or ctypes modules wich both support unsigned integers. But numpy more simplest using than ctypes (at least for me):
import numpy as np
a = np.uint8(0b1100)
y = ~x
Check result:
>>> bin(x)
'0b1100'
>>> bin(y)
'0b11110011'
And finally check:
>>> x + y
255
Unsigned integer '255' for 8-bits integers (bytes) mean the same as '-1' becouse has all bits set to 1. Make sure:
>>> np.uint8(-1)
255
And another simplest solution, not quite right, but if you want to include additional modules, you can invert all bits with XOR operation, where second argument has all bits are set to 1:
a = 0b1100
b = a ^ 0xFF
This operation will also drop most significant bit of signed integer and we can see result like this:
>>> print('{:>08b}'.format(a))
00001100
>>> print('{:>08b}'.format(b))
11110011
Finally solution contains one more operation and therefore is not optimal:
>>> b = ~a & 0xFF
>>> print('{:>08b}'.format(b))
11110011
Try this, it's called the bitwise complement operator:
~0b1100
The answers here collectively have great nuggets in each one, but all do not scale well with depending on edge cases.
Rather than fix upon an 8-bit mask or requiring the programmer to change how many bits are in the mask, simply create a mask based on input via bit_length():
def bit_not(num):
return num ^ ((1 << num.bit_length()) - 1)
string of binary can be used to preserve the left 0s, since we know that:
bin(0b000101) # '0b101'
bin(0b101) # '0b101'
This function will return string format of the NOT of input number
def not_bitwise(n):
'''
n: input string of binary number (positive or negative)
return: binary number (string format)
'''
head, tail = n.split('b')
not_bin = head+'b'+tail.replace('0','a').replace('1','0').replace('a','1')
return not_bin
Example:
In[266]: not_bitwise('0b0001101')
Out[266]: '0b1110010'
In[267]: int(not_bitwise('0b0001101'), 2)
Out[267]: 114
In[268]: not_bitwise('-0b1010101')
Out[268]: '-0b0101010'
In[269]: int(not_bitwise('-0b1010101'), 2)
Out[269]: -42
The general form given by John La Rooy, can be simplified in this way (python == 2.7 and >=3.1):
def bit_not(n):
return (1 << n.bit_length()) - 1 - n

Categories