We Keep Coding

Fixed point binary and back python

I am fairly new to python is there a fast way to convert decimal to 16-bit fixed-point binary (1-bit sign – 7-bit integer – 8-bit fraction) and back in python.
I would like to manipulate the binary and convert this manipulated binary back to decimal.
Example 1.25 -> 00000001.01000000
Manipulate first fraction part (0->1)
00000001.11000000 -> 1.75
Would really appreciate any help.

If you have N bits in the fractional part then you just need to divide by 2N, since the stored bit pattern is actually the real value multiplied by 2N. Therefore with Q8.8 like in your case you'll have to divide by 256
For your 00000001.01000000 and 00000001.11000000 examples above:
>>> 0b0000000101000000/256.0
1.25
>>> 0b0000000111000000/256.0
1.75

You can use the Binary fractions package.
Example:
>>> from binary_fractions import Binary
>>> b = Binary(15.5)
>>> print(b)
0b1111.1
>>> Binary(1.25).lfill(8).rfill(8)
Binary(00000001.01000000, 0, False)
>>> Binary(1.75).lfill(8).rfill(8)
Binary(00000001.11000000, 0, False)
>>> Binary('0b01.110').lfill(8).rfill(8)
Binary(00000001.11000000, 0, False)
>>> Binary('0b01.110').lfill(8).rfill(8).string()
'00000001.11000000'
It has many more helper functions to manipulate binary strings such as: shift, add, fill, to_exponential, invert...
PS: Shameless plug, I'm the author of this package.

You can use fxpmath to do calculations simplier.
Info about fxpmath:
https://github.com/francof2a/fxpmath
Your example could be solved like:
from fxpmath import Fxp
x = Fxp(1.25, True, 16, 8) # signed=True, n_word=16, n_frac=8
x.bin(frac_dot=True)
out:
'00000001.01000000'
Now you can apply an OR mask to do 0 bit val to 1:
y = x | Fxp('0b01.110', True, 16, 8)
print(y.bin(frac_dot=True))
print(y)
out:
'00000001.11000000'
1.75

numfi can transform floating point number to fixed point with certain word/fraction length, but directly manipulate binary bits is not possible.
You can use bitwise logical operation like and/or/xor to modify bits as workaround, for computation heavy program, bitwise operation should be faster than string evaluation
>>> from numfi import numfi
>>> x = numfi(1.25,1,16,8)
>>> x
numfi([1.25]) s16/8-r/s
>>> x.bin
array(['0000000101000000'], dtype='<U16')
>>> x.bin_
array(['11111110.11000000'], dtype='<U17')
>>> y = x | 0b0000000010000000
>>> y
numfi([1.75]) s16/8-r/s
>>> y.bin_
array(['00000001.11000000'], dtype='<U17')

How do I do a bitwise Not operation in Python?

In order to test building an Xor operation with more basic building blocks (using Nand, Or, and And in my case) I need to be able to do a Not operation. The built-in not only seems to do this with single bits. If I do:
x = 0b1100
x = not x
I should get 0b0011 but instead I just get 0b0. What am I doing wrong? Or is Python just missing this basic functionality?
I know that Python has a built-in Xor function but I've been using Python to test things for an HDL project/course where I need to build an Xor gate. I wanted to test this in Python but I can't without an equivalent to a Not gate.

The problem with using ~ in Python, is that it works with signed integers. This is also the only way that really makes sense unless you limit yourself to a particular number of bits. It will work ok with bitwise math, but it can make it hard to interpret the intermediate results.
For 4 bit logic, you should just subtract from 0b1111
0b1111 - 0b1100 # == 0b0011
For 8 bit logic, subtract from 0b11111111 etc.
The general form is
def bit_not(n, numbits=8):
return (1 << numbits) - 1 - n

Another way to achieve this, is to assign a mask like this (should be all 1's):
mask = 0b1111
Then xor it with your number like this:
number = 0b1100
mask = 0b1111
print(bin(number ^ mask))
You can refer the xor truth table to know why it works.

Python bitwise ~ operator invert all bits of integer but we can't see native result because all integers in Python has signed representation.
Indirectly we can examine that:
>>> a = 65
>>> a ^ ~a
-1
Or the same:
>>> a + ~a
-1
Ther result -1 means all bits are set. But the minus sign ahead don't allow us to directly examine this fact:
>>> bin(-1)
'-0b1'
The solution is simple: we must use unsigned integers.
First way is to import numpy or ctypes modules wich both support unsigned integers. But numpy more simplest using than ctypes (at least for me):
import numpy as np
a = np.uint8(0b1100)
y = ~x
Check result:
>>> bin(x)
'0b1100'
>>> bin(y)
'0b11110011'
And finally check:
>>> x + y
255
Unsigned integer '255' for 8-bits integers (bytes) mean the same as '-1' becouse has all bits set to 1. Make sure:
>>> np.uint8(-1)
255
And another simplest solution, not quite right, but if you want to include additional modules, you can invert all bits with XOR operation, where second argument has all bits are set to 1:
a = 0b1100
b = a ^ 0xFF
This operation will also drop most significant bit of signed integer and we can see result like this:
>>> print('{:>08b}'.format(a))
00001100
>>> print('{:>08b}'.format(b))
11110011
Finally solution contains one more operation and therefore is not optimal:
>>> b = ~a & 0xFF
>>> print('{:>08b}'.format(b))
11110011

Try this, it's called the bitwise complement operator:
~0b1100

The answers here collectively have great nuggets in each one, but all do not scale well with depending on edge cases.
Rather than fix upon an 8-bit mask or requiring the programmer to change how many bits are in the mask, simply create a mask based on input via bit_length():
def bit_not(num):
return num ^ ((1 << num.bit_length()) - 1)

string of binary can be used to preserve the left 0s, since we know that:
bin(0b000101) # '0b101'
bin(0b101) # '0b101'
This function will return string format of the NOT of input number
def not_bitwise(n):
'''
n: input string of binary number (positive or negative)
return: binary number (string format)
'''
head, tail = n.split('b')
not_bin = head+'b'+tail.replace('0','a').replace('1','0').replace('a','1')
return not_bin
Example:
In[266]: not_bitwise('0b0001101')
Out[266]: '0b1110010'
In[267]: int(not_bitwise('0b0001101'), 2)
Out[267]: 114
In[268]: not_bitwise('-0b1010101')
Out[268]: '-0b0101010'
In[269]: int(not_bitwise('-0b1010101'), 2)
Out[269]: -42

The general form given by John La Rooy, can be simplified in this way (python == 2.7 and >=3.1):
def bit_not(n):
return (1 << n.bit_length()) - 1 - n

How to shift bits in a 2-5 byte long bytes object in python?

I am trying to extract data out of a byte object. For example:
From b'\x93\x4c\x00' my integer hides from bit 8 to 21.
I tried to do bytes >> 3 but that isn't possible with more than one byte.
I also tried to solve this with struct but the byte object must have a specific length.
How can I shift the bits to the right?

Don't use bytes to represent integer values; if you need bits, convert to an int:
value = int.from_bytes(your_bytes_value, byteorder='big')
bits_21_to_8 = (value & 0x1fffff) >> 8
where the 0x1fffff mask could also be calculated with:
mask = 2 ** 21 - 1
Demo:
>>> your_bytes_value = b'\x93\x4c\x00'
>>> value = int.from_bytes(your_bytes_value, byteorder='big')
>>> (value & 0x1fffff) >> 8
4940
You can then move back to bytes with the int.to_bytes() method:
>>> ((value & 0x1fffff) >> 8).to_bytes(2, byteorder='big')
b'\x13L'

As you have a bytes string and you want to strip the right-most eight bits (i.e. one byte), you can simply it from the bytes string:
>>> b'\x93\x4c\x00'[:-1]
b'\x93L'
If you want to convert that then to an integer, you can use Python’s struct to unpack it. As you correctly said, you need a fixed size to use structs, so you can just pad the bytes string to add as many zeros as you need:
>>> data = b'\x93\x4c\x00'
>>> data[:-1]
b'\x93L'
>>> data[:-1].rjust(4, b'\x00')
b'\x00\x00\x93L'
>>> struct.unpack('>L', data[:-1].rjust(4, b'\x00'))[0]
37708
Of course, you can also convert it first, and then shift off the 8 bits from the resulting integer:
>>> struct.unpack('>Q', data.rjust(8, b'\x00'))[0] >> 8
37708
If you want to make sure that you don’t actually interpret more than those 13 bits (bits 8 to 21), you have to apply the bit mask 0x1FFF of course:
>>> 37708 & 0x1FFF
4940
(If you need big-endianness instead, just use <L or <Q respectively.)
If you are really counting the bits from left to right (which would be unusual but okay), then you can use that padding technique too:
>>> struct.unpack('>Q', data.ljust(8, b'\x00'))[0] >> 43
1206656
Note that we’re adding the padding to the other side, and are shifting it by 43 bits (your 3 bits plus 5 bytes for the padded data we won’t need to look at)

Another approach that works for arbitrarily long byte sequences is to use the bitstring library which allows for bitwise operations on bitstrings e.g.
>>> import bitstring
>>> bitstring.BitArray(bytes=b'\x93\x4c\x00') >> 3
BitArray('0x126980')

You could convert your bytes to an integer then multiply or divide by powers of two to accomplish the shifting

Interpreting 5bit subsets within Packed Binary Data Python

I have been having some real trouble with this for a while. I am receiving a string of binary data in python and I am having trouble unpacking and interpreting only a 5bit subset (not an entire byte) of the data. It seems like whatever method comes to mind just simply fails miserably.
Let's say I have two bytes packed binary data, and I would like to interpret the first 10bits within the 16. How could I convert this to an 2 integers representing 5bits each?

Use bitmasks and bitshifting:
>>> example = 0x1234 # Hexadecimal example; 2 bytes, 4660 decimal.
>>> bin(example) # Show as binary digits
'0b1001000110100'
>>> example & 31 # Grab 5 most significant bits
20
>>> bin(example & 31) # Same, now represented as binary digits
'0b10100'
>>> (example >> 5) & 31 # Grab the next 5 bits (shift right 5 times first)
17
>>> bin(example >> 5 & 31)
'0b10001'
The trick here is to know that 31 is a 5-bit bitmask:
>>> bin(31)
'0b11111'
>>> 0b11111
31
>>> example & 0b11111
20
As you can see you could also just use the 0b binary number literal notation if you find that easier to work with.
See the Python Wiki on bit manipulation for more background info.

Split integer into two concatenated hex strings- Python

I need to transmit a value that is larger than 65535 via two different hex strings so that when the strings are received, they can be concatenated to form the integer again. For example if the value was 70000 then the two strings would be 0x0001 and 0x1170.
I thought it would be as simple as converting the integer to hex then shifting it right by 4 to get the top string and removing all but the last 4 characters for the bottom.
I think I might be struggling with some syntax (fairly new to Python) and probably some of the logic too. Can anyone think of an easy way to do this?
Thanks

Use divmod builtin function:
>>> [hex(x) for x in divmod(70000, 65536)]
['0x1', '0x1170']

Your algorithm can be implemented easily, as in Lev Levitsky's answer:
hex(big)[2:-4], hex(big)[-4:]
However, it will fail for numbers under 65536.
You could fix that, but you're probably better off splitting the number, then converting the two halves into hex, instead of splitting the hex string.
ecatmur's answer is probably the simplest way to do this:
[hex(x) for x in divmod(70000, 65536)]
Or you could translate your "shift right/truncate" algorithm on the numbers like this:
hex(x >> 16), hex(x & 0xFFFF)
If you need these to be strings like '0x0006' rather than '0x6', instead of calling hex on the parts, you can do this:
['%#06x' % (x,) for x in divmod(x, 65536)]
Or, using the more modern string formatting style:
['0x{:04x}'.format(x) for x in divmod(x, 65536)]
But on the other side, you again probably want to undo this by converting to ints first and then shifting and masking the numbers, instead of concatenating the strings. The inverse of ecatmur's answer is:
int(bighalf) * 65536 + int(smallhalf)
The (equivalent) inverse of the shift/mask implementation is:
(int(bighalf) << 16) | int(smallhalf)
And in that case, you don't need the extra 0s on the left.
It's also worth pointing out that none of these algorithms will work if the number can be negative, or greater than 4294967295, but only because the problem is impossible in those cases.

You mean like this?
In [1]: big = 12345678
In [2]: first, second = hex(big)[2:][:-4], hex(big)[2:][-4:]
In [3]: first, second
Out[3]: ('bc', '614e')
In [4]: int(first+second, 16)
Out[4]: 12345678

Being wary of big/little endians, what you could do to keep it simple is:
val = 70000
to_send = '{:08X}'.format(val) # '00011170'
decoded = int('00011170', 16) # 70000
EDIT: to be very clear then...
hex1, hex2 = to_send[:4], to_send[4:] # send these two and on receipt
my_number = int(hex1 + hex2, 16)

for numbers greater than 65536 or for numbers whose with length >=5, you can use slicing:
>>> num=70000
>>> var1=hex(num)[:-4]
>>> var2='0x'+hex(num)[-4:]
>>> integ=int(var1+var2[2:],16)
>>> print(integ)
70000