We Keep Coding

Python - Fastest way to strip the trailing zeros from the bit representation of a number

This is the python version of the same C++ question.
Given a number, num, what is the fastest way to strip off the trailing zeros from its binary representation?
For example, let num = 232. We have bin(num) equal to 0b11101000 and we would like to strip the trailing zeros, which would produce 0b11101. This can be done via string manipulation, but it'd probably be faster via bit manipulation. So far, I have thought of something using num & -num
Assuming num != 0, num & -num produces the binary 0b1<trailing zeros>. For example,
num 0b11101000
-num 0b00011000
& 0b1000
If we have a dict having powers of two as keys and the powers as values, we could use that to know by how much to right bit shift num in order to strip just the trailing zeros:
# 0b1 0b10 0b100 0b1000
POW2s = { 1: 0, 2: 1, 4: 2, 8: 3, ... }
def stripTrailingZeros(num):
pow2 = num & -num
pow_ = POW2s[pow2] # equivalent to math.log2(pow2), but hopefully faster
return num >> pow_
The use of dictionary POW2s trades space for speed - the alternative is to use math.log2(pow2).
Is there a faster way?
Perhaps another useful tidbit is num ^ (num - 1) which produces 0b1!<trailing zeros> where !<trailing zeros> means take the trailing zeros and flip them into ones. For example,
num 0b11101000
num-1 0b11100111
^ 0b1111
Yet another alternative is to use a while loop
def stripTrailingZeros_iterative(num):
while num & 0b1 == 0: # equivalent to `num % 2 == 0`
num >>= 1
return num
Ultimately, I need to execute this function on a big list of numbers. Once I do that, I want the maximum. So if I have [64, 38, 22, 20] to begin with, I would have [1, 19, 11, 5] after performing the stripping. Then I would want the maximum of that, which is 19.

There's really no answer to questions like this in the absence of specifying the expected distribution of inputs. For example, if all inputs are in range(256), you can't beat a single indexed lookup into a precomputed list of the 256 possible cases.
If inputs can be two bytes, but you don't want to burn the space for 2**16 precomputed results, it's hard to beat (assuming that_table[i] gives the count of trailing zeroes in i):
low = i & 0xff
result = that_table[low] if low else 8 + that_table[i >> 8]
And so on.
You do not want to rely on log2(). The accuracy of that is entirely up to the C library on the platform CPython is compiled for.
What I actually use, in a context where ints can be up to hundreds of millions of bits:
assert d
if d & 1 == 0:
ntz = (d & -d).bit_length() - 1
d >>= ntz
A while loop would be a disaster in this context, taking time quadratic in the number of bits shifted off. Even one needless shift in that context would be a significant expense, which is why the code above first checks to see that at least one bit needs to shifted off. But if ints "are much smaller", that check would probably cost more than it saves. "No answer in the absence of specifying the expected distribution of inputs."

On my computer, a simple integer divide is fastest:
import timeit
timeit.timeit(setup='num=232', stmt='num // (num & -num)')
0.1088077999993402
timeit.timeit(setup='d = { 1: 0, 2 : 1, 4: 2, 8 : 3, 16 : 4, 32 : 5 }; num=232', stmt='num >> d[num & -num]')
0.13014470000052825
timeit.timeit(setup='import math; num=232', stmt='num >> int(math.log2(num & -num))')
0.2980690999993385

You say you "Ultimately, [..] execute this function on a big list of numbers to get odd numbers and find the maximum of said odd numbers."
So why not simply:
from random import randint
numbers = [randint(0, 10000) for _ in range(5000)]
odd_numbers = [n for n in numbers if n & 1]
max_odd = max(odd_numbers)
print(max_odd)
To do what you say you want to do ultimately, there seems to be little point in performing the "shift right until the result is odd" operation? Unless you want the maximum of the result of that operation performed on all elements, which is not what you stated?
I agree with #TimPeters answer, but if you put Python through its paces and actually generate some data sets and try the various solutions proposed, they maintain their spread for any number of integer size when using Python ints, so your best option is integer division for numbers up to 32-bits, after that see the chart below:
from pandas import DataFrame
from timeit import timeit
import math
from random import randint
def reduce0(ns):
return [n // (n & -n)
for n in ns]
def reduce1(ns, d):
return [n >> d[n & -n]
for n in ns]
def reduce2(ns):
return [n >> int(math.log2(n & -n))
for n in ns]
def reduce3(ns, t):
return [n >> t.index(n & -n)
for n in ns]
def reduce4(ns):
return [n if n & 1 else n >> ((n & -n).bit_length() - 1)
for n in ns]
def single5(n):
while (n & 0xffffffff) == 0:
n >>= 32
if (n & 0xffff) == 0:
n >>= 16
if (n & 0xff) == 0:
n >>= 8
if (n & 0xf) == 0:
n >>= 4
if (n & 0x3) == 0:
n >>= 2
if (n & 0x1) == 0:
n >>= 1
return n
def reduce5(ns):
return [single5(n)
for n in ns]
numbers = [randint(1, 2 ** 16 - 1) for _ in range(5000)]
d = {2 ** n: n for n in range(16)}
t = tuple(2 ** n for n in range(16))
assert(reduce0(numbers) == reduce1(numbers, d) == reduce2(numbers) == reduce3(numbers, t) == reduce4(numbers) == reduce5(numbers))
df = DataFrame([{}, {}, {}, {}, {}, {}])
for p in range(1, 16):
p = 2 ** p
numbers = [randint(1, 2 ** p - 1) for _ in range(4096)]
d = {2**n: n for n in range(p)}
t = tuple(2 ** n for n in range(p))
df[p] = [
timeit(lambda: reduce0(numbers), number=100),
timeit(lambda: reduce1(numbers, d), number=100),
timeit(lambda: reduce2(numbers), number=100),
timeit(lambda: reduce3(numbers, t), number=100),
timeit(lambda: reduce4(numbers), number=100),
timeit(lambda: reduce5(numbers), number=100)
]
print(f'Complete for {p} bit numbers.')
print(df)
df.to_csv('test_results.csv')
Result (when plotted in Excel):
Note that the plot that was previously here was wrong! The code and data were not though. The code has been updated to include #MarkRansom's solution, since it turns out to be the optimal solution for very large numbers (over 4k-bit numbers).

while (num & 0xffffffff) == 0:
num >>= 32
if (num & 0xffff) == 0:
num >>= 16
if (num & 0xff) == 0:
num >>= 8
if (num & 0xf) == 0:
num >>= 4
if (num & 0x3) == 0:
num >>= 2
if (num & 0x1) == 0:
num >>= 1
The idea here is to perform as few shifts as possible. The initial while loop handles numbers that are over 32 bits long, which I consider unlikely but it has to be provided for completeness. After that each statement shifts half as many bits; if you can't shift by 16, then the most you could shift is 15 which is (8+4+2+1). All possible cases are covered by those 5 if statements.

How to fix bit length when doing bitshift right?

I have generated x = 0 - 2047 in hexadecimal (0 - 7ff).
Then, I tried to shift one bit to right, but I want to keep all the value.
for example, for x = 2047 = h7ff.
I want to shift this value to right. can I set fixed bit length, so that even if I shift, it will not cut the 8?
tried:
x = 2047
hex(x)='0x7ff'
hex(x >> 1)='0x3ff'
expecting:
0x3ff8

If you want to add extra digit only if a bit in the result will be chopped off, you can use a simple modulo. Check if the last bit in the value is a 1, and if it is, multiply the number by 0x10 or decimal 16 to add the extra padding. Then bit shift as normal.
hex((x * 0x10 if x % 2 == 1 else x) >> 1)
adding this to a method for reuse:
def rotate(x):
return (x * 0x10 if x % 2 == 1 else x) >> 1
print(hex(x := 0x7ff))
print(hex(x := rotate(x)))
print(hex(x := rotate(x)))
print(hex(x := rotate(x)))
print(hex(x := rotate(x)))
0x7ff
0x3ff8
0x1ffc
0xffe
0x7ff

Using Floor/Ceil to round decimals to integer of 5

I have been using the code below to handily round to 1dp where I needed the .5s to round upwards, e.g. I wanted a value such as 36.25 to round to 36.3, not 36.2.
def my_round(n, ndigits=1):
try:
part = n * 10 ** ndigits
delta = part - int(part)
# always round "away from 0"
if delta >= 0.5 or -0.5 < delta <= 0:
part = math.ceil(part)
else:
part = math.floor(part)
val = part/(10 ** ndigits)
except ValueError:
val = np.nan
return val
I now need to round a different decimal column in a df to the nearest 5 whole numbers. Can anyone help suggest how I can tweak my original code above to do this? I need, for example, a value such as 702.5000 rounding to 705. Using the normal Round() it obviously goes to 700. In the example, all other normal rounding rules should apply e.g.
702.222 to 700,
707.466 to 705,
703.021 to 705

To round to a certain number of decimal places, use this:
def my_round(x: float, round_to: float = 5, dp: int = 0) -> float:
round_to *= 10**dp
x *= 10**dp
return ((x + round_to / 2) // round_to * round_to) / 10**dp
Testing it:
values = [702.500, 702.222, 707.466, 703.021]
targets = [705., 700., 705., 705.]
assert [my_round(x, 5, 0) for x in values] == targets
assert my_round(3.25, 0.1, 1) == 36.3

def my_round(n):
lower = (n//5)*5;
upper = lower+5;
if (n-lower)<(upper-n):
return int(lower)
return int(upper)
print(my_round(703.021))
The above program tries to find out the proper multiple of 5 before and after the given number.
Then it finds the differences between the lower and upper possible number and returns the number with least difference. This is as good as rounding.

The quickest way would be to divide by 5, round and then multiply by 5. It's purely mathematical fact that this will work.
5*round(n/5)
And there's no need for the if statement deciding on when to use floor or ceil since round already defaults to this logic.
EDIT: As pointed out this works for all values except those ending in 2.5 when requiring that this rounds up (in practice many examples won't need this).
As Ashwin's answer accomplishes this already, I'll give a fix to do it on one line without a function definition, though it's not pretty:
5 * (int(x/5) + 1 - round(x/5 % 1))
Or slightly cleaner with ceil:
5*(math.ceil(x/5) - round(x/5 % 1))
exploiting the the fact that round behaves as we would like from above.
Even better, define:
def my_round(x): return int(x + 1) - round(x % 1)
def round5(x): return 5*my_round(x/5)
to 'fix' the round function and create a nice round5 function in 2 lines. This is what I'd do.

You could divide the number you are trying to round by 5, then take its round() and multiply again by five:
num = 707.466
int = num / 5 # returns 141.4932
rounded = round(int) # returns 141
answer = rounded * 5 # 705
Or as a function:
def round_to_five(num):
num = num / 5
num = round(num)
return num * 5

Hope this helps
# my standard point to which rounding shoulding happen
>>> standard_nums = [700, 702, 705, 708]
# example value I am taking
>>> my_num = 702.5
# this is how I would round my number
>>> ret = abs(standard_nums[0] - my_num)
>>> val = standard_nums[0]
>>> for each in standard_nums[1:]:
... if abs(each - my_num) < ret:
... ret = abs(each - my_num)
... val = each
...
>>> ret # min diff I can find
0.5
>>> val # value you are lookin for
702
>>>
## this above has a complexity of O(N)
## where N is the size of your standard number

Assuming that you use Python 3, would the following line do the job ?
5*(x//5)+5*(x%5>=2.5)
It seems to work for x being a scalar, numpy.array or a pandas.series

Binary inversion

I need the cleanest way to invert a specific bit in a number, where the leftmost bit is the LSB.
For example, if I have a function invert(n, b) and I were to execute invert(15, 0), it should invert the zeroth digit from the left. And if I were to execute invert(15, 1) it would invert the first digit from the left, etc.

If you need to invert specific bits of a (integer) number, you can use:
def flipBit (n, b): #n the number, b the bit, 0 = LSB
return n ^ (1 << b)
If you need a string of this number, use bin(x)[2:].
Example:
def flipBit(n, b):
return n ^ (1 << b)
def toBinStr(n):
return bin(n)[2:]
y = 42
print('Number is {}.'.format(toBinStr(y)))
for x in range(8):
print('Flipping bit {} gives {}.'.format (x, toBinStr(flipBit(y, x))))
Your example:
#number that needs inversion
number = '1010'
#bit that needs to be inverted (first digit here)
bit_to_invert = 1
##code here##
inverted = bin(int(number, 2) ^ (1 << (bit_to_invert - 1)))[2:]
#this should output 1011
print inverted

This can be achieved through the modulo operation:
reverse_bin[x] = MOD(x+1,2)

Python >>= help for decimal to binary function

Hello I was writing a decimal to binary function and I found this code that works perfectly:
while n > 0:
b = str(n % 2) + b
n >>= 1
However I do not know what >>= does could you enlighten me?
Many thanks

It's a binary right shift operation. The bits in n are shifted to the right by 1. It's equivalent of saying n = n >> 1.
From BitwiseOperators in python:
x >> y:
Returns x with the bits shifted to the right by y places. This is the same as //'ing x by 2**y.
For instance, assume an integer 4 and let's shift it to the right by 1 places.
# First let's look at what 4 is in binary.
>>> bin(4)[2:].zfill(8) # this pads to 8 bits.
'00000100'
# If you shift all the bits towards the right 1 places the result is
# '00000010', which in turn is 2 in base 10.
>>> 4 >> 1
2
>>> bin(2)[2:].zfill(8)
'00000010'

it's right shift operation. One bit right is equivalent to divide by 2.