We Keep Coding

Is there a built-in python function to count bit flip in a binary string?

Is there a built-in python function to count bit flip in a binary string? The question I am trying to solve is given a binary string of arbitrary length, how can I return the number of bit flips of the string. For example, the bit flip number of '01101' is 3, and '1110011' is 2, etc.
The way I can come up with to solve this problem is to use a for loop and a counter. However, that seems too lengthy. Is there a way I can do that faster? Or is there a built-in function in python that allows me to do that directly? Thanks for the help!

There is a very fast way to do that without any explicit loops and only using Python builtins: you can convert the string to a binary number, then detect all the bit flips using a XOR-based integer tricks and then convert the integer back to a string to count the number of bit flips. Here is the code:
# Convert the binary string `s` to an integer: "01101" -> 0b01101
n = int(s, 2)
# Build a binary mask to skip the most significant bit of n: 0b01101 -> 0b01111
mask = (1 << (len(s)-1)) - 1
# Check if the ith bit of n is different from the (i+1)th bit of n using a bit-wise XOR:
# 0b01101 & 0b01111 -> 0b1101 (discard the first bit)
# 0b01101 >> 1 -> 0b0110
# 0b1101 ^ 0b0110 -> 0b1011
bitFlips = (n & mask) ^ (n >> 1)
# Convert the integer back to a string and count the bit flips: 0b1011 -> "0b1011" -> 3
flipCount = bin(bitFlips).count('1')
This trick is much faster than other methods since integer operations are very optimized compare to a loop-based interpreted codes or the ones working on iterables. Here are performance results for a string of size 1000 on my machine:
ljdyer's solution: 96 us x1.0
Karl's solution: 39 us x2.5
This solution: 4 us x24.0
If you are working with short bounded strings, then there are even faster ways to count the number of bits set in an integer.

Don't know about a built in function, but here's a one-liner:
bit_flip_count = len([x for x in range(1, len(x0)) if x0[x] != x0[x-1]])

Given a sequence of values, you can find the number of times that the value changes by grouping contiguous values and then counting the groups. There will be one more group than the number of changes (since the elements before the first change are also in a group). (Of course, for an empty sequence, this gives you a result of -1; you may want to handle this case separately.)
Grouping in Python is built-in, via the standard library itertools.groupby. This tool only considers contiguous groups, which is often a drawback (if you want to make a histogram, for example, you have to sort the data first) but in our case is exactly what we want. The overall interface of this tool is a bit complex, but in our case we can use it simply:
from itertools import groupby
def changes_in(sequence):
return len(list(groupby(sequence))) - 1

Split integer into two concatenated hex strings- Python

I need to transmit a value that is larger than 65535 via two different hex strings so that when the strings are received, they can be concatenated to form the integer again. For example if the value was 70000 then the two strings would be 0x0001 and 0x1170.
I thought it would be as simple as converting the integer to hex then shifting it right by 4 to get the top string and removing all but the last 4 characters for the bottom.
I think I might be struggling with some syntax (fairly new to Python) and probably some of the logic too. Can anyone think of an easy way to do this?
Thanks

Use divmod builtin function:
>>> [hex(x) for x in divmod(70000, 65536)]
['0x1', '0x1170']

Your algorithm can be implemented easily, as in Lev Levitsky's answer:
hex(big)[2:-4], hex(big)[-4:]
However, it will fail for numbers under 65536.
You could fix that, but you're probably better off splitting the number, then converting the two halves into hex, instead of splitting the hex string.
ecatmur's answer is probably the simplest way to do this:
[hex(x) for x in divmod(70000, 65536)]
Or you could translate your "shift right/truncate" algorithm on the numbers like this:
hex(x >> 16), hex(x & 0xFFFF)
If you need these to be strings like '0x0006' rather than '0x6', instead of calling hex on the parts, you can do this:
['%#06x' % (x,) for x in divmod(x, 65536)]
Or, using the more modern string formatting style:
['0x{:04x}'.format(x) for x in divmod(x, 65536)]
But on the other side, you again probably want to undo this by converting to ints first and then shifting and masking the numbers, instead of concatenating the strings. The inverse of ecatmur's answer is:
int(bighalf) * 65536 + int(smallhalf)
The (equivalent) inverse of the shift/mask implementation is:
(int(bighalf) << 16) | int(smallhalf)
And in that case, you don't need the extra 0s on the left.
It's also worth pointing out that none of these algorithms will work if the number can be negative, or greater than 4294967295, but only because the problem is impossible in those cases.

You mean like this?
In [1]: big = 12345678
In [2]: first, second = hex(big)[2:][:-4], hex(big)[2:][-4:]
In [3]: first, second
Out[3]: ('bc', '614e')
In [4]: int(first+second, 16)
Out[4]: 12345678

Being wary of big/little endians, what you could do to keep it simple is:
val = 70000
to_send = '{:08X}'.format(val) # '00011170'
decoded = int('00011170', 16) # 70000
EDIT: to be very clear then...
hex1, hex2 = to_send[:4], to_send[4:] # send these two and on receipt
my_number = int(hex1 + hex2, 16)

for numbers greater than 65536 or for numbers whose with length >=5, you can use slicing:
>>> num=70000
>>> var1=hex(num)[:-4]
>>> var2='0x'+hex(num)[-4:]
>>> integ=int(var1+var2[2:],16)
>>> print(integ)
70000

The tilde operator in Python

What's the usage of the tilde operator in Python?
One thing I can think about is do something in both sides of a string or list, such as check if a string is palindromic or not:
def is_palindromic(s):
return all(s[i] == s[~i] for i in range(len(s) / 2))
Any other good usage?

It is a unary operator (taking a single argument) that is borrowed from C, where all data types are just different ways of interpreting bytes. It is the "invert" or "complement" operation, in which all the bits of the input data are reversed.
In Python, for integers, the bits of the twos-complement representation of the integer are reversed (as in b <- b XOR 1 for each individual bit), and the result interpreted again as a twos-complement integer. So for integers, ~x is equivalent to (-x) - 1.
The reified form of the ~ operator is provided as operator.invert. To support this operator in your own class, give it an __invert__(self) method.
>>> import operator
>>> class Foo:
... def __invert__(self):
... print 'invert'
...
>>> x = Foo()
>>> operator.invert(x)
invert
>>> ~x
invert
Any class in which it is meaningful to have a "complement" or "inverse" of an instance that is also an instance of the same class is a possible candidate for the invert operator. However, operator overloading can lead to confusion if misused, so be sure that it really makes sense to do so before supplying an __invert__ method to your class. (Note that byte-strings [ex: '\xff'] do not support this operator, even though it is meaningful to invert all the bits of a byte-string.)

~ is the bitwise complement operator in python which essentially calculates -x - 1
So a table would look like
i ~i
-----
0 -1
1 -2
2 -3
3 -4
4 -5
5 -6
So for i = 0 it would compare s[0] with s[len(s) - 1], for i = 1, s[1] with s[len(s) - 2].
As for your other question, this can be useful for a range of bitwise hacks.

One should note that in the case of array indexing, array[~i] amounts to reversed_array[i]. It can be seen as indexing starting from the end of the array:
[0, 1, 2, 3, 4, 5, 6, 7, 8]
^ ^
i ~i

Besides being a bitwise complement operator, ~ can also help revert a boolean value, though it is not the conventional bool type here, rather you should use numpy.bool_.
This is explained in,
import numpy as np
assert ~np.True_ == np.False_
Reversing logical value can be useful sometimes, e.g., below ~ operator is used to cleanse your dataset and return you a column without NaN.
from numpy import NaN
import pandas as pd
matrix = pd.DataFrame([1,2,3,4,NaN], columns=['Number'], dtype='float64')
# Remove NaN in column 'Number'
matrix['Number'][~matrix['Number'].isnull()]

The only time I've ever used this in practice is with numpy/pandas. For example, with the .isin() dataframe method.
In the docs they show this basic example
>>> df.isin([0, 2])
num_legs num_wings
falcon True True
dog False True
But what if instead you wanted all the rows not in [0, 2]?
>>> ~df.isin([0, 2])
num_legs num_wings
falcon False False
dog True False

I was solving this leetcode problem and I came across this beautiful solution by a user named Zitao Wang.
The problem goes like this for each element in the given array find the product of all the remaining numbers without making use of divison and in O(n) time
The standard solution is:
Pass 1: For all elements compute product of all the elements to the left of it
Pass 2: For all elements compute product of all the elements to the right of it
and then multiplying them for the final answer
His solution uses only one for loop by making use of. He computes the left product and right product on the fly using ~
def productExceptSelf(self, nums):
res = [1]*len(nums)
lprod = 1
rprod = 1
for i in range(len(nums)):
res[i] *= lprod
lprod *= nums[i]
res[~i] *= rprod
rprod *= nums[~i]
return res

Explaining why -x -1 is correct in general (for integers)
Sometimes (example), people are surprised by the mathematical behaviour of the ~ operator. They might reason, for example, that rather than evaluating to -19, the result of ~18 should be 13 (since bin(18) gives '0b10010', inverting the bits would give '0b01101' which represents 13 - right?). Or perhaps they might expect 237 (treating the input as signed 8-bit quantity), or some other positive value corresponding to larger integer sizes (such as the machine word size).
Note, here, that the signed interpretation of the bits 11101101 (which, treated as unsigned, give 237) is... -19. The same will happen for larger numbers of bits. In fact, as long as we use at least 6 bits, and treating the result as signed, we get the same answer: -19.
The mathematical rule - negate, and then subtract one - holds for all inputs, as long as we use enough bits, and treat the result as signed.
And, this being Python, conceptually numbers use an arbitrary number of bits. The implementation will allocate more space automatically, according to what is necessary to represent the number. (For example, if the value would "fit" in one machine word, then only one is used; the data type abstracts the process of sign-extending the number out to infinity.) It also does not have any separate unsigned-integer type; integers simply are signed in Python. (After all, since we aren't in control of the amount of memory used anyway, what's the point in denying access to negative values?)
This breaks intuition for a lot of people coming from a C environment, in which it's arguably best practice to use only unsigned types for bit manipulation and then apply 2s-complement interpretation later (and only if appropriate; if a value is being treated as a group of "flags", then a signed interpretation is unlikely to make sense). Python's implementation of ~, however, is consistent with its other design choices.
How to force unsigned behaviour
If we wanted to get 13, 237 or anything else like that from inverting the bits of 18, we would need some external mechanism to specify how many bits to invert. (Again, 18 conceptually has arbitrarily many leading 0s in its binary representation in an arbitrary number of bits; inverting them would result in something with leading 1s; and interpreting that in 2s complement would give a negative result.)
The simplest approach is to simply mask off those arbitrarily-many bits. To get 13 from inverting 18, we want 5 bits, so we mask with 0b11111, i.e., 31. More generally (and giving the same interface for the original behaviour):
def invert(value, bits=None):
result = ~value
return result if bits is None else (result & ((1 << bits) - 1))
Another way, per Andrew Jenkins' answer at the linked example question, is to XOR directly with the mask. Interestingly enough, we can use XOR to handle the default, arbitrary-precision case. We simply use an arbitrary-sized mask, i.e. an integer that conceptually has an arbitrary number of 1 bits in its binary representation - i.e., -1. Thus:
def invert(value, bits=None):
return value ^ (-1 if bits is None else ((1 << bits) - 1))
However, using XOR like this will give strange results for a negative value - because all those arbitrarily-many set bits "before" (in more-significant positions) the XOR mask weren't cleared:
>>> invert(-19, 5) # notice the result is equal to 18 - 32
-14

it's called Binary One’s Complement (~)
It returns the one’s complement of a number’s binary. It flips the bits. Binary for 2 is 00000010. Its one’s complement is 11111101.
This is binary for -3. So, this results in -3. Similarly, ~1 results in -2.
~-3
Output : 2
Again, one’s complement of -3 is 2.

This is minor usage is tilde...
def split_train_test_by_id(data, test_ratio, id_column):
ids = data[id_column]
in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio))
return data.loc[~in_test_set], data.loc[in_test_set]
the code above is from "Hands On Machine Learning"
you use tilde (~ sign) as alternative to - sign index marker
just like you use minus - is for integer index
ex)
array = [1,2,3,4,5,6]
print(array[-1])
is the samething as
print(array[~1])

how to convert negative integer value to hex in python

I use python 2.6
>>> hex(-199703103)
'-0xbe73a3f'
>>> hex(199703103)
'0xbe73a3f'
Positive and negative value are the same?
When I use calc, the value is FFFFFFFFF418C5C1.

Python's integers can grow arbitrarily large. In order to compute the raw two's-complement the way you want it, you would need to specify the desired bit width. Your example shows -199703103 in 64-bit two's complement, but it just as well could have been 32-bit or 128-bit, resulting in a different number of 0xf's at the start.
hex() doesn't do that. I suggest the following as an alternative:
def tohex(val, nbits):
return hex((val + (1 << nbits)) % (1 << nbits))
print tohex(-199703103, 64)
print tohex(199703103, 64)
This prints out:
0xfffffffff418c5c1L
0xbe73a3fL

Because Python integers are arbitrarily large, you have to mask the values to limit conversion to the number of bits you want for your 2s complement representation.
>>> hex(-199703103 & (2**32-1)) # 32-bit
'0xf418c5c1L'
>>> hex(-199703103 & (2**64-1)) # 64-bit
'0xfffffffff418c5c1L'
Python displays the simple case of hex(-199703103) as a negative hex value (-0xbe73a3f) because the 2s complement representation would have an infinite number of Fs in front of it for an arbitrary precision number. The mask value (2**32-1 == 0xFFFFFFFF) limits this:
FFF...FFFFFFFFFFFFFFFFFFFFFFFFF418c5c1
& FFFFFFFF
--------------------------------------
F418c5c1

Adding to Marks answer, if you want a different output format, use
'{:X}'.format(-199703103 & (2**32-1))

For those who want leading zeros for positive numbers, try this:
val = 42
nbits = 16
'{:04X}'.format(val & ((1 << nbits)-1))
Thanks #tm1, for the inspiration!

Best way to overwrite some bits in a particular range

Given a series of bits, what's the best way to overwrite a particular range of them.
For example, given:
0100 1010
Say I want to overwrite the middle 2 bits with 10 to make the result:
0101 0010
What would be the best way of doing this?
At first, I thought I would just shift the overwriting bits I want to the correct position (10000), and then use a bitwise OR. But I realized that while it preserves the other bits, there's no way of specifying which bits I want to actually overwrite.
I was looking into Python's bitarray module, but I just want to double-check that I'm not looking over an extremely simple bitwise operation to do this for me.
Thanks.

This is done by first masking the bits you want to erase (forcing them to zero while preserving the other bits) before applying the bitwise OR.
Use a bitwise AND with the pattern (in this case) 11100111.
If you already have a "positive" version of the pattern (here this would be 00011000), which is easier to generate, you can obtain the "negative" version 11100111 using what is called 1's complement, available as ~ in Python and most languages with a C-like syntax.

a = 0b01001010
a &= 0b11100111
a |= 0b00010000
The and first zeroes the target bits:
01001010
& 11100111
--------
01000010
then the or fills them with the desired data:
01000010
| 00010000
--------
01010010

I see you got excellent answers in the "bit manipulation" vein. If you have to do a lot of this, though, I would still recommend reading the discussion here and links therefrom, and, as that wiki suggests, a few of the packages found this way (BitVector, BitPacket, bitarray) -- readability is important, and experience shows that removing non-obvious inline code from your application-level flow in favor of calls to well-named APIs enhances it.
If you do decide to go with manipulation, automatic generation of the bit-ranges masks given bit-indices is clearly crucial. I would recommend starting with an "atomic bitmask" with just one 1 bit, built by shifting:
>>> bin(1 << 7)
'0b10000000'
as you see, 1 << 7 has a one followed by 7 zeros -- and therefore:
>>> bin((1 << 7) - 1)
'0b1111111'
(1 << 7) - 1 has exactly 7 ones (you need the parentheses because the priority of the shifting operator << is lower than that of the subtraction operator -) as the least significant bits aka LSB (and of course all zeros left of that).
Given an easy way to make "a bitmask with N ones" (as the LSB), making ones with bits N included to M excluded set and all other cleared is easy -- using named functions for extra readability:
>>> def n_lsb(x): return (1 << x) - 1
...
>>> def n_to_m(n, m): return n_lsb(n) & ~ n_lsb(m)
...
>>> bin(n_to_m(7, 3))
'0b1111000'
So, now, to set bits N included to M excluded to a certain pattern, as other answers have shown:
>>> def set_bits(i, n, m, pat): return (i & ~n_to_m(n, m))|(pat<<m)
...
>>> bin(set_bits(511, 7, 3, 0b0101))
'0b110101111'
While this answer does not, per se, allow you to do anything more than the others, I do hope it can help "teach you to fish" (vs. just giving you a fish;-) and thereby help you (and other readers) in the future.
BTW, I would recommend reducing to a minimum the mix of bitwise and arithmetic operations -- in this A the only arithmetic operation I use is that - 1 in n_lsb, a hopefully very clear case; in more general cases, two's complement arithmetic (what ordinary integer arithmetic looks like when looked at from a bitwise point of view) could be tricky.

You can do it in 2 steps:
Create a number with all the 0's you
want to write in that looks like
1111 0111 and use an AND
Create a number with all the 1's
that looks like 0001 0000 and use
an OR
BTW: The first number can easily be created by subtracting a 1 shifted into the position of your zeros from 255. So for example, do 255 - (0000 1000).
a = 255 - (1 << x)
b = 1 << y
result = (source & a) | b
where x and y are the positions of your respective bits.

>>> bin(0b01001010 & 0b11110111 | 0b00010000)
'0b1010010'

This is a small basic set of bitwise functions I made from a couple of answers( Thanks to #Alex Martelli for the great answer).
Just open a new python file call it BITManipulation.py copy this code to it import it from your code and you can access all the functions without any need for working with bits.
Anyway I always suggest to understand how it work deeply.
"""
BitManipulation.py
This file is used to define some basic bit manipulations functions
"""
##func: get_bit
##params: value,n
##return: the N-th bit from value
def get_bit(value, n):
return ((value >> n & 1) != 0)
##func: set_bit
##params: value,n
##return: value with the N-th bit Set to 1
def set_bit(value, n):
return value | (1 << n)
##func: clear_bit
##params: value,n
##return: value with the N-th bit Set to 0
def clear_bit(value, n):
return value & ~(1 << n)
##func: n_lsb
##params: x ( Example 7)
##return: a number with exactly x bits of 1 ( Example 0b1111111 )
def n_lsb(x):
return (1 << x) - 1
##func: n_to_m
##params: n,m (n > m)( Example 5,3 )
##return: a number with bits n -> m set to 1 ( Example 0b00011000 )
def n_to_m(n, m):
return n_lsb(n) & ~ n_lsb(m)
##func: set_bits
##params: x,n,m,pat (n > m)( Example 511,5,3,0b0101 )
##return: sets bits n -> m in x to pat ( Example 0b110101111 )
def set_bits(x, n, m, pat):
return (x & ~n_to_m(n, m))|(pat<<m)