Related
I have two arrays A and B. A has multiple values (these values can be string or integer or float) and B has values 0 and 1. I need, for each unique value in A, the count of points that coincide with the 1s in B and the 0s in B. Both the counts need to be stored as separate variables.
For example:
A = [1, 1, 3, 2, 2, 1, 1, 3, 3] # input multivalue array; it has three unique values – 1,2,3
B = [0, 0, 0, 1, 1, 1, 0, 1, 0] # input binary array
#Desired result:
countA1_B1 = 1 #for unique value of '1' in A the count of places where there is '1' in B
countA1_B0 = 3 #for unique value of '1' in A the count of places where there is '0' in B
countAno1_B1 = 3 #for unique value of '1' in A the count of places where there is no '1' in A but there is '1' in B
countAno1_B0 = 2 #for unique value of '1' in A the count of places where there is no '1' in A and there is '0' in B
I need this for all the unique values in A. The A array/list would be a raster and hence the unique values will not be known. So the code would first extract the unique values in A and then do the remaining calculations
My approach to solving this (see post previous question:)
Import numpy as np
A = [1, 1, 3, 2, 2, 1, 1, 3, 3] # input array
B = [0, 0, 0, 1, 1, 1, 0, 1, 0] # input binary array
A_arr = np.array(A)
A_unq = np.unique(A_arr)
#code 1
A_masked_arrays = np.array((A_arr[None, :] == A_unq[:, None]).astype(int))
#code 2
# A_masked_arrays = [(A==unique_val).astype(int) for unique_val in
np.unique(A)]
print(A_masked_arrays)
out = {val: arr for val, arr in zip(list(A_unq), list(A_arr))}
#zip() throws error
#TypeError: 'zip' object is not callable.
dict = {}
for i in A_unq:
for j in A_masked_arrays:
dict = i, j
print(dict)
Result obtained:
# from code 1
[[1 1 0 0 0 1 1 0 0]
[0 0 0 1 1 0 0 0 0]
[0 0 1 0 0 0 0 1 1]]
# from code 2
[array([1, 1, 0, 0, 0, 1, 1, 0, 0]), array([0, 0, 0, 1, 1, 0, 0, 0, 0]),
array([0, 0, 1, 0, 0, 0, 0, 1, 1])]
Using dictionary creation I get this result
(1, array([1, 1, 0, 0, 0, 1, 1, 0, 0]))
(1, array([0, 0, 0, 1, 1, 0, 0, 0, 0]))
(1, array([0, 0, 1, 0, 0, 0, 0, 1, 1]))
(2, array([1, 1, 0, 0, 0, 1, 1, 0, 0]))
(2, array([0, 0, 0, 1, 1, 0, 0, 0, 0]))
(2, array([0, 0, 1, 0, 0, 0, 0, 1, 1]))
(3, array([1, 1, 0, 0, 0, 1, 1, 0, 0]))
(3, array([0, 0, 0, 1, 1, 0, 0, 0, 0]))
(3, array([0, 0, 1, 0, 0, 0, 0, 1, 1]))
This is where I am stuck up. From here how to get to the final count of each unique value in A as countA1_B1, countA1_B0, countAno1_B1, countAno1_B0 and so on. Need help with this. Thanks in advance.
Selective use of np.bincount should do the trick
Au, Ai = np.unique(A, return_index = True)
out = np.empty((2, Au.size))
out[0] = np.bincount(Ai, weight = 1-np.array(B), size = Au.size)
out[1] = bp.bincount(Ai, weight = np.array(B), size = Au.size)
outdict = {}
for i in range(Au.size):
for j in [0, 1]:
outdict[(Au(i), j)] = out[j, i]
It's much easier to use pandas to do this kind of groupby operation:
In [11]: import pandas as pd
In [12]: df = pd.DataFrame({"A": A, "B": B})
In [13]: df
Out[13]:
A B
0 1 0
1 1 0
2 3 0
3 2 1
4 2 1
5 1 1
6 1 0
7 3 1
8 3 0
Now you can use groupby:
In [14]: gb = df.groupby("A")["B"]
In [15]: gb.count() # number of As
Out[15]:
A
1 4
2 2
3 3
Name: B, dtype: int64
In [16]: gb.sum() # number of As where B == 1
Out[16]:
A
1 1
2 2
3 1
Name: B, dtype: int64
In [17]: gb.count() - gb.sum() # number of As where B == 0
Out[17]:
A
1 3
2 0
3 2
Name: B, dtype: int64
You can also do this more explicitly and more generally (e.g. if it's not just 0 and 1) with an apply:
In [18]: gb.apply(lambda x: (x == 1).sum())
Out[18]:
A
1 1
2 2
3 1
Name: B, dtype: int64
I need to generate a two-dimensional python list. It would consist of 10 columns, with each column value being either '1' or '0'. Given these conditions, my list needs to be an exhaustive list of all the combinations that can be formed in this way. It'd naturally end up being 1024 rows long (2^10). However, I have no clue where to get started on this. Can anyone help?
So this is how I worked through this problem. First I saw that we wanted to loop over all 1024 combinations of 0s and 1s. This is essentially counting from 0 to 1023 in binary. So I made a for loop from 0 to 1023 and at each iteration, I converted the iteration variable i into binary with format(i, 'b') and then turned it into a list with the list method. In the case of a number like 1, this gets converted into ['1'] but we want to convert that into ['0', '0', '0', '0', '0', '0', '0', '0', '0', '1'] which is what line
4 does. Finally, we append each result into the 'table' variable.
table=[]
for i in range(1024):
binaryRepresentation = list(format(i, 'b'))
finalRepresentation = ['0']*(10-len(binaryRepresentation)) + binaryRepresentation
table.append(finalRepresentation)
You can use combinations from itertools module, that can create a list of tuples, not a list of lists:
from itertools import combinations
# Generate all the combinations of 0 and 1
# in a list of tuples where each tuple is formed by 10 elements
# Which leads to 184756 combinations
gen_list = combinations([0,1]*10, 10)
# remove duplicates
unique_elements = list(set(gen_list))
# len(unique_elements)
# >>> 1024
An overview of the created list:
>>> unique_elements
[
(0, 1, 0, 1, 0, 1, 0, 1, 0, 1)
(0, 1, 0, 1, 0, 1, 0, 1, 0, 0)
...
(1, 1, 1, 0, 1, 0, 0, 1, 1, 1)
]
You should use numpy for this. For array creation, you could do:
import numpy as np
rows, cols = 1024, 10
arr = np.zeros((rows, cols))
Now, for setting certain values to 1 based on your condition, this could be used:
# this should be changed based on your needs
arr[arr > 0] = 1 # set all values greater than 0 to 1
If you need to create an array initialized with random binary data, you could do:
arr = np.random.randint(2, size=(rows,cols))
If performance isn't a concern, you could always count from 0 to 1023, create the string representation of that number in base 2 then convert to a list.
def binary_cols(n_cols):
for i in range(2**n_cols):
k = [int(j) for j in "{:b}".format(i)]
k = [0]*(n_cols - len(k)) + k
yield k
for col in binary_cols(10):
print col
gives
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
[0, 0, 0, 0, 0, 0, 0, 0, 1, 0]
[0, 0, 0, 0, 0, 0, 0, 0, 1, 1]
...
[1, 1, 1, 1, 1, 1, 1, 1, 0, 1]
[1, 1, 1, 1, 1, 1, 1, 1, 1, 0]
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
print(len(list(binary_cols(10))))
1024
edit: I've just noticed that my answer is essentially a duplicate of Saleh Hindi's. Leaving mine here as there is a big enough difference in the specific tools used for the task, but their answer has a better explanation.
Ok, I have had a browse around and I have looking for either a C or python solution to this problem. I would prefer python...although it is my weaker language (of 2 very weak languages).
A set of numbers, such as 0 0 1 7 0 0 3 0 0 4
Find all permutations of the set.
The numbers >0 must stay in that order (NOT POSITION!)
There MUST be a 0 between numbers, a 0 is not required at the start and end of the set though. As long as there is AT LEAST ONE 0 between numbers >0.
So firstly, I thought of just finding all possible permutations and then removing the chaff (checking that if n>0 , !n+1>0) for each permutation and then the first number >0 == 1, 2nd # >0 ==7 etc. etc.
I then stopped and thought that was daft, say there were 12 numbers, that would give 12! permutations. This in the order of 500,000,000 permutations of which I would have to run through again to get rid of chaff.
Say I had 40-50 sets of these number sets to go through, that is a fair whack of time.
Is there a more logical way?
I thought of somehow having python do permutations somehow taking those rules in to account (if n>0, n+1 MUST == 0) and (n=first number, n2=2nd etc.)
An example of a smaller set would be (NOT ALL PERMUTATIONS, but gives idea):
1,2,3,0,0,0,0,0
1,0,2,0,3,0,0,0
0,1,0,2,0,3,0,0
0,0,1,0,2,0,3,0
0,0,1,0,0,2,0,3
0,1,0,0,2,0,3,0
etc. etc.
So 1,2,3 is in order but the "0"s are just shifted about?
Thanks!
Basically you want to reduce the number of combinations you have to compute by grouping things according to their invariants. Since the non-zero numbers must be in a fixed order let's start with that:
1 2 3
Since there must be 0's between them add them in
1 0 2 0 3
Now what you are left with is three 0's to place and you need to figure how many combinations give distinct sequences. Clearly from this example the possible positions you have are: before the 1, between the 1 and 2, between the 2 and 3 and after the 3. You have 4 positions in which to decide how to split up the remaining three 0's. This is a combination problem with repetition, for which the solution here is (3 + 4 - 1) Choose 3 (which is 20).
Hopefully the way that I went through this example problem is enough for you to generalize this to arbitrary sequences, so I will leave that as an exercise to the reader.
def find_permutations(l):
n = [e for e in l if e] # Strip zeros.
# Interspace non-zeros with zeros.
m = [j for i in n for j in (i,0)][:-1]
def fill(m):
if len(m) == len(l):
yield tuple(m)
else:
# Recursively fill with zeros.
for i in range(len(m)+1):
for o in fill(m[:i] + [0] + m[i:]):
yield tuple(o)
return sorted(set(fill(m)))
I think this should cover it. So for instance (in python 3), you could do:
>>> [print(p) for p in find_permutations([1,2,3,0,0,0,0,0])]
(0, 0, 0, 1, 0, 2, 0, 3)
(0, 0, 1, 0, 0, 2, 0, 3)
(0, 0, 1, 0, 2, 0, 0, 3)
(0, 0, 1, 0, 2, 0, 3, 0)
(0, 1, 0, 0, 0, 2, 0, 3)
(0, 1, 0, 0, 2, 0, 0, 3)
(0, 1, 0, 0, 2, 0, 3, 0)
(0, 1, 0, 2, 0, 0, 0, 3)
(0, 1, 0, 2, 0, 0, 3, 0)
(0, 1, 0, 2, 0, 3, 0, 0)
(1, 0, 0, 0, 0, 2, 0, 3)
(1, 0, 0, 0, 2, 0, 0, 3)
(1, 0, 0, 0, 2, 0, 3, 0)
(1, 0, 0, 2, 0, 0, 0, 3)
(1, 0, 0, 2, 0, 0, 3, 0)
(1, 0, 0, 2, 0, 3, 0, 0)
(1, 0, 2, 0, 0, 0, 0, 3)
(1, 0, 2, 0, 0, 0, 3, 0)
(1, 0, 2, 0, 0, 3, 0, 0)
(1, 0, 2, 0, 3, 0, 0, 0)
Was this similar to what you had in mind?
Edit: basically what the function called fill does is insert a zero between each number of the list, and recurse. Whenever enough numbers are recorded in the fill function (list length of recursively generated numbers equals list length of original input) a tuple of numbers is returned.
The only reason for converting to tuples when returning is that the type must be hashable to use in a set, as seen on the last line of the find_permutations function. sorted is for niceness.
I'm trying to represent an integer as a list of bits and left pad it to 8 bits only if the integer is < 128:
Example input: 0x15
Desired output: [0, 0, 0, 1, 0, 1, 0, 1]
I do it in the following way:
input = 0x15
output = deque([int(i) for i in list(bin(input))[2:]])
while len(output) != 8:
output.appendleft(0)
I would like to convert any integer to a binary-list. Pad to 8 only if the number requires less than 8 bits to represent.
Another Example input: 0x715
Desired output: [1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1]
How can I do this both for numbers less then 8 bits and also for larger ones?
For a fixed size of 8 bits:
num = 0x15
out = [1 if num & (1 << (7-n)) else 0 for n in range(8)]
The (1 << (7-n)) creates a single bit mask for a given position, and then bitwise & tests to see if that bit is set in the number. Having n work through 0 to 7 results in all 8 bits in the byte being tested in order.
For arbitrarily sized numbers:
import math
num = 0x715
bits = int(max(8, math.log(num, 2)+1))
out = [1 if num & (1 << (bits-1-n)) else 0 for n in range(bits)]
number = 0x15
output = [int(x) for x in '{:08b}'.format(number)]
'{:08b}'.format(number) represents your number in binary format with 0 padding to 8 digits, then using list comprehension to create a list of bits.
Alternatively, you can use map function:
output = map(int, '{:08b}'.format(0x15))
If you want to use a variable number of bits, here is one way:
width = 8 # 8bit width
output = [int(x) for x in '{:0{size}b}'.format(0x15, size=width)]
output = map(int, '{:0{size}b}'.format(0x15, size=width))
For Python 3, wrap the map(...) call with list() (map returned a list in Python 2 but returns an iterator in 3).
>>> [int(n) for n in bin(0x15)[2:].zfill(8)]
[0, 0, 0, 1, 0, 1, 0, 1]
The slice [2:] is to remove 0b prefix, zfill(8) is to pad zeros on the left.
Solution
Works for any number of bits, faster than the accepted
answer and the current
highest voted answer:
num = 0xAAAA
bit_list = [(num >> shift_ind) & 1
for shift_ind in range(num.bit_length())] # little endian
bit_list.reverse() # big endian
[1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0]
Timings
from timeit import timeit
def accepted_answer(number):
output = [int(x) for x in '{:08b}'.format(number)]
return output
def highest_voted_answer(num):
out = [1 if num & (1 << (7-n)) else 0 for n in range(8)]
return out
def this_answer(num):
bit_list = [(num >> shift_ind) & 1
for shift_ind in range(num.bit_length())] # little endian
bit_list.reverse() # big endian
return bit_list
NUM = 0x15
ITERATIONS = int(1e7)
print(timeit(lambda: accepted_answer(NUM), number=ITERATIONS))
print(timeit(lambda: highest_voted_answer(NUM), number=ITERATIONS))
print(timeit(lambda: this_answer(NUM), number=ITERATIONS))
9.884788331000891
9.262861715000327
6.484631327999523
It's easy to do this with format strings
>>> "{:08b}".format(0x15)
'00010101'
>>> "{:08b}".format(0x151)
'101010001'
>>> "{:08b}".format(0x1511)
'1010100010001'
to convert to a list
>>> [1 if x=='1' else 0 for x in "{:08b}".format(0x15)]
[0, 0, 0, 1, 0, 1, 0, 1]
>>> [1 if x=='1' else 0 for x in "{:08b}".format(0x1511)]
[1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1]
It's likely to be faster using bit twiddling as in #Amber's answer, but then you'll have to check for special cases and end up with quite a bit of code. If utmost performance isn't required, it's safer to build on what you know already works
np.unpackbits(np.frombuffer(number, np.dtype('B')))
You can shift the number x steps to the right and then make a bitwise and the result with 1 to get the bit at position x, do this with list-comprehension and you get your list. If you need to support negative numbers we may need to add a leading zero to the list to ensure that positive numbers doesn't start with a 1:
import math
def bits(n):
# The number of bits we need to represent the number
num_bits = max(8, int(math.log(abs(n), 2)) + 1)
# The bit representation of the number
bits = [ (n >> i) & 1 for i in range(num_bits) ]
bits.reverse()
# Do we need a leading zero?
if n < 0 or bits[0] == 0:
return bits
return [0] + bits
# Examples
for n in (-0x15, 0x15, 128, 255, 256, -256):
print("{: 4} = {}".format(n, bits(n)))
-21 = [1, 1, 1, 0, 1, 0, 1, 1]
21 = [0, 0, 0, 1, 0, 1, 0, 1]
128 = [0, 1, 0, 0, 0, 0, 0, 0, 0]
255 = [0, 1, 1, 1, 1, 1, 1, 1, 1]
256 = [0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
-256 = [1, 0, 0, 0, 0, 0, 0, 0, 0]
from math import ceil
input = 0x15
bin_string = bin(input)[2:]
binary = map(int,bin_string.zfill(int(ceil(len(bin_string)/8.0)*8)))
print(binary)
This will round to nearest multiple of 8 , if u want to round to multiple of 8 only if <128, use a simple if else statement and remove zfill in else
Output for 0x15:
[0, 0, 0, 1, 0, 1, 0, 1]
Output for 0x715:
[0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1]
If you only want to add zeros if less than 128, use this:
input = 0x715
bin_string = bin(input)[2:]
num_bits = (8 if input < 128 else 0)
binary = map(int,bin_string.zfill(num_bits))
print(binary)
Ouput for 0x15:
[0, 0, 0, 1, 0, 1, 0, 1]
Output for 0x715:
[1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1]
I use bitmasks far less than I used to, but when I do use them I often have the need to decompose a value into its parts
def get_Bits(number):
i = 1
list_of_hex = []
while (i <= number):
if ((number & i) > 0):
bitRepresentation = hex(i)
list_of_hex.append(bitRepresentation)
i = i*2
return list_of_hex
you can change the (hex) function to (bin) if you need the binary decomposition
Numpy has a library function, np.unpackbits, which will unpack a uint8 into a bit vector of length 8. Is there a correspondingly fast way to unpack larger numeric types? E.g. uint16 or uint32. I am working on a question that involves frequent translation between numbers, for array indexing, and their bit vector representations, and the bottleneck is our pack and unpack functions.
You can do this with view and unpackbits
Input:
unpackbits(arange(2, dtype=uint16).view(uint8))
Output:
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0]
For a = arange(int(1e6), dtype=uint16) this is pretty fast at around 7 ms on my machine
%%timeit
unpackbits(a.view(uint8))
100 loops, best of 3: 7.03 ms per loop
As for endianness, you'll have to look at http://docs.scipy.org/doc/numpy/user/basics.byteswapping.html and apply the suggestions there depending on your needs.
This is the solution I use:
def unpackbits(x, num_bits):
if np.issubdtype(x.dtype, np.floating):
raise ValueError("numpy data type needs to be int-like")
xshape = list(x.shape)
x = x.reshape([-1, 1])
mask = 2**np.arange(num_bits, dtype=x.dtype).reshape([1, num_bits])
return (x & mask).astype(bool).astype(int).reshape(xshape + [num_bits])
This is a completely vectorized solution that works with any dimension ndarray and can unpack however many bits you want.
I have not found any function for this too, but maybe using Python's builtin struct.unpack can help make the custom function faster than shifting and anding longer uint (note that I am using uint64).
>>> import struct
>>> N = np.uint64(2 + 2**10 + 2**18 + 2**26)
>>> struct.unpack('>BBBBBBBB', N)
(2, 4, 4, 4, 0, 0, 0, 0)
The idea is to convert those to uint8, use unpackbits, concatenate the result. Or, depending on your application, it may be more convenient to use structured arrays.
There is also built-in bin() function, which produces string of 0s and 1s, but I am not sure how fast it is and it requires postprocessing too.
This works for arbitrary arrays of arbitrary uint (i.e. also for multidimensional arrays and also for numbers larger than the uint8 max value).
It cycles over the number of bits, rather than over the number of array elements, so it is reasonably fast.
def my_ManyParallel_uint2bits(in_intAr,Nbits):
''' convert (numpyarray of uint => array of Nbits bits) for many bits in parallel'''
inSize_T= in_intAr.shape
in_intAr_flat=in_intAr.flatten()
out_NbitAr= numpy.zeros((len(in_intAr_flat),Nbits))
for iBits in xrange(Nbits):
out_NbitAr[:,iBits]= (in_intAr_flat>>iBits)&1
out_NbitAr= out_NbitAr.reshape(inSize_T+(Nbits,))
return out_NbitAr
A=numpy.arange(256,261).astype('uint16')
# array([256, 257, 258, 259, 260], dtype=uint16)
B=my_ManyParallel_uint2bits(A,16).astype('uint16')
# array([[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0],
# [1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0],
# [0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0],
# [1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0],
# [0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]], dtype=uint16)