I have an issue with numpy that I can't solve.
I have 3D arrays (x,y,z) filled with 0 and 1.
For instance, one slice in the z axis :
array([[1, 0, 1, 0, 1, 1, 0, 0],
[0, 0, 1, 1, 0, 1, 1, 0],
[1, 0, 1, 1, 0, 0, 0, 1],
[0, 0, 0, 0, 0, 0, 0, 0],
[1, 1, 1, 0, 1, 0, 0, 1],
[1, 0, 0, 0, 0, 1, 0, 1],
[0, 0, 0, 0, 1, 0, 0, 0],
[0, 0, 1, 0, 1, 1, 0, 1]])
And I want this result :
array([[1, 1, 1, 1, 1, 1, 0, 0],
[0, 0, 1, 1, 1, 1, 1, 0],
[1, 1, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 1, 0, 0, 0],
[0, 0, 1, 1, 1, 1, 1, 1]])
That is to say, what I want to do for each slice z is to scan line by line right to left and left to right (x axis) and the first time I have a 1 I want to fill the rest of the line with ones.
Is there an efficient way to compute that ?
Thanks a lot.
Nico !
Accessing NumPy array elements one by one is not very efficient. You may do better with just plain Python lists. They also have an index method which can search for the first entry of the value in the list.
from numpy import *
a = array([[1, 0, 1, 0, 1, 1, 0, 0],
[0, 0, 1, 1, 0, 1, 1, 0],
[1, 0, 1, 1, 0, 0, 0, 1],
[0, 1, 0, 0, 1, 0, 1, 0],
[1, 1, 1, 0, 1, 0, 0, 1],
[1, 0, 0, 0, 0, 1, 0, 1],
[0, 0, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 1, 0, 1, 1, 0, 1]])
def idx_front(ln):
try:
return list(ln).index(1)
except ValueError:
return len(ln) # an index beyond line end
def idx_back(ln):
try:
return len(ln) - list(reversed(ln)).index(1) - 1
except ValueError:
return len(ln) # an index beyond line end
ranges = [ (idx_front(ln), idx_back(ln)) for ln in a ]
for ln, (lo,hi) in zip(a, ranges):
ln[lo:hi] = 1 # attention: destructive update in-place
print "ranges =", ranges
print a
Output:
ranges = [(0, 5), (2, 6), (0, 7), (1, 6), (0, 7), (0, 7), (4, 4), (8, 8), (2, 7)]
[[1 1 1 1 1 1 0 0]
[0 0 1 1 1 1 1 0]
[1 1 1 1 1 1 1 1]
[0 1 1 1 1 1 1 0]
[1 1 1 1 1 1 1 1]
[1 1 1 1 1 1 1 1]
[0 0 0 0 1 0 0 0]
[0 0 0 0 0 0 0 0]
[0 0 1 1 1 1 1 1]]
Actually, this is a basic binary image morphology operation.
You can do it in one step for the entire 3D array using scipy.ndimage.morphology.binary_fill_holes
You just need a slightly different structure element. In a nutshell, you want a structuring element that looks like this for the 2D case:
[[0, 0, 0],
[1, 1, 1],
[0, 0, 0]]
Here's a quick example:
import numpy as np
import scipy.ndimage as ndimage
a = np.array( [[1, 0, 1, 0, 1, 1, 0, 0],
[0, 0, 1, 1, 0, 1, 1, 0],
[1, 0, 1, 1, 0, 0, 0, 1],
[0, 1, 0, 0, 1, 0, 1, 0],
[1, 1, 1, 0, 1, 0, 0, 1],
[1, 0, 0, 0, 0, 1, 0, 1],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0, 0],
[0, 0, 1, 0, 1, 1, 0, 1]])
structure = np.zeros((3,3), dtype=np.int)
structure[1,:] = 1
filled = ndimage.morphology.binary_fill_holes(a, structure)
print filled.astype(np.int)
This yields:
[[1 1 1 1 1 1 0 0]
[0 0 1 1 1 1 1 0]
[1 1 1 1 1 1 1 1]
[0 1 1 1 1 1 1 0]
[1 1 1 1 1 1 1 1]
[1 1 1 1 1 1 1 1]
[0 0 0 0 0 0 0 0]
[0 0 0 0 1 0 0 0]
[0 0 1 1 1 1 1 1]]
The real advantage to this (Other than speed... It will be much faster and more memory efficient than using lists!) is that it will work just as well for 3D, 4D, 5D, etc arrays.
We just need to adjust the structuring element to match the number of dimensions.
import numpy as np
import scipy.ndimage as ndimage
# Generate some random 3D data to match what we want...
x = (np.random.random((10,10,20)) + 0.5).astype(np.int)
# Make the structure (I'm assuming that "z" is the _last_ dimension!)
structure = np.zeros((3,3,3))
structure[1,:,1] = 1
filled = ndimage.morphology.binary_fill_holes(x, structure)
print x[:,:,5]
print filled[:,:,5].astype(np.int)
Here's a slice from the random input 3D array:
[[1 0 1 0 1 1 0 1 0 0]
[1 0 1 1 0 1 0 1 0 0]
[1 0 0 1 0 1 1 1 1 0]
[0 0 0 1 1 0 1 0 0 0]
[1 0 1 0 1 0 0 1 1 0]
[1 0 1 1 0 1 0 0 0 1]
[0 1 0 1 0 0 1 0 1 0]
[0 1 1 0 1 0 0 0 0 1]
[0 0 0 1 1 1 1 1 0 1]
[1 0 1 1 1 1 0 0 0 1]]
And here's the filled version:
[[1 1 1 1 1 1 1 1 0 0]
[1 1 1 1 1 1 1 1 0 0]
[1 1 1 1 1 1 1 1 1 0]
[0 0 0 1 1 1 1 0 0 0]
[1 1 1 1 1 1 1 1 1 0]
[1 1 1 1 1 1 1 1 1 1]
[0 1 1 1 1 1 1 1 1 0]
[0 1 1 1 1 1 1 1 1 1]
[0 0 0 1 1 1 1 1 1 1]
[1 1 1 1 1 1 1 1 1 1]]
The key difference here is that we did this for every slice of the entire 3D array in one step.
After a moments thought, following your description and corner case with all zero rows, this will be still quite straightforward with numpylike:
In []: A
Out[]:
array([[1, 0, 1, 0, 1, 1, 0, 0],
[0, 0, 1, 1, 0, 1, 1, 0],
[1, 0, 1, 1, 0, 0, 0, 1],
[0, 1, 0, 0, 1, 0, 1, 0],
[1, 1, 1, 0, 1, 0, 0, 1],
[1, 0, 0, 0, 0, 1, 0, 1],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0, 0],
[0, 0, 1, 0, 1, 1, 0, 1]])
In []: v= 0< A.sum(1) # work only with rows at least one 1
In []: A_v= A[v, :]
In []: (r, s), a= A_v.nonzero(), arange(v.sum())
In []: se= c_[searchsorted(r, a), searchsorted(r, a, side= 'right')- 1]
In []: for k in a: A_v[k, s[se[k, 0]]: s[se[k, 1]]]= 1
..:
In []: A[v, :]= A_v
In []: A
Out[]:
array([[1, 1, 1, 1, 1, 1, 0, 0],
[0, 0, 1, 1, 1, 1, 1, 0],
[1, 1, 1, 1, 1, 1, 1, 1],
[0, 1, 1, 1, 1, 1, 1, 0],
[1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0, 0],
[0, 0, 1, 1, 1, 1, 1, 1]])
Update:
After some more tinkering, here is a more 'pythonic' implementation and way much simpler, than the above one. So, the following lines:
for k in xrange(A.shape[0]):
m= A[k].nonzero()[0]
try: A[k, m[0]: m[-1]]= 1
except IndexError: continue
are quite straightforward ones. And they'll perform very well, indeed.
I can't think of a more efficient way than what you describe:
For every line
Scan line from the left until you find a 1.
If no 1 is find continue with next line.
Otherwise scan from the right to find the last 1 in the line.
Fill everything in the current line between the positions from 1. and 3. with 1s.
Related
rgb_list = []
int_list = [1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1]
for num in range(0, len(int_list)-3, 3):
rgb_list.append(received_int[num:num+3])
received_array = np.array(rgb_list)
print(received_array)
received_array_2d = np.ndarray.reshape(received_array, (5, 2))
print(received_array_2d)
So up until received_array, everything was fine, but when I try to reshape it into a 2D array, I get an error code, I assume it's because numpy is considering each integer individually, not the arrays.
ValueError: cannot reshape array of size 30 into shape (5,2)
the output of print(received_array) is
[[1 0 0]
[1 0 0]
[1 1 0]
[1 0 0]
[1 1 1]
[0 0 1]
[0 1 0]
[1 0 1]
[0 1 0]
[0 1 1]]
I want to get a 2D array that resembles this
[[1 0 0] [1 0 0] [1 1 0] [1 0 0] [1 1 1]
[0 0 1] [0 1 0] [1 0 1] [0 1 0] [0 1 1]]
How would I go about doing that?
If you are using numpy arrays, use numpy methods: reshape is appropriate here.
You first need to trim your array to a multiple of the expected dimensions:
int_list = np.array([1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1])
X,Y,Z = 2,5,3
int_list[:X*Y*Z].reshape((2,5,3))
output:
array([[[1, 0, 0], [1, 0, 0], [1, 1, 0], [1, 0, 0], [1, 1, 1]],
[[0, 0, 1], [0, 1, 0], [1, 0, 1], [0, 1, 0], [0, 1, 1]],
])
I want to make matrix like below using numpy
matrix_example = [[1, 1, 1, 1, 1, 1, 1, 1, 1],
[1, 0, 0, 0, 0, 0, 0, 0, 1],
[1, 0, 1, 1, 1, 1, 1, 0, 1],
[1, 0, 1, 0, 0, 0, 1, 0, 1],
[1, 0, 1, 0, 1, 0, 1, 0, 1],
[1, 0, 1, 0, 0, 0, 1, 0, 1],
[1, 0, 1, 1, 1, 1, 1, 0, 1],
[1, 0, 0, 0, 0, 0, 0, 0, 1],
[1, 1, 1, 1, 1, 1, 1, 1, 1]]
my Idea is using np.where but It doesn't work well..
I want hint about generate matrix like that.
my second idea is
make 9 by 9 matrix fill with zero using numpy.zeros([9, 9])
change 0 to 1 where index is include 0, 2, 4.
a2D = np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1],[1, 0, 0, 0, 0, 0, 0, 0, 1],[1, 0, 1, 1, 1, 1, 1, 0, 1],[1, 0, 1, 0, 0, 0, 1, 0, 1],[1, 0, 1, 0, 1, 0, 1, 0, 1],[1, 0, 1, 0, 0, 0, 1, 0, 1],[1, 0, 1, 1, 1, 1, 1, 0, 1],[1, 0, 0, 0, 0, 0, 0, 0, 1],[1, 1, 1, 1, 1, 1, 1, 1, 1]])
try this
you can use np.ones and np.zeros to do it like:
first_mat = np.ones([9, 9])
second_mat = np.zeros([7, 7])
third_mat = np.ones([5, 5])
forth_mat = np.zeros([3, 3])
first_mat[1:-1, 1:-1] = second_mat
first_mat[2:-2, 2:-2] = third_mat
first_mat[3:-3, 3:-3] = forth_mat
first_mat[4:-4, 4:-4] = 1
and this will give you your output, it maybe not the easiest way, but I hope it can help, and of course first_mat is the maxrix you need
There's already a np.matrix function that makes what you probably want
For you example, it should be as easy as
my_matrix = np.matrix(matrix_example)
Have a look at the official documentation for further info :)
https://numpy.org/doc/stable/reference/generated/numpy.matrix.html
Inspired by Mohamed Yahya's answer and generalizing it to any number of "squares":
import numpy as np
def cool_matrix(squares):
final_matrix = np.zeros((squares * 2 - 1, squares * 2 - 1), dtype=np.int)
for square in range(squares, 0, -1):
square_dimensions = (square * 2 - 1, square * 2 - 1)
if square % 2 == 0:
curr_square = np.zeros(square_dimensions, dtype=np.int)
else:
curr_square = np.ones(square_dimensions, dtype=np.int)
offset = squares + square - 1
final_matrix[-offset:offset, -offset:offset] = curr_square
return final_matrix
print(cool_matrix(5))
output is:
[[1 1 1 1 1 1 1 1 1]
[1 0 0 0 0 0 0 0 1]
[1 0 1 1 1 1 1 0 1]
[1 0 1 0 0 0 1 0 1]
[1 0 1 0 1 0 1 0 1]
[1 0 1 0 0 0 1 0 1]
[1 0 1 1 1 1 1 0 1]
[1 0 0 0 0 0 0 0 1]
[1 1 1 1 1 1 1 1 1]]
I've got three puzzle pieces defined as a number of arrays, 7x7, in a following manner:
R3LRU = pd.DataFrame([
[1, 1, 1, 1, 1, 1, 1],
[1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1]
])
I am trying to join them by the following rules: 1111111 can be joined with 1000001, 1000001 can be joined with 1000001, but 1111111 cannot be joined with 1111111. Better illustration will be the following:
I have tried using pd.concat function, but it just glues them together instead of joining by sides, like this:
Or, in terms of code output, like this:
0 1 2 3 4 5 6 0 1 2 3 4 5 6 0 1 2 3 4 5 6
0 1 1 1 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1
1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 0 0
2 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 0 0
3 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 0 0
4 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 0 0
5 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 0 0
6 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
I suppose I would like to join by columns 6 and 0, or rows 6 and 0
How can I define "joining" sides, so that the pieces would join through the proposed rules?
I take it you want to concatenate if the last column and first columns match and then "overlap" both parts. I dont think, pandas is a good fit for this problem as you only need values, no columns or basically any features you would use pandas for.
I would recommend simple numpy arrays. Then you could do something like
In [1]: import numpy as np
In [2]: R3LRU = np.array([
...: [1, 1, 1, 1, 1, 1, 1],
...: [1, 0, 0, 0, 0, 0, 1],
...: [1, 0, 0, 0, 0, 0, 1],
...: [1, 0, 0, 0, 0, 0, 1],
...: [1, 0, 0, 0, 0, 0, 1],
...: [1, 0, 0, 0, 0, 0, 1],
...: [1, 0, 0, 0, 0, 0, 1]
...: ])
In [3]: R3LRU
Out[3]:
array([[1, 1, 1, 1, 1, 1, 1],
[1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1]])
Get the last column of the first part and the first column of the second part
In [4]: R3LRU[:,0]
Out[4]: array([1, 1, 1, 1, 1, 1, 1])
In [5]: R3LRU[:,-1]
Out[5]: array([1, 1, 1, 1, 1, 1, 1])
Compare them
In [6]: R3LRU[:,0] == R3LRU[:,-1]
Out[6]: array([ True, True, True, True, True, True, True])
In [7]: np.all(R3LRU[:,0] == R3LRU[:,-1])
Out[7]: True
If they are equal, combine them
In [8]: if np.all(R3LRU[:,0] == R3LRU[:,-1]):
...: combined = np.hstack([R3LRU[:,:-1], R3LRU])
In [9]: combined
Out[9]:
array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1]])
Maybe your rules are a bit more complicated than a simple == comparison, but you can just make that if statement more complicated to reflect all rules you have ;)
say n = 4:
then i want to generate the following matrix:
1 0 0 0 1 0 0 1 1 0 1 1
0 1 0 0 1 1 0 0 1 1 0 1
0 0 1 0 0 1 1 0 1 1 1 0
0 0 0 1 0 0 1 1 0 1 1 1
which you can get by appending 3 matrices, (A0, A1, A2) horizontaly
if I'=
0 0 0 1
1 0 0 0
0 1 0 0
0 0 1 0
then:
A0 = I'^0
A1 = I'^1 + I'^0
A2 = I'^2 + I'^1 + I'^0
how can i achieve this efficiently using numpy for any n?
EDIT:
when n = 3 for ex
I' would become
0 0 1
1 0 0
0 1 0
and the desired result would be A0 appended to A1
You can use the modulo operator:
>>> n = 4
>>> i,j,k = np.ogrid[:n, :n-1, :n]
>>> ((j-i+k)%n <= j).reshape(n, -1).view(np.int8)
array([[1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1],
[0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1],
[0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0],
[0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1]], dtype=int8)
Sandwiched between columns of 0s and 1s:
>>> i, jk = np.ogrid[:n, :n*n - n + 2]
>>> j, k = divmod(jk + n - 1, n)
>>> print(((j-i+k-1)%n < j).view(np.int8))
[[0 1 0 0 0 1 0 0 1 1 0 1 1 1]
[0 0 1 0 0 1 1 0 0 1 1 0 1 1]
[0 0 0 1 0 0 1 1 0 1 1 1 0 1]
[0 0 0 0 1 0 0 1 1 0 1 1 1 1]]
Using some list comprehensions (I assume you know that concept, otherwise please google it, it's really helpful in this case) and np.linalg.matrix_power, np.sum and np.concatenate:
In [47]: n = 4
In [48]: np.concatenate(
...: [
...: np.sum(
...: [np.linalg.matrix_power(I, i) for i in range(exp+1)],
...: axis=0 # sum them correct over the axis not the whole data
...: )
...: for exp in range(n-1)
...: ],
...: axis=1 # concat horizontal not vertical
...: )
Out[48]:
array([[1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1],
[0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1],
[0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0],
[0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1]])
This variably works with other n values too:
In [49]: n = 5
In [50]: np.concatenate(
...: [
...: np.sum([np.linalg.matrix_power(I, i) for i in range(exp+1)], axis=0)
...: for exp in range(n-1)
...: ], axis=1)
Out[50]:
array([[1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1],
[0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1],
[0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1],
[0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1]])
In [51]: n = 3
In [52]: np.concatenate(
...: [
...: np.sum([np.linalg.matrix_power(I, i) for i in range(exp+1)], axis=0)
...: for exp in range(n-1)
...: ], axis=1)
Out[52]:
array([[1, 0, 0, 0, 1, 0, 0, 1],
[0, 1, 0, 0, 1, 1, 0, 0],
[0, 0, 1, 0, 0, 1, 1, 0],
[0, 0, 0, 1, 0, 0, 1, 1]])
EDIT:
You can generate your I like requested with eye:
In [68]: n=3
In [69]: I = np.eye(n, k=-1) + np.eye(n, k=n-1)
In [70]: I
Out[70]:
array([[0., 0., 1.],
[1., 0., 0.],
[0., 1., 0.]])
I have a pandas DataFrame with multiple columns.
2u 2s 4r 4n 4m 7h 7v
0 1 1 0 0 0 1
0 1 0 1 0 0 1
1 0 0 1 0 1 0
1 0 0 0 1 1 0
1 0 1 0 0 1 0
0 1 1 0 0 0 1
What I want to do is to convert this pandas.DataFrame into a list like following
X = [
[0, 0, 1, 1, 1, 0],
[1, 1, 0, 0, 0, 1],
[1, 0, 0, 0, 1, 1],
[0, 1, 1, 0, 0, 0],
[0, 0, 0, 1, 0, 0],
[0, 0, 1, 1, 1, 0],
[1, 1, 0, 0, 0, 1]
]
2u 2s 4r 4n 4m 7h 7v are column headings. It will change in different situations, so don't bother about it.
It looks like a transposed matrix:
df.values.T.tolist()
[list(l) for l in zip(*df.values)]
[[0, 0, 1, 1, 1, 0],
[1, 1, 0, 0, 0, 1],
[1, 0, 0, 0, 1, 1],
[0, 1, 1, 0, 0, 0],
[0, 0, 0, 1, 0, 0],
[0, 0, 1, 1, 1, 0],
[1, 1, 0, 0, 0, 1]]
To change Dataframe into list use tolist() function to convert
Let use say i have Dataframe df
to change into list you can simply use tolist() function
df.values.tolist()
You can also change a particular column in to list by using
df['column name'].values.tolist()