I have created the array from a csv using pandas and numpy.
This is my code that convert 2D csv to 3D array:
>>> import pandas as pd
>>> import numpy as npp
>>> df = pd.read_csv("test.csv")
>>> df_mat = df.values
>>> seq_len = 3
>>> data=[]
>>> for index in range(len(df_mat) - seq_len):
... data.append(df_mat[index: index + seq_len])
...
>>> data = np.array(data)
>>> data.shape
(4, 3, 9)
The csv is used is:
input1,input2,input3,input4,input5,input6,input7,input8,output
1,2,3,4,5,6,7,8,1
2,3,4,5,6,7,8,9,0
3,4,5,6,7,8,9,10,-1
4,5,6,7,8,9,10,11,-1
5,6,7,8,9,10,11,12,1
6,7,8,9,10,11,12,13,0
7,8,9,10,11,12,13,14,1
Now I want to get the 3D array back to 2D array format.
Kindly, let me know how I can I do that. Not getting any clue.
Slice on the 0th rows along each each block until the last block and stack with the last one -
np.vstack((data[np.arange(data.shape[0]-1),0],data[-1]))
Output with given sample data -
In [24]: np.vstack((data[np.arange(data.shape[0]-1),0],data[-1]))
Out[24]:
array([[ 1, 2, 3, 4, 5, 6, 7, 8, 1],
[ 2, 3, 4, 5, 6, 7, 8, 9, 0],
[ 3, 4, 5, 6, 7, 8, 9, 10, -1],
[ 4, 5, 6, 7, 8, 9, 10, 11, -1],
[ 5, 6, 7, 8, 9, 10, 11, 12, 1],
[ 6, 7, 8, 9, 10, 11, 12, 13, 0],
[ 7, 8, 9, 10, 11, 12, 13, 14, 1]], dtype=int64)
Or slice 0th rows across all blocks and stack with the last block skipping the first row -
In [28]: np.vstack((data[np.arange(data.shape[0]),0],data[-1,1:]))
Out[28]:
array([[ 1, 2, 3, 4, 5, 6, 7, 8, 1],
[ 2, 3, 4, 5, 6, 7, 8, 9, 0],
[ 3, 4, 5, 6, 7, 8, 9, 10, -1],
[ 4, 5, 6, 7, 8, 9, 10, 11, -1],
[ 5, 6, 7, 8, 9, 10, 11, 12, 1],
[ 6, 7, 8, 9, 10, 11, 12, 13, 0],
[ 7, 8, 9, 10, 11, 12, 13, 14, 1]], dtype=int64)
Related
So I'm trying to start an empty numpy array with a = np.array([]), but when i append other numpy arrays (like [1, 2, 3, 4, 5, 6, 7, 8] and [9, 10, 11, 12, 13, 14, 15, 16] to this array, then the result im basically getting is
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16].
But what i want as result is: [[1, 2, 3, 4, 5, 6, 7, 8], [9, 10, 11, 12, 13, 14, 15, 16]]
IIUC you want to keep adding lists to your np.array. In that case, you can use something like np.vstack to "append" the new lists to the array.
a = np.array([[1, 2, 3],[4, 5, 6]])
np.vstack([a, [7, 8, 9]])
>>> array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
You can also use np.c_[], especially if a and b are already 1D arrays (but it also works with lists):
a = [1, 2, 3, 4, 5, 6, 7, 8]
b = [9, 10, 11, 12, 13, 14, 15, 16]
>>> np.c_[a, b]
array([[ 1, 9],
[ 2, 10],
[ 3, 11],
[ 4, 12],
[ 5, 13],
[ 6, 14],
[ 7, 15],
[ 8, 16]])
It also works "multiple times":
>>> np.c_[np.c_[a, b], a, b]
array([[ 1, 9, 1, 9],
[ 2, 10, 2, 10],
[ 3, 11, 3, 11],
[ 4, 12, 4, 12],
[ 5, 13, 5, 13],
[ 6, 14, 6, 14],
[ 7, 15, 7, 15],
[ 8, 16, 8, 16]])
I need to create 2D np array by the known shape and parameters in this way:
import numpy as np
rows = 9
cols = 8
xrows = 3
xcols = 2
the wanted results:
([[ 1, 1, 4, 4, 7, 7, 10, 10],
[ 1, 1, 4, 4, 7, 7, 10, 10],
[ 1, 1, 4, 4, 7, 7, 10, 10],
[ 2, 2, 5, 5, 8, 8, 11, 11],
[ 2, 2, 5, 5, 8, 8, 11, 11],
[ 2, 2, 5, 5, 8, 8, 11, 11],
[ 3, 3, 6, 6, 9, 9, 12, 12],
[ 3, 3, 6, 6, 9, 9, 12, 12],
[ 3, 3, 6, 6, 9, 9, 12, 12]])
rows%xrows = 0
cols%xcols = 0
rows is the number of rows of the array, cols is the number of columns of the array, xrows is the number of rows in each slice, xcols is the number of column in each slice.
the answer should be general not for this parameters
IIUC, you can use:
((np.arange(rows//xrows*cols//xcols, dtype=int)+1)
.reshape((rows//xrows,cols//xcols), order='F')
.repeat(xrows,0)
.repeat(xcols,1)
)
Output:
array([[ 1, 1, 4, 4, 7, 7, 10, 10],
[ 1, 1, 4, 4, 7, 7, 10, 10],
[ 1, 1, 4, 4, 7, 7, 10, 10],
[ 2, 2, 5, 5, 8, 8, 11, 11],
[ 2, 2, 5, 5, 8, 8, 11, 11],
[ 2, 2, 5, 5, 8, 8, 11, 11],
[ 3, 3, 6, 6, 9, 9, 12, 12],
[ 3, 3, 6, 6, 9, 9, 12, 12],
[ 3, 3, 6, 6, 9, 9, 12, 12]])
I have a list(T) of 6500 images(arrays) that I am using for image classification, and I would like to see how increasing the data affects the accuracy.
So, starting from n=2000 images, I am thinking of having a loop that will add 500(n+=500) images at each iteration till it reaches 6500 and therefore compare the accuracy between 2000, 2500, 3000, ... 6500. I have simplified the problem below by having a list of 20 elements.
lst = [1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0]
My second list (slist) contains the first 9 elements of the first list (lst).
I am trying to add 2 values to slist at each iteration, starting from lst[9:]. I know rather than using append, extend should be used to add multiple values at once. However, I couldn't find a way to do it.
In the following code, one element is added to slist (from lst) at each loop.
lst = [1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0]
slist = lst[:9]
for i in lst[9:]:
slist.append(i)
How can I add 2 or 3 elements simultaneously at each loop? An example output would be:
[1,2,3,4,5,6,7,8,9,0,1]
[1,2,3,4,5,6,7,8,9,0,1,2,3]
[1,2,3,4,5,6,7,8,9,0,1,2,3,4,5]
You could try using extend:
l = [1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0]
slist = l[:9]
for i in l[9:][::2]:
if i == l[9]:
slist.extend(l[9+i: 9+i+1])
else:
slist.extend(l[9+i-1: 9+i+1])
print(slist)
Output:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 0]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 0]
lst=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
16,17,18,19,20]
iter = 9
while True:
print(lst[:iter])
iter+=2
if len(lst) <= iter:
print(lst[:iter])
break
This code does the job
lst=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
16,17,18,19,21]
slist= lst[:9]
s,f=0,2
while True:
slist.extend(lst[9:][s:f])
print(slist)
s+=2
f+=2
if len(slist) >= len(lst):
break
It prints out:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]
I want to implement a vectorized SGD algorithm and would like to generate multiple mini batches at once.
Suppose data = np.arange(0, 100), miniBatchSize=10, n_miniBatches=10 and indices = np.random.randint(0, n_miniBatches, 5) (5 mini batches). What I would like to achieve is
miniBatches = np.zeros(5, miniBatchSize)
for i in range(5):
miniBatches[i] = data[indices[i]: indices[i] + miniBatchSize]
Is there any way to avoid for loop?
Thanks!
It can be done using stride tricks:
from numpy.lib.stride_tricks import as_strided
a = as_strided(data[:n_miniBatches], shape=(miniBatchSize, n_miniBatches), strides=2*data.strides, writeable=False)
miniBatches = a[:, indices].T
# E.g. indices = array([0, 7, 1, 0, 0])
Output:
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]])
This is my first time handling multidimensional arrays and I'm having problems accessing elements. I'm trying to get the red pixels of a picture but just the first 8 elements within the array. Here's the code
import Image
import numpy as np
im = Image.open("C:\Users\Jones\Pictures\1.jpg")
pix = im.load()
r, g, b = np.array(im).T
print r[0:8]
Since you're dealing with images, r is a 2-D array. To get the first 8 pixels in the image, try
r.flatten()[:8]
This will wrap around automatically if the first row has less than 8 pixels.
do you want all rows too? Try this r[:,:8]
only want the first row? Try this r[0,:8]
You can do it like this:
r[0][:8]
Note, however, that this will not work if the first row has less than 8 pixels. To fix that, do this:
from itertools import chain
r = list(chain.from_iterable(r))
r[:8]
or (if you don't want to import an entire module):
r = [val for element in r for val in element]
r[:8]
I think it could be more simple. This example uses a random matrix (this will be your r matrix):
In [7]: from pylab import * # convention
In [8]: r = randint(0,10,(10,10)) # this is your image
In [9]: r
array([[7, 9, 5, 5, 6, 8, 1, 4, 3, 4],
[5, 4, 4, 4, 2, 6, 2, 6, 4, 2],
[1, 4, 9, 9, 2, 6, 1, 9, 0, 6],
[5, 9, 0, 7, 9, 9, 5, 2, 0, 7],
[8, 3, 3, 9, 0, 0, 5, 9, 2, 2],
[5, 3, 7, 8, 8, 1, 6, 3, 2, 0],
[0, 2, 5, 7, 0, 1, 0, 2, 1, 2],
[4, 0, 4, 5, 9, 9, 3, 8, 3, 7],
[4, 6, 9, 9, 5, 9, 3, 0, 5, 1],
[6, 9, 9, 0, 3, 4, 9, 7, 9, 6]])
Then, extract first 8 columns and do something
In [17]: r_8 = r[:,:8] # extract columns
In [18]: r_8
Out[18]:
array([[7, 9, 5, 5, 6, 8, 1, 4],
[5, 4, 4, 4, 2, 6, 2, 6],
[1, 4, 9, 9, 2, 6, 1, 9],
[5, 9, 0, 7, 9, 9, 5, 2],
[8, 3, 3, 9, 0, 0, 5, 9],
[5, 3, 7, 8, 8, 1, 6, 3],
[0, 2, 5, 7, 0, 1, 0, 2],
[4, 0, 4, 5, 9, 9, 3, 8],
[4, 6, 9, 9, 5, 9, 3, 0],
[6, 9, 9, 0, 3, 4, 9, 7]])
In [19]: r_8 = r_8 * 2 # do something
In [20]: r_8
Out[20]:
array([[14, 18, 10, 10, 12, 16, 2, 8],
[10, 8, 8, 8, 4, 12, 4, 12],
[ 2, 8, 18, 18, 4, 12, 2, 18],
[10, 18, 0, 14, 18, 18, 10, 4],
[16, 6, 6, 18, 0, 0, 10, 18],
[10, 6, 14, 16, 16, 2, 12, 6],
[ 0, 4, 10, 14, 0, 2, 0, 4],
[ 8, 0, 8, 10, 18, 18, 6, 16],
[ 8, 12, 18, 18, 10, 18, 6, 0],
[12, 18, 18, 0, 6, 8, 18, 14]])
Now, this is the trick. Replace the first 8 columns in r using hstack:
In [21]: r = hstack((r_8, r[:,8:])) # it replaces the FISRT 8 columns, note the indexing notation
In [22]: r
Out[22]:
array([[14, 18, 10, 10, 12, 16, 2, 8, 3, 4], # it does not touch the last 2 columns
[10, 8, 8, 8, 4, 12, 4, 12, 4, 2],
[ 2, 8, 18, 18, 4, 12, 2, 18, 0, 6],
[10, 18, 0, 14, 18, 18, 10, 4, 0, 7],
[16, 6, 6, 18, 0, 0, 10, 18, 2, 2],
[10, 6, 14, 16, 16, 2, 12, 6, 2, 0],
[ 0, 4, 10, 14, 0, 2, 0, 4, 1, 2],
[ 8, 0, 8, 10, 18, 18, 6, 16, 3, 7],
[ 8, 12, 18, 18, 10, 18, 6, 0, 5, 1],
[12, 18, 18, 0, 6, 8, 18, 14, 9, 6]])
EDIT: as to what DSM pointed out, OP is infact using a numpy array.
i retract my answer as nneonneo's correct