I'm trying to figure out what the parameters to the reshape() function are below.
I didn't find anything about pickle having a reshape() method, and import cPickle as pickle, import numpy as np were given in the file, so I'm assuming (maybe a bad assumption) that the reshape function is because of numpy. I found the definition of the reshape method for numpy (also below). However, I can't tell which arguments belong to which parameter.
Because this thing is supposed to load in picture data, I'm guessing 32,32 might be the image size, and would correspond to the newshape parameter?
I don't have a clue what 1000,3 are doing: the term "array_like" for the a parameter is confusing, and I don't know why 4 parameters are given if there's only 3 for the method, or how python would know that 32,32 is one argument, if it really is (why no []?)
Basically, what parameter does each argument (passed in) belong to? And how on earth can it tell? And how did X go from being an object from the pickle load that has numpy methods on it? Is that even possible?
datadict = pickle.load(f)
X = datadict['data']
Y = datadict['labels']
X = X.reshape(10000, 3, 32, 32)
The documentation you've linked is slightly different than what is actually happening, which may explain your confusion. The actual documentation, which is effectively the same function but set up as an object method instead of a library method, is here.
In this case, the (10000, 3, 32, 32) corresponds to the shape of the output array. So your output is actually a 4-dimensional array with shape (10000, 3, 32, 32). I suspect that if this is supposed to be image data, you could have a 32x32 image with RGB values and 1,000 images.
Additionally, pickle stores type information when you store objects, so this is how it knows that the object is a numpy array!
This loads a dictionary from the file:
datadict = pickle.load(f)
Then select two values from the dictionary. Ordinary dictionary key indexing:
X = datadict['data']
Y = datadict['labels']
Evidently X is a numpy array. reshape is a method (a function that 'belongs' to the array).
X = X.reshape(10000, 3, 32, 32)
A numpy array has a property called shape. After this reshape, X.shape should return (10000, 3, 32, 32), the shape of a 4dimensional array. The numbers are the newshape parameter described in the documentation.
The documentation is for the function version of reshape. It would be used as:
X = np.reshape(X, (10000, 3, 32, 32))
Same functionality, just a different way of invoking it.
To go on from here you probably need to study numpy documentation.
The documentation for the method version is:
a.reshape(shape, order='C')
Returns an array containing the same data with a new shape.
Refer to numpy.reshape for full documentation.
Related
I am in the process of converting some matlab code to python. I working with a 3d volume h x w x d represented as an numpy array, I am extracting smaller 3d patches from this volume using the function from SO here. So if I have 32x32x32 array and extract 16x16x16 patches I end up with a shape (2, 2, 2, 16, 16, 16) After processing each patch I would like to put it back into shape h x w x d basically reverse window_nd What would be the idiomatic numpy way without looping each dimension? Since I also need to work with 2d and 4d data I would like to avoid creating a function for each dimension.
Normally, writing back to as_strided views is not advised because it can cause race conditions, but since you only made blocks, this should work:
original_shaped_array = windowed_array.transpose(0,3,1,4,2,5).reshape(32,32,32)
Additionally, if you never copied the windowed array, and do calculations in-place, the data should be changed in the original array - a windowed view is simply a new view into the same data. Don't do this if there is any overlap
I have an numpy array X which contains 2d images. numpy array dimensions are (1000,60,40) (1000=no.of img).
I want to feed this array to my model but requires dimensions to be
(1000,60,40,1) (appended 1 is for no. of channels).
so i reshape the array by
Y=X.reshape(1000,60,40,1)
as I was having wrong predictions I checked by re-reshaping the reshaped array to check if it was same as my orig img,
I did that by doing
Z=Y.reshape(1000,60,40)
And I saved them as PNG by doing
for i in range(1000):
misc.imsave('img_rereshaped'+str(i)+'.png',Z[i])
It gives some png files as output but they are not same as the respective original ones from the X numpy array
Am I reshaping in the wrong way or reshaping changes the input data and again reshaping the reshaped data would give different result than the original data?
To test whether the reshaping is causing a problem, it's better to test it without involving other potential errors coming from, say, misc.imsave() etc.
Running something like:
import numpy as np
a = np.random.rand(10,3)
b = np.reshape(a, [10, 3, 1])
c = np.reshape(b, [10, 3])
print(np.sum(c - a))
you'll see that going back and forth using reshape doesn't cause a problem.
Could be you're not using the PNG save correctly. Perhaps the function expects 3 channels for example. Try plotting it locally using matplotlib.
I found an odd behavior of numpy's random number generators.
It seems that they do not generate consistent matrix shapes for a given argument.
It is just super annoying to spend an extra line for conversion afterward which I'd like to circumvent.
How can I tell matlib.randn directly to generate a vector of size (200,)?
import numpy as np
A = np.zeros((200,))
B = np.matlib.randn((200,))
print(A.shape) # prints (200,)
print(B.shape) # prints (1, 200)
Use numpy.random instead of numpy.matlib:
numpy.random.randn(200).shape # prints (200,)
numpy.random.randn can create any shape, whereas numpy.matlib.randn always creates a matrix.
B is a matrix object, not a ndarray. The matrix object doesn't have an 1D equivalent objects and are not recommended to use anymore, so you should use np.random.random instead.
I have a different shape of 3D matrices. Such as:
Matrix shape = [5,10,2048]
Matrix shape = [5,6,2048]
Matrix shape = [5,1,2048]
and so on....
I would like to put them into big matrix, but I am normally getting a shape error (since they have different shape) when I am trying to use numpy.asarray(list_of_matrix) function.
What would be your recommendation to handle such a case?
My implementation was like the following:
matrices = []
matrices.append(mat1)
matrices.append(mat2)
matrices.append(mat3)
result_matrix = numpy.asarray(matrices)
and having shape error!!
UPDATE
I am willing to have a result matrix that is 4D.
Thank you.
I'm not entirely certain if this would work for you, but it looks as though your matrices only disagree along the 1st axis, so why not concatenate them:
e.g.
>>> import numpy as np
>>> c=np.zeros((5,10,2048))
>>> d=np.zeros((5,6,2048))
>>> e=np.zeros((5,1,2048))
>>> f=np.concatenate((c,d,e),axis=1)
>>> f.shape
(5, 17, 2048)
Now, you'd have to keep track of which indices of the 1st axis corresponds to which matrices, but maybe this could work for you?
Use of unsqueeze():
input = torch.Tensor(2, 4, 3) # input: 2 x 4 x 3
print(input.unsqueeze(0).size()) # prints - torch.size([1, 2, 4, 3])
Use of view():
input = torch.Tensor(2, 4, 3) # input: 2 x 4 x 3
print(input.view(1, -1, -1, -1).size()) # prints - torch.size([1, 2, 4, 3])
According to documentation, unsqueeze() inserts singleton dim at position given as parameter and view() creates a view with different dimensions of the storage associated with tensor.
What view() does is clear to me, but I am unable to distinguish it from unsqueeze(). Moreover, I don't understand when to use view() and when to use unsqueeze()?
Any help with good explanation would be appreciated!
view() can only take a single -1 argument.
So, if you want to add a singleton dimension, you would need to provide all the dimensions as arguments. For e.g., if A is a 2x3x4 tensor, to add a singleton dimension, you would need to do A:view(2, 1, 3, 4).
However, sometimes, the dimensionality of the input is unknown when the operation is being used. Thus, we dont know that A is 2x3x4, but we would still like to insert a singleton dimension. This happens a lot when using minibatches of tensors, where the last dimension is usually unknown. In these cases, the nn.Unsqueeze is useful and lets us insert the dimension without explicitly being aware of the other dimensions when writing the code.
unsqueeze() is a special case of view()
For convenience, many python libraries have short-hand aliases for common uses of more general functions.
view() reshapes a tensor to the specified shape
unsqueeze() reshapes a tensor by adding a new dimension of depth 1
(i.e. turning an n.d tensor into an n+1.d tensor)
When to use unsqueeze()?
Some example use cases:
You have a model designed to intake RGB image tensors (3d: CxHxW), but your data is 2d greyscale images (HxW)
Your model is designed to intake batches of data (batch_size x dim1 x dim2 x ...), and you want to feed it a single sample (i.e. a batch of size 1).