Given an input tensor of shape (C, B, H) torch.Size([2, 5, 32]) of some neural net layers, where
channels = 2
batch_size = 5
hidden_size = 32
The goal is to flatten the channels and manipulate the input tensor to the shape (B, C*H) torch.Size([5, 2 * 32]), where:
batch_size = 5
hidden_size = 32 * 2
I've tried to do the following:
import torch
t = torch.rand([2, 5, 32])
# Changed from (channels, batch_size, hidden_size)
# -> (batch_size, channels, hidden_size)
t = t.permute(1, 0, 2)
# Reshape using view(), where batch_size is t.size(0)
# and -1 is to flatten the left over values to the other dimension.
z = t.contiguous().view(t.size(0), -1)
print(z.shape)
print(z)
[out]:
torch.Size([5, 64])
tensor([[0.3911, 0.9586, 0.2104, 0.3937, 0.9976, 0.3378, 0.0630, 0.6676, 0.0806,
0.9311, 0.5219, 0.1697, 0.7442, 0.5162, 0.2555, 0.0826, 0.5502, 0.9700,
0.3375, 0.5012, 0.9025, 0.8176, 0.1465, 0.1848, 0.3460, 0.9999, 0.7892,
0.7577, 0.6615, 0.2620, 0.6868, 0.2003, 0.4840, 0.8354, 0.9253, 0.3172,
0.9516, 0.8962, 0.1272, 0.2268, 0.6510, 0.5166, 0.6772, 0.9616, 0.9826,
0.5254, 0.9191, 0.4378, 0.7048, 0.8808, 0.0299, 0.1102, 0.9710, 0.8714,
0.7256, 0.9684, 0.6117, 0.1957, 0.8663, 0.4742, 0.2843, 0.6548, 0.9592,
0.1559],
[0.2333, 0.0858, 0.5284, 0.2965, 0.3863, 0.3370, 0.6940, 0.3387, 0.3513,
0.1022, 0.3731, 0.3575, 0.7095, 0.0053, 0.7024, 0.4091, 0.3289, 0.5808,
0.5640, 0.8847, 0.7584, 0.8878, 0.9873, 0.0525, 0.7731, 0.2501, 0.9926,
0.5226, 0.0925, 0.0300, 0.4176, 0.0456, 0.4643, 0.4497, 0.5920, 0.9519,
0.6647, 0.2379, 0.4927, 0.9666, 0.1675, 0.9887, 0.7741, 0.5668, 0.7376,
0.4452, 0.7449, 0.1298, 0.9065, 0.3561, 0.5813, 0.1439, 0.2115, 0.5874,
0.2038, 0.1066, 0.3843, 0.6179, 0.8321, 0.9428, 0.1067, 0.5045, 0.9324,
0.3326],
[0.6556, 0.1479, 0.9288, 0.9238, 0.1324, 0.0718, 0.6620, 0.2659, 0.7162,
0.7559, 0.7564, 0.2120, 0.3943, 0.9497, 0.7520, 0.8455, 0.4444, 0.4708,
0.8371, 0.6365, 0.3616, 0.0326, 0.1581, 0.4973, 0.6701, 0.9245, 0.8274,
0.3464, 0.7044, 0.5376, 0.0441, 0.5210, 0.8603, 0.7396, 0.2544, 0.3514,
0.5686, 0.3283, 0.7248, 0.4303, 0.9531, 0.5587, 0.8703, 0.1585, 0.9161,
0.9043, 0.9778, 0.4489, 0.9463, 0.8655, 0.5576, 0.1135, 0.1268, 0.3424,
0.1504, 0.2265, 0.1734, 0.1872, 0.3995, 0.1191, 0.0532, 0.6109, 0.1662,
0.6937],
[0.6342, 0.1922, 0.1758, 0.4625, 0.7654, 0.6509, 0.2908, 0.1546, 0.4768,
0.3779, 0.2490, 0.0086, 0.6170, 0.5425, 0.6953, 0.4730, 0.5834, 0.8326,
0.0165, 0.8236, 0.0023, 0.7479, 0.5621, 0.9894, 0.5957, 0.0857, 0.6087,
0.5667, 0.5478, 0.8197, 0.9228, 0.7329, 0.4434, 0.5894, 0.9860, 0.6133,
0.2395, 0.4718, 0.8830, 0.6361, 0.6104, 0.6630, 0.5084, 0.7604, 0.7591,
0.3601, 0.6888, 0.6767, 0.9178, 0.5291, 0.0591, 0.4320, 0.7875, 0.5038,
0.4419, 0.0319, 0.3719, 0.5843, 0.0334, 0.3525, 0.0023, 0.1205, 0.4040,
0.7908],
[0.0989, 0.8436, 0.0425, 0.6247, 0.6091, 0.4778, 0.2692, 0.4785, 0.9217,
0.9604, 0.6355, 0.4686, 0.9414, 0.7722, 0.8013, 0.1660, 0.6578, 0.6414,
0.6814, 0.6212, 0.4124, 0.7102, 0.7416, 0.7404, 0.9842, 0.6542, 0.0106,
0.3826, 0.5529, 0.8079, 0.9855, 0.3012, 0.2341, 0.9353, 0.6597, 0.7177,
0.8214, 0.1438, 0.4729, 0.6747, 0.9310, 0.4167, 0.3689, 0.8464, 0.9395,
0.9407, 0.8419, 0.5486, 0.1786, 0.1423, 0.9900, 0.9365, 0.3996, 0.1862,
0.6232, 0.7547, 0.7779, 0.4767, 0.6218, 0.9079, 0.6153, 0.1488, 0.5960,
0.4015]])
Although the permute() + view() achieve the desired output, are there other ways to perform the same operation? Is there a better way that can directly rehape without first permutating the order of the shape?
Let's look "behind the curtain" and see why one must have both permute/transpose and view in order to go from a C-B-H to B-C*H:
Elements of tensors are stored as a long contiguous vector in memory. For instance, if you look at a 2-3-4 tensor it has 24 elements stored at 24 consecutive places in memory. This tensor also has a "header" that tells pytorch to treat these 24 values as a 2-by-3-by-4 tensor. This is done by storing not only the size of the tensor, but also "strides": what is the "stride" one need to jump in order to get to the next element along each dimension. In our example, size=(2,3,4) and strides=(12, 4, 1) (you can check this out yourself, and you can see more about it here).
Now, if you only want to change the size to 2-(3*4) you do not need to move any item of the tensor in memory, only to update the "header" of the tensor. By setting size=(2, 12) and strides=(12, 1) you are done!
Alternatively, if you want to "transpose" the tensor to 3-2-4 that's a bit more tricky, but you can still do that by manipulating the strides. Setting size=(3, 2, 4) and strides=(4, 12, 1) gives you exactly what you want without moving any of the real tensor elements in memory.
However, once you manipulated the strides, you cannot trivially change the size of the tensor - because now you will need to have two different "stride" values for one (or more) dimensions. This is why you must call contiguous() at this point.
Summary
If you want to move from shape (C, B, H) to (B, C*H) you must have permute, contiguous and view operations, otherwise you just scramble the entries of your tensor.
A small example with 2-3-4 tensor:
a =
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
If you just change the view of the tensor you get
a.view(3,8)
array([[ 0, 1, 2, 3, 4, 5, 6, 7],
[ 8, 9, 10, 11, 12, 13, 14, 15],
[16, 17, 18, 19, 20, 21, 22, 23]])
Which is not what you want!
You need to have
a.permute(1,0,2).contiguous().view(3, 8)
array([[ 0, 1, 2, 3, 12, 13, 14, 15],
[ 4, 5, 6, 7, 16, 17, 18, 19],
[ 8, 9, 10, 11, 20, 21, 22, 23]])
Einops allows doing such element rearrangements in one (readable) line
from einops import rearrange
import torch
t = torch.rand([2, 5, 32])
y = rearrange(t, 'c b h -> b (c h)')
y.shape # prints torch.Size([5, 64])
I'm trying to classify product items in order to predict their category based on the product title and their base price.
An example(product title, price, category):
['notebook sony vaio vgn-z770td dockstation', 3000.0, u'MLA54559']
Previously I was only using product title for the prediction task but I'd like to include the price to see if the accuracy improves.
The problem with my code is that I can't merge the text/numeric features, I've been reading some questions here in SO and this is my code excerpt:
#extracting features from text
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform([e[0] for e in training_set])
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
#extracting numerical features
X_train_price = np.array([e[1] for e in training_set])
X = sparse.hstack([X_train_tfidf, X_train_price]) #this is where the problem begins
clf = svm.LinearSVC().fit(X, [e[2] for e in training_set])
I try to merge the data types with sparse.hstack but I get the following error:
ValueError: blocks[0,:] has incompatible row dimensions
I guess the problem lies in X_train_price(a list of prices) but I don't know how to format it for the sparse function to succesfully work.
These are the shapes of both arrays:
>>> X_train_tfidf.shape
(65845, 23136)
>>>X_train_price.shape
(65845,)
It looks to me like this should be as simple as stacking the arrays. If scikit-learn follows the conventions I'm familiar with, then each row in X_train_tfidf is a training datapoint, and there are a total of 65845 points. So you just have to do an hstack -- as you said you tried to do.
However, you need to make sure the dimensions are compatible! In vanilla numpy you get this error otherwise:
>>> a = numpy.arange(15).reshape(5, 3)
>>> b = numpy.arange(15, 20)
>>> numpy.hstack((a, b))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/
Extras/lib/python/numpy/core/shape_base.py", line 270, in hstack
return _nx.concatenate(map(atleast_1d,tup),1)
ValueError: arrays must have same number of dimensions
Reshape b to have the correct dimensions -- noting that a 1-d array of shape (5,) is totally different from a 2-d array of shape (5, 1).
>>> b
array([15, 16, 17, 18, 19])
>>> b.reshape(5, 1)
array([[15],
[16],
[17],
[18],
[19]])
>>> numpy.hstack((a, b.reshape(5, 1)))
array([[ 0, 1, 2, 15],
[ 3, 4, 5, 16],
[ 6, 7, 8, 17],
[ 9, 10, 11, 18],
[12, 13, 14, 19]])
So in your case, you want an array of shape (65845, 1) instead of (65845,). I might be missing something because you are using sparse arrays. Nonetheless, the principle ought be the same. I have no idea what sparse format you're using based on the above code, so I just picked one to test:
>>> a = scipy.sparse.lil_matrix(numpy.arange(15).reshape(5, 3))
>>> scipy.sparse.hstack((a, b.reshape(5, 1))).toarray()
array([[ 0, 1, 2, 15],
[ 3, 4, 5, 16],
[ 6, 7, 8, 17],
[ 9, 10, 11, 18],
[12, 13, 14, 19]])
Is there a way to efficiently implement a rolling window for 1D arrays in Numpy?
For example, I have this pure Python code snippet to calculate the rolling standard deviations for a 1D list, where observations is the 1D list of values, and n is the window length for the standard deviation:
stdev = []
for i, data in enumerate(observations[n-1:]):
strip = observations[i:i+n]
mean = sum(strip) / n
stdev.append(sqrt(250*sum([(s-mean)**2 for s in strip])/(n-1)))
Is there a way to do this completely within Numpy, i.e., without any Python loops? The standard deviation is trivial with numpy.std, but the rolling window part completely stumps me.
I found this blog post regarding a rolling window in Numpy, but it doesn't seem to be for 1D arrays.
Just use the blog code, but apply your function to the result.
i.e.
numpy.std(rolling_window(observations, n), 1)
where you have (from the blog):
def rolling_window(a, window):
shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)
strides = a.strides + (a.strides[-1],)
return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
Starting in Numpy 1.20, you can directly get a rolling window with sliding_window_view:
from numpy.lib.stride_tricks import sliding_window_view
sliding_window_view(np.array([1, 2, 3, 4, 5, 6]), window_shape = 3)
# array([[1, 2, 3],
# [2, 3, 4],
# [3, 4, 5],
# [4, 5, 6]])
I tried using so12311's answer listed above on a 2D array with shape [samples, features] in order to get an output array with shape [samples, timesteps, features] for use with a convolution or lstm neural network, but it wasn't working quite right. After digging into how the strides were working, I realized that it was moving the window along the last axis, so I made some adjustments so that the window is moved along the first axis instead:
def rolling_window(a, window_size):
shape = (a.shape[0] - window_size + 1, window_size) + a.shape[1:]
strides = (a.strides[0],) + a.strides
return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
NOTE: there is no difference in the output if you are only using a 1D input array. In my search this was the first result to get close to what I wanted to do, so I am adding this to help any others searching for a similar answer.
With only one line of code...
import pandas as pd
pd.Series(observations).rolling(n).std()
Based on latter answers, here I add code for rolling 1-D numpy arrays choosing window size and window steps frequency.
a = np.arange(50)
def rolling_window(array, window_size,freq):
shape = (array.shape[0] - window_size + 1, window_size)
strides = (array.strides[0],) + array.strides
rolled = np.lib.stride_tricks.as_strided(array, shape=shape, strides=strides)
return rolled[np.arange(0,shape[0],freq)]
rolling_window(a,10,5)
Output:
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[15, 16, 17, 18, 19, 20, 21, 22, 23, 24],
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[25, 26, 27, 28, 29, 30, 31, 32, 33, 34],
[30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[35, 36, 37, 38, 39, 40, 41, 42, 43, 44],
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49]])
def moving_avg(x,n):
mv = np.convolve(x,np.ones(n)/n,mode='valid')
return np.concatenate(([np.NaN for k in range(n-1)],mv))
I needed a rolling window to apply to any intermediate axis of an n-dimensional array, so I extended the code from the already accepted answer and #Miguel Gonzalez. The corresponding code to apply a rolling window to an n-d array along any axis:
def rolling_window(array, window, freq, axis=0):
shape = array.shape[:axis] + (array.shape[axis] - window_size + 1, window_size) + array.shape[axis+1:]
strides = array.strides[:axis] + (array.strides[axis],) + array.strides[axis:]
rolled = np.lib.stride_tricks.as_strided(array, shape=shape, strides=strides)
return np.take(rolled, np.arange(0,shape[axis],freq), axis=axis)
An example to create a test to assert validity of the function:
arr = np.random.randint(1, 1000, size=(2,108,21,5))
arr_windowed = rolling_window_ndimensional(arr, 12, 12, axis=1)
print(arr.shape)
print(arr_windowed.shape)
np.allclose(arr, arr_windowed.reshape(2,-1, 21,5))