I don't know why the error occurs, and I've tried to reshape but no luck.
Thanks for answering.
X_test = np.append(X_test, scaler.transform(working_data.iloc[-1][0]))
And here is the error message I receive.
ValueError: Expected 2D array, got scalar array instead:
array=9583.994119999077.
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
And the full code is here:https://activewizards.com/blog/bitcoin-price-forecasting-with-deep-learning-algorithms/
Really appreciate your help.
It is a common error that happens while working with arrays in Python. So let's say you have an array
a = [[1, 2, 3],
[4, 5, 6],
[7, 8, 9]]
shape of this array will be (3,3). Now if you access a[-1,0] you will get 7, which will have no shape as it's scalar. If you want to append this to an array say [[2, 3, 4]] this will not work, as latter is a 2-D array and for append to work, you need arrays of same dimension. So if you call reshape like this: a[-1,0].reshape(-1,1) then you will get an array of shape (1,1), that is instead of 7 you will get [[7]]. Another way to bypass this would be to access the value like this: a[-1,:1], this essentially does the same thing as a[-1,0] but it indicates the array that you want the resulting array as 2-D.
Another thing to note will be to check the axis you are trying to append to, as I do not know the shape of X_test, I can not say you want to add a new row, or a new column, so keep that in mind in case that causes an error later.
Related
I want to extract parts of an numpy ndarray based on arrays of index positions for some of the dimensions. Let me show this on an example
Example data
dummy = np.random.rand(5,2,100)
X = np.array([[0,1],[4,1],[2,0]])
dummy is the original ndarray with dimensionality 5x2x100. This dimensionality is arbitrary, it could as well be 5x2x4x100.
X is a matrix of index values, here X[:,0] are the indices of the first dimension of dummy, X[:,1] those of the second dimension. The number of columns in X is always the number of dimensions in dummy minus 1.
Example output
I want to extract an ndarray of the following form for this example
[
dummy[0,1,:],
dummy[4,1,:],
dummy[2,0,:]
]
Complications
If the number of dimensions in dummy were fixed, this could just be done by dummy[X[:,0],X[:,1],:] . Sadly the dimensionality can be different, e.g. dummy could be a 5x2x4x6x100 ndarray and X correspondingly would then be 3x4 . My attempts at dealing with it have not yielded the desired result.
dummy[X,:] yields a 3x2x2x100 ndarray for this example same as dummy[X]
Iteratively reducing dummy by doing something like dummy = dummy[X[:,i],:] with i an iterator over the number of columns of X also does not reduce the ndarray in the example past 3x2x100
I have a feeling that this should be pretty simple with numpy indexing, but I guess my search for a solution was missing the right terms for this.
Does anyone have a solution to this?
I will try to provide some explainability to #Michael Szczesny answer.
First, notice that if you have an np.array with dimension n and pass m indexes where m<n, then it will be the same as using : in the dimensions >=m. In your case, for example:
dummy[(0, 0)] == dummy[0, 0, :]
Given that, note that you can also pass an array as an index. Thus:
dummy[([0, 1], [0, 0])]
It would be the same as:
np.array([dummy[(0,0)], dummy[(1,0)]])
You can validate that using:
dummy[([0, 1], [0, 0])] == np.array([dummy[(0,0)], dummy[(1,0)]])
Finally, notice that:
(*X.T,)
# (array([0, 4, 2]), array([1, 1, 0]))
You are here getting each dimension as an array, and then you will get:
[
dummy[0,1],
dummy[4,1],
dummy[2,0]
]
Which is the same as:
[
dummy[0,1,:],
dummy[4,1,:],
dummy[2,0,:]
]
Edit: Instead of using (*X.T,), you can use tuple(X.T), which for me, makes more sense
as Michael Szczesny wrote, the best solution is dummy[(*X.T,)].
Since X[:,0] are the indices of the first dimension of dummy and X[:,1] are the indices of the second dimension of dummy, if you transpose X (X.T) you'll have the the indices of the first dimension of dummy as X.T[0] and the indices of the second dimension of dummy as X.T[1].
Now to slice dummy as you want, you can specify the indices of the first and of the second dimension in this way:
dummy[(first_dim_indices, second_dim_indices)] = dummy[(X.T[0], X.T[1])]
In order to simplify the code (and since you doesn't want to transpose the X matrix twice) you can unpack X.T in a tuple as (*X.T,) and so write X[(*X.T,)] is the same thing to write dummy[(X.T[0], X.T[1])].
This writing is also useful if you have an unfixed number of dimensions to slice trough because you will unpack from X.T as many lines as there are dimensions to slice in dummy. For example suppose you want to retrieve an 1D-array from dummy given the following indices:
first_dim: (0, 4, 2)
second_dim: (1, 1, 0)
third_dim: (9, 8, 7)
You can specify the indices of the 3 dimensions as X = np.array([[0,1,9],[4,1,8],[2,0,7]]) and dim[(*X.T,)] is still valid.
I am getting the error ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 2, the array at index 0 has size 3 and the array at index 1 has size 1 while running the below code.
for i in range(6):
print('current b', current_batch)
current_pred = model.predict(current_batch)[0]
print('current pred', current_pred)
test_predictions.append(current_pred)
print('current batch', current_batch)
print('current batch => ', current_batch[:,1:,:])
current_batch = np.append(current_batch[:,1:,:], [[current_pred]], axis=1)
getting this error
Can anyone please explain me why this is happening.
Thanks,
Basically, Numpy is telling you that the shapes of the concatenated matrices should align. For example, it is possible to concatenate a 3x4 matrix with 3x5 matrix so that we get 3x9 matrix (we added dimension 1).
The problem here is that Numpy is telling you that the axis don't align. In my example, that would be trying to concatenate 3x4 matrix with 10x10 matrix. This is not possible as the shapes are not aligned.
This usually means that the you are trying to concatenate wrong things. If you are sure though, try using np.reshape function, which will change the shape of one of the matrices so that they can be concatenated.
As the traceback shows, np.append is actually using np.concatenate. Did you read (study) the docs for either function? Understand what they say about dimensions?
From the display [[current_pred]], converted to array will be (1,1,1) shape. Do you understand that?
current_batch[:,1:,:] is, as best I can tell from the small image (1,5,3)
You are asking to join on axis 1, which is 1 and 5, ok. But it's saying that the last dimension, axis 2, doesn't match. That 1 does not equal 3. Do you understand that?
List append as you do with test_predictions.append(current_pred) works well in an iteration.
np.append does not work well. Even when it works, it is slow. And here it doesn't work, because you aren't taking sufficient care to match dimensions.
I have a tensor for example called tensor1 of shape (1,20,4). I am trying to create a tensor using certain indices (1,4,5) from this tensor. I could do this form numpy for example using: tensor[:,[1,4,5],:]. From what I understand this could be done using "tf.gather_nd" but I don't really see how it could be done.
What you want can be done with tf.gather:
tensor2 = tf.gather(tensor1, [1, 4, 5], axis=1)
I have a numpy array of arrays, for example:
x = np.array([[1,2,3],[10,20,30]])
Now lets say I want to extend each array with [4,40], to generate the following resulting array:
[[1,2,3,4],[10,20,30,40]]
How can I do this without making a copy of the whole array? I tried to change the shape of the array in place but it throws a ValueError:
x[0] = np.append(x[0],4)
x[1] = np.append(x[1],40)
ValueError : could not broadcast input array from shape (4) into shape (3)
You can't do this. Numpy arrays allocate contiguous blocks of memory, if at all possible. Any change to the array size will force an inefficient copy of the whole array. You should use Python lists to grow your structure if possible, then convert the end result back to an array.
However, if you know the final size of the resulting array, you could instantiate it with something like np.empty() and then assign values by index, rather than appending. This does not change the size of the array itself, only reassigns values, so should not require copying.
While #roganjosh is right that you cannot modify the numpy arrays without making a copy (in the underlying process), there is a simpler way of appending each value of an ndarray to the end of each numpy array in a 2d ndarray, by using numpy.column_stack
x = np.array([[1,2,3],[10,20,30]])
array([[ 1, 2, 3],
[10, 20, 30]])
stack_y = np.array([4,40])
array([ 4, 40])
numpy.column_stack((x, stack_y))
array([[ 1, 2, 3, 4],
[10, 20, 30, 40]])
Create a new matrix
Insert the values of your old matrix
Then, insert your new values in the last positions
x = np.array([[1,2,3],[10,20,30]])
new_X = np.zeros((2, 4))
new_X[:2,:3] = x
new_X[0][-1] = 4
new_X[1][-1] = 40
x=new_X
Or Use np.reshape() or np.resize() instead
I have a simple, one dimensional Python array with random numbers. What I want to do is convert it into a numpy Matrix of a specific shape. My current attempt looks like this:
randomWeights = []
for i in range(80):
randomWeights.append(random.uniform(-1, 1))
W = np.mat(randomWeights)
W.reshape(8,10)
Unfortunately it always creates a matrix of the form:
[[random1, random2, random3, ...]]
So only the first element of one dimension gets used and the reshape command has no effect. Is there a way to convert the 1D array to a matrix so that the first x items will be row 1 of the matrix, the next x items will be row 2 and so on?
Basically this would be the intended shape:
[[1, 2, 3, 4, 5, 6, 7, 8],
[9, 10, 11, ... , 16],
[..., 800]]
I suppose I can always build a new matrix in the desired form manually by parsing through the input array. But I'd like to know if there is a simpler, more eleganz solution with built-in functions I'm not seeing. If I have to build those matrices manually I'll have a ton of extra work in other areas of the code since all my source data comes in simple 1D arrays but will be computed as matrices.
reshape() doesn't reshape in place, you need to assign the result:
>>> W = W.reshape(8,10)
>>> W.shape
(8,10)
You can use W.resize(), ndarray.resize()