I have a tensor in Tensor Flow that is of the size (2, 16384, 11). I am trying to use the tf.slice function to pull 1D tensors out of that array. I can name two point in the array as the start and stop.
The first column is [0, 0, 1] ---> [0, 16383, 1].
The second column is [1, 0, 1] ---> [1, 16383, 1].
But the problem is this returns a tensor with the dimensions of (0, 16383) and (1, 16383). Accessing an array with a length 0 is a problem; I understand that you can get there by only using the [:] accessor as opposed to the [0] accessor, otherwise you get the error
'0 is out of bounds for axis 0 of length 0'.
How else can I get TF to output a single column of numbers? Here is the code.
Xdata = tf.slice(x, [0,0,1], [0,16383,1])
Ydata = tf.slice(x, [1,0,1], [1,16383,1])
Xarry = Xdata.numpy()
Yarry = Ydata.numpy()
# Outputs
print(Xarry.shape) # (0, 16383, 1)
print(Yarry.shape) # (1, 16383, 1)
print(Xarry[:,:,0]) # []
print(Yarry[0,:,0]) # [22.05 20.92 22.11 ... 22.53 22.03 22.47]
plt.plot(Xarry[:,:,0],Yarry[0,:,0]) # <--- Error is here
Which produces:
(0, 16383, 1)
(1, 16383, 1)
[]
[22.05 20.92 22.11 ... 22.53 22.03 22.47]
ValueError: x and y must have same first dimension, but have shapes
(0, 16383) and (16383,)
I have tried using .flatten() but this does not get around the problem. I have also looked at using tf.gather().
Xarry shape is (0, 16383, 1), which means an empty list, since the first dimension is zero. So, if you print it you will get []. As it has nothing to present, then slicing like Xarry[:,:,0] will also give you another [].
I think you have a misunderstanding of tf.slice arguments. The second argument is the size of tensor, beginning from the first argument index. The second argument in tf.slice() is not the stop index, but the size.
So, consider to change the size to something like this:
Xdata = tf.slice(x, [0,0,1], [1,16383,1]) # start from [0,0,1] index and slice an array with size (1,16383,1)
Ydata = tf.slice(x, [1,0,1], [1,16383,1])
Xarry = Xdata.numpy()
Yarry = Ydata.numpy()
# Outputs
print(Xarry.shape) # (0, 16383, 1)
print(Yarry.shape) # (1, 16383, 1)
print(Xarry[0,:,0]) # [22.05 20.92 22.11 ... 22.53 22.03 22.47]
print(Yarry[0,:,0]) # [22.05 20.92 22.11 ... 22.53 22.03 22.47]
import matplotlib.pyplot as plt
plt.plot(Xarry[0,:,0],Yarry[0,:,0])
plt.show()
Related
I have two numpy arrays, one with shape let's say (10, 5, 200), and another one with the shape (1, 200), how can I stack them so I get as a result an array of dimensions (10, 6, 200)? Basically by stacking it to each 2-d array iterating along the first dimension
a = np.random.random((10, 5, 200))
b = np.zeros((1, 200))
I'v tried with hstack and vstack but I get an error in incorrect number of axis
Let's say:
a = np.random.random((10, 5, 200))
b = np.zeros((1, 200))
Let's look at the volume (number of elements) of each array:
The volume of a is 10*5*200 = 10000.
The volume of an array with (10,6,200) is 10*5*200=1200.
That is you want to create an array that has 2000 more elements.
However, the volume of b is 1*200 = 200.
This means a and b can't be stacked.
As hpaulj mentioned in the comments, one way is to define an numpy array and then fill it:
result = np.empty((a.shape[0], a.shape[1] + b.shape[0], a.shape[2]))
result[:, :a.shape[1], :] = a
result[:, a.shape[1]:, :] = b
I have a tensor A of shape (100, 16, 16) and tensor B of shape (100), where 100 is the batch size. I want to create a binary mask of A that has shape (100, 16, 16), where in each element (element has shape (1, 16, 16)) of the mask, the value is 1 if the element is greater than the computed quantile value, else 0. Each element in tensor B indicates the percentile value for each individual element in A, in sequence. If B is simply a scalar, I can use:
flat_A = torch.reshape(A, (100, -1))
quants = torch.quantile(flat_A, B, dim=1)
quants = torch.reshape(quants, (100, 1, 1))
mask = torch.where(A >= quants, 1, 0)
# quants will have shape (100, 1, 1)
The question is: if B is a 1D tensor of shape (100) like I said above, how can I compute the percentile value for each individual element in A? I tried the following, but the results did not look like what I expected:
>>> torch.quantile(flat_A, B, dim=1).shape
torch.Size([100, 100])
>>> torch.quantile(flat_A, B, dim=0).shape
torch.Size([100, 256])
I think the result's shape should be (100), so I can use mask = torch.where(A >= quants, 1, 0), or maybe I misunderstand it?
For more context, this question is also the extension of the scalar B value question I had previously here.
This is one way using torch.quantile() function. Note that here I am using tensors of shape (5, 2, 2) instead of (100, 16, 16) for simplicity.
import torch
# Generate some data of shape (5, 2, 2)
A = torch.arange(5 * 2 * 2).reshape(5, 2, 2) + 1.0
B = torch.linspace(0, 1, 5) # 5 quantile values for each element in A
Af = A.reshape(A.shape[0], -1) # flattens A to a 2D tensor
quantiles = torch.quantile(Af, B, dim = 1, keepdim = True)
quants = quantiles[torch.arange(A.shape[0]), torch.arange(A.shape[0]), 0]
mask = (A >= quants[:, None, None]).type(torch.uint8)
Here the tensor quantiles is of shape torch.Size([5, 5, 1]) because it stores the thresholds for each quantile value in B for each element in A (or row in Af). Since we have 5 quantile values, we get 5 thresholds for each element in A.
For instance, quantiles[i, j, 0] has the threshold for B[i]th quantile of A[j] or Af[j], and you essentially need the values quantiles[k, k, 0] for k in range of batch size or 5 here.
Now to satisfy the requirement that you need thresholds for corresponding quantiles in B and elements in A, simply index out the diagonal elements from quantiles and populate quants that has shape torch.Size([5]).
Finally to get the mask, compare A with the corresponding thresholds for each element. Note that this uses a broadcasted elementwise comparison with the thresholds. mask has the required shape of torch.Size([5, 2, 2]).
I am going through the following lines of code but I didn't understand image[...,list()]. What do the three dots mean?
self.probability = 0.5
self.indices = list(permutations(range(3), 3))
if random.random() < self.probability:
image = np.asarray(image)
image = Image.fromarray(image[...,list(self.indices[random.randint(0, len(self.indices) - 1)])])
What exactly is happening in the above lines?
I have understood that the list() part is taking random channels from image? Am I correct?
It is an object in Python called Ellipsis (for example, as a placeholder for something missing).
x = np.random.rand(3,3,3,3,3)
elem = x[:, :, :, :, 0]
elem = x[..., 0] # same as above
This should be helpful if you want to access a specific element in a multi-dimensional array in NumPy.
list(permutations(range(3), 3)) generates all permutations of the intergers 0,1,2.
from itertools import permutations
list(permutations(range(3), 3))
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]
So the following chooses among these tuples of permutations:
list(self.indices[random.randint(0, len(self.indices) - 1)])]
In any case you'll have a permutation over the last axis of image which is usually the image channels RGB (note that with the ellipsis (...) here image[...,ixs] we are taking full slices over all axes except for the last. So this is performing a shuffling of the image channels.
An example run -
indices = list(permutations(range(3), 3))
indices[np.random.randint(0, len(indices) - 1)]
# (2, 0, 1)
Here's an example, note that this does not change the shape, we are using integer array indexing to index on the last axis only:
a = np.random.randint(0,5,(5,5,3))
a[...,(0,2,1)].shape
# (5, 5, 3)
I need to insert 3-dimensional matrices into a new variable.
I'm trying to do that by:
Creating a 4-dimensional matrix and by promoting the fourth dimension saving the three dimensions respectively.
Sample code:
from python_speech_features import mfcc
import numpy as np
X = np.zeros((0,0,0,0),float) #4-dimensional - (0, 0, 0, 0)
ii = 0
for ii in range 1000:
data, fs = sf.read(curfile[ii])
sig = mfcc(data, fs, winstep=winstep,winlen=winlen,nfft=1024) #size - (49, 13)
sig = sig[:, :, np.newaxis] #add third-dimensional - (49, 13, 1)
X[:,:,:,ii] = sig
Error:
IndexError: index 0 is out of bounds for axis 3 with size 0
Someone can help me with that problem?
You are not creating array in right way. You cannot insert value in axis which have zero length at least specify some length for axis
X = np.zeros((10, 10, 10,1000), float)
print(X.shape)
# (10, 10, 10, 1000)
Now you can set value in whatever axis you want by simply,
X[:, :, :, 2] = 1
# this will simply set value of 3rd axis's 3rd element to 1
Either use np.stack (i think it is the best way of doing it) or create the initial array in its final size:
np.zeros((49,13,1,1000), float)
In your case
I have a numpy array v with shape (1000, 68), v is supposed to padding to 100 dimension with 0s. As a result, the v's shape will be (1000, 100)
I tried to use the following approaches:
t = np.lib.pad(v, (16, 16), 'minimum') # numpy method
t = sequence.pad_sequences(v, maxlen = 100, padding = 'post') # Keras text processing method
Above two methods returned the t with correct shape (1000, 100), but each array t[n] (n from 0 to 99) is a zero vector [0, 0, 0, ....0]
Following numpy.pad documentation, I tried
np.pad(v, [(0,0), (16,16)], 'constant')
with the expected result: 16 columns of zeros added on the left, and 16 on the right.