I have a .npy file here
Its just a file with an object that is a list of images and their labels. for example:
{
'2007_002760': array([0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.,0., 0., 0.], dtype=float32),
'2008_004036': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 0.,0., 0., 0.], dtype=float32)
}
I want to open the file and get its length, and then possibly add to it or modify it
I am able to open the file, but I cant get the length of items in it.
Heres how i open it:
import numpy as np
file = np.load('cls_labels.npy', allow_pickle = True)
print(file.size)
What am I missing here?
Your file contains a dictionary wrapped inside a 0-dimensional numpy object. The magic to extract the actual information is:
my_dictionary = file[()]
This is a standard dictionary whose keys are strings like '2008_004036' and whose values are numpy arrays.
Edit: And as mentioned above, you shouldn't be saving dictionaries using numpy.save(), you should have been using pickle. You end up with horrors like file[()].
here is the correct and easiest way to do it:
cls_labels = np.load('cls_labels.npy', allow_pickle = True).item()
Related
I want to run combined non max suppression in a set of
windows for an image.
I am using tf.image.combined_non_max_suppression from tensorflow as follow:
import matplotlib.pyplot as plt
import matplotlib.patches as patches
from PIL import Image
import numpy as np
import tensorflow as tf
boxesX=np.array(([200,100,150,100],[220,120,150,100],[190,110,150,100],[210,112,150,100])).astype('float32')
scoresX=np.array(([0.2,0.7,0.1],[0.1,0.8,0.1],[0.3,0.6,0.1],[0.05,0.9,0.05]))
boxes1=tf.reshape(boxesX,(1,4,1,4))
boxes2=tf.dtypes.cast(boxes1, tf.float32)
scores1=tf.reshape(scoresX,(1,4,3))
scores2=tf.dtypes.cast(scores1, tf.float32)
boxes, scores, classes, valid_detections = tf.image.combined_non_max_suppression(
boxes=boxes2,
scores=scores2,
max_output_size_per_class=10,
max_total_size=10,
iou_threshold=0.5,
score_threshold=0.2)
But the output 'boxes' is just an array of zeros and ones:
array([[[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.],
[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.]]], dtype=float32)>
The boxes are being clipped between [0,1]. All you need to do is add the argument clip_boxes=False:
boxes, scores, classes, valid_detections = tf.image.combined_non_max_suppression(
boxes=boxes2,
scores=scores2,
max_output_size_per_class=10,
max_total_size=10,
iou_threshold=0.5,
score_threshold=0.2,
clip_boxes=False)
I use UnivariateSpline from scipy module to fit data.It works for almost all cases except for this one, which gives rise to Process finished with exit code -1073741819 (0xC0000005) error. If I change smoothing factor s to 0, it also works. Any suggestions to solve this problem will help.
Update1
My working environment is:
python 3.7
scipy 1.3.2
numpy 1.17.4
import matplotlib.pyplot as plt
import numpy as np
from scipy.interpolate import UnivariateSpline, InterpolatedUnivariateSpline
x = np.arange(78)
y = np.asarray([
0., 0., 0., 0., 0., 0.,
0., 0., 5.03989319, 4.03191455, 4.03191455, 3.02393591,
3.02393591, 2.01595727, 2.01595727, 1.00797864, 0., 0.,
0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0.])
spl = UnivariateSpline(x, y, k=1, s=0.01)
knots = list(map(int, spl.get_knots()))
plt.plot(knots, y[knots], 'rx')
plt.plot(knots, y[knots], 'r-')
plt.plot(x, y, 'b-')
plt.show()
The combination of you s and k parameter are causing the issue.
According to the documentation, the number of knots increases until the condition sum((w[i] * (y[i]-spl(x[i])))**2, axis=0) <= s is met. However, because you have a limited number of non-zero data points, you can only add so many meaningful knots to the data set, and because you are doing k=1 spline (as opposed to cubic for example), the difference between the spline value and the data values is never reaching the prescribed s value.
Your options include increasing k (I tested with k=3 and it worked) or increase the s value to have a less strict condition (anything above s=0.08 worked for me). Note your code worked when s=0 because for that condition, instead of doing a smoothing, the algorithm just interpolates between each point and does no smoothing (which maybe is what you want).
Python 3.x
I have for loop which is making some calculations and creating one Slice/2D Array lets say (x = 3, y = 3) per iteration and I want at the same time in the same for loop (append?/stack) them in a third dimension.
I have been trying with Numpy stack, vstack, hstack, dstack but I still don't get how to get them together in the 3rd dimension as I want.
So I would like to have at them end something like this:
(z = 10, x = 3, y = 3)
array([ [[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]],
[[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]],
[[2., 2., 2.],
[2., 2., 2.],
[2., 2., 2.]],
.
.
.
])
Thanks,
you can do it like this
arrays = []
for i in range(5):
arr = np.full((3,3), i)
arrays.append(arr)
np.asarray(arrays)
If you want to you can do np.asarray(arrays) inside loop. But it will be not very efficient. Not that np.concatenate will also effectively creates new numpy array so efficiency will be similar. Doing these operation once outside the loop is better
I am trying to do multiple indices. However, it is showing the error of too many indices. Please help me and if you have some doubts or confusion in it then please leave comments in the comment box.
My result has (6561114,) shape and I want to indices the whole first row for instance ([array([-1., 1., 0., 0., 1.]), array([[43., 0., 43., 1., 2.]]), array([-43., 43., 0., 2., 3.]) and then from 3 columns I want to extract each value of array-like [-1,43,-43],[1,0,43], and so on.
This is Output:-
array([array([-1., 1., 0., 0., 1.]),
array([[43., 0., 43., 1., 2.]]),
array([-43., 43., 0., 2., 3.]), ...,
array([-1.406830e+01, 3.552240e+01, 2.145410e+01,
9.492236e+06,
9.492237e+06]),
array([[1.421949e+02, 2.145410e+01, 1.636490e+02, 9.492237e+06,
9.492238e+06],
[3.387300e+01, 1.636490e+02, 1.975220e+02, 9.492238e+06,
9.492239e+06]]),
array([-1.9052487e+02, 1.9752200e+02, 6.9971300e+00,
9.4922390e+06,
9.4922400e+06])], dtype=object)
This is what error looks like:-
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-29-537ba6ddfd42> in <module>
----> 1 result1[0,:]
IndexError: too many indices for array
Check in second array you are using 2 [] braces.
Here array([[43., 0., 43., 1., 2.]]) you are using 2 [] braces, remove [] braces, and use it
I am using this code, and its working fine
Check using print(a[0]) and post your output.
Code
import numpy as np
a=np.array([np.array([-1., 1., 0., 0., 1.]),np.array([43., 0., 43., 1., 2.]),np.array([-43., 43., 0.2., 3.])],dtype=object)
print(a[:,0])
For example I need 30x30 numpy arrays created from images to be fed to a neural net. If I have a directory of images to predict, I should be able to loop through the directory, get image data and create an (n,30,30) shape np array
This is my current method, I intend to reshape each row before feeding to the model
def get_image_vectors(path):
img_list=os.listdir(path)
print(img_list)
X=np.empty((900,))
for img_file in img_list:
img= Image.open(os.path.join(path,img_file))
img_grey= img.convert("L")
resized = img_grey.resize((30,30))
flattened = np.array(resized.getdata())
# print(flattened.shape)
X=np.vstack((X,flattened))
print(img_file,'=>',X.shape)
return X[1:,:]
Instead of appending to an existing array, it will probably be better to use a list initially, appending to it, and converting to an array at the end. thus saving many redundant modifications of np arrays.
Here a toy example:
import numpy as np
def get_image_vectors():
X= [] #Create empty list
for i in range(10):
flattened = np.zeros(900)
X.append(flattened) #Append some np array to it
return np.array(X) #Create array from the list
With result:
array([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]])