We Keep Coding

unable to get length of items in .npy file

I have a .npy file here
Its just a file with an object that is a list of images and their labels. for example:
{
'2007_002760': array([0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.,0., 0., 0.], dtype=float32),
'2008_004036': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 0.,0., 0., 0.], dtype=float32)
}
I want to open the file and get its length, and then possibly add to it or modify it
I am able to open the file, but I cant get the length of items in it.
Heres how i open it:
import numpy as np
file = np.load('cls_labels.npy', allow_pickle = True)
print(file.size)
What am I missing here?

Your file contains a dictionary wrapped inside a 0-dimensional numpy object. The magic to extract the actual information is:
my_dictionary = file[()]
This is a standard dictionary whose keys are strings like '2008_004036' and whose values are numpy arrays.
Edit: And as mentioned above, you shouldn't be saving dictionaries using numpy.save(), you should have been using pickle. You end up with horrors like file[()].

here is the correct and easiest way to do it:
cls_labels = np.load('cls_labels.npy', allow_pickle = True).item()

How to avoid two variables refering to the same data? #Pytorch

During initializing, I tried to reduce the repeats in my code, so instead of:
output= (torch.zeros(2, 3),
torch.zeros(2, 3))
I wrote:
z = torch.zeros(2, 3)
output= (z,z)
However, I find that the second method is wrong.
If I assign the data to variables h,c, any change on h would also be applied to c
h,c = output
print(h,c)
h +=torch.ones(2,3)
print('-----------------')
print(h,c)
results of the test above:
tensor([[0., 0., 0.],
[0., 0., 0.]]) tensor([[0., 0., 0.],
[0., 0., 0.]])
-----------------
tensor([[1., 1., 1.],
[1., 1., 1.]]) tensor([[1., 1., 1.],
[1., 1., 1.]])
Is there a more elegent way to initialize two indenpendent variables?

I agree that your initial line needs no modification but if you do want an alternative, consider:
z = torch.zeros(2, 3)
output= (z,z.clone())
The reason the other one (output = (z,z)) doesn't work, as you've correctly discovered is that no copy is made. You're only passing the same reference in each entry of the tuple to z

Assign it in a single statement but use separate reference for each, as below:
h, c = output, output

How to remove duplicates in a numpy array and keep its sorting

I have a list of numpy arrays and want to remove duplicates and also keep the order of my sorted data. This is my array with duplicates:
dup_arr=[np.array([[0., 10., 10.],\
[0., 2., 30.],\
[0., 3., 5.],\
[0., 3., 5.],\
[0., 3., 40.]]),\
np.array([[0., -1., -4.],\
[0., -2., -3.],\
[0., -3., -5.],\
[0., -3., -6.],\
[0., -3., -6.]])]
I tried to do it using the following code:
clean_arr=[]
for i in dup_arr:
new_array = [tuple(row) for row in i]
uniques = np.unique(new_array, axis=0)
clean_arr.append(uniques)
But the problem of this method is that it changes the sort of my data and I do not want to to sort them again because it is a tough task for my real data. I want to have the following result:
clean_arr=[np.array([[0., 10., 10.],\
[0., 2., 30.],\
[0., 3., 5.],\
[0., 3., 40.]]),\
np.array([[0., -1., -4.],\
[0., -2., -3.],\
[0., -3., -5.],\
[0., -3., -6.]])]
But the code shuffle it. I also tried the foolowing for loops but it was not also successful because I can not iterate until the end of my data and stop the second for loop before reaching to the end of each array of my list.
clean_arr=[]
for arrays in dup_arr:
for rows in range (len(arrays)-1):
if np.all(arrays [rows]== arrays [rows+1]):
continue
else:
dat= arrays [rows]
clean_arr.append(dat)
In advance, I do appreciate any help and contribution.

You can simply use np.unique with axis=0. If you want to keep the order from the original sequence try this -
[i[np.sort(np.unique(i, axis=0, return_index=True)[1])] for i in dup_arr]
[array([[ 0., 10., 10.],
[ 0., 2., 30.],
[ 0., 3., 5.],
[ 0., 3., 40.]]),
array([[ 0., -1., -4.],
[ 0., -2., -3.],
[ 0., -3., -5.],
[ 0., -3., -6.]])]
np.unique(i, axis=0, return_index=True)[1] returns the indexes of the unique elements.
np.sort() sorts these indexes back to original sequence in array.
[f(i) for i in dup_arr] applies the above 2 steps over each element in dup_arr.
NOTE: You will NOT be able to completely vectorize this operation (say by np.stack on this operations since it will may have variable duplicates removed from each matrix. This will cause the numpy array to have unequal shapes over an axis.
Breaking the steps as a function -
def f(a):
indexes = np.unique(a, axis=0, return_index=True)[1]
return a[np.sort(indexes)]
[f(i) for i in dup_arr]

In my humble opinion, every time you want to remove duplicates from an array or list in Python, you should consider using a set.
Also, try to avoid using multiple nested loops since errors easily occur and they're hard to find. I suggest you give the following code a try:
removed_duplicates=[]
for subarr in dup_arr:
removed_duplicates.append(np.array([list(item) for item in set(tuple(row) for row in subarr)]))
Basically what's happening is that you convert your array to a tuple, then to a set that removes all duplicates, and then to a list. Since your original data had an array of np.arrays, your convert the list back to a np.array before you append it to the new array.
Would this work?

Python 3.x create 3D Volume with 2D slices

Python 3.x
I have for loop which is making some calculations and creating one Slice/2D Array lets say (x = 3, y = 3) per iteration and I want at the same time in the same for loop (append?/stack) them in a third dimension.
I have been trying with Numpy stack, vstack, hstack, dstack but I still don't get how to get them together in the 3rd dimension as I want.
So I would like to have at them end something like this:
(z = 10, x = 3, y = 3)
array([ [[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]],
[[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]],
[[2., 2., 2.],
[2., 2., 2.],
[2., 2., 2.]],
.
.
.
])
Thanks,

you can do it like this
arrays = []
for i in range(5):
arr = np.full((3,3), i)
arrays.append(arr)
np.asarray(arrays)
If you want to you can do np.asarray(arrays) inside loop. But it will be not very efficient. Not that np.concatenate will also effectively creates new numpy array so efficiency will be similar. Doing these operation once outside the loop is better

How to append ND numpy arrays to (N+1)D numpy array through loop?

For example I need 30x30 numpy arrays created from images to be fed to a neural net. If I have a directory of images to predict, I should be able to loop through the directory, get image data and create an (n,30,30) shape np array
This is my current method, I intend to reshape each row before feeding to the model
def get_image_vectors(path):
img_list=os.listdir(path)
print(img_list)
X=np.empty((900,))
for img_file in img_list:
img= Image.open(os.path.join(path,img_file))
img_grey= img.convert("L")
resized = img_grey.resize((30,30))
flattened = np.array(resized.getdata())
# print(flattened.shape)
X=np.vstack((X,flattened))
print(img_file,'=>',X.shape)
return X[1:,:]

Instead of appending to an existing array, it will probably be better to use a list initially, appending to it, and converting to an array at the end. thus saving many redundant modifications of np arrays.
Here a toy example:
import numpy as np
def get_image_vectors():
X= [] #Create empty list
for i in range(10):
flattened = np.zeros(900)
X.append(flattened) #Append some np array to it
return np.array(X) #Create array from the list
With result:
array([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]])