We Keep Coding

In regard with an image classification machine learning basic exercise in python

Sorry because this code contains comments about the exercise in Spanish. The thing is, I am trying to train 3 models with 3 different algorithms of machine learning which are in charge of image classification based on labels. I got to a point where, I have created a dataframe in pandas that contains, a column for the images, and a column for each image labels. At the beggining I am showing some code of how do I create the image, the thing is, I get them from a data file which is binary encoded. I use unpickle to uncode it, obtaining a dictionary with all the images. I build the images using np.dstack() with the 3 channels of color of the image, this is the input that is given for us. I think the problem is with the dimensions of the images because, each image in the panda's dataframe columns size's is (32x32x3), mainly because each image is 32x32 pixels, and contains 3 layers of color red green and blue. I am getting errors all the time regarding "ValueError: setting an array element with a sequence". I think I am no way near close to getting this right. I have tried using the function flatten but it seems like it does not work either. What should I do to adapt the input? I am very lost right now, have tried everything.
# Los datos de cada entidad contienen los valores de la imagen. La imagen se obtiene por la combinación de tres canales/capas (red, green, blue) de la siguiente forma:
ch0 = d0[0:1024]
ch1 = d0[1024:2048]
ch2 = d0[2048:]
# Cada canal es una capa del correspondiente color
ch0 = np.reshape(ch0, (32,32)) # red
ch1 = np.reshape(ch1, (32,32)) # green
ch2 = np.reshape(ch2, (32,32)) # blue
# La combinación de ellas da una imagen con los tres colores:
image = np.dstack((ch0, ch1, ch2))
fig, ax = plt.subplots(figsize=(2, 2))
ax.imshow(image)
plt.show()
import random as r
import cv2
categoriasAleatorias = []
for i in range(3):
num = r.randint(0,19)
categoriasAleatorias.append(dataMeta[b'coarse_label_names'][num])
print(categoriasAleatorias)
#Para cada imagen.
columnaImagenes = []
columnaEtiquetas = []
for i in range (len(data[b'data'])):
if dataMeta[b'coarse_label_names'][data[b'coarse_labels'][i]] in categoriasAleatorias:
columnaEtiquetas.append(dataMeta[b'coarse_label_names'][data[b'coarse_labels'][i]])
# Cada entidad, se encuentra en una posición de cada uno de los anteriores atributos
# Vamos a ver la entidad '0'
d0 = data[b'data'][i]
# Los datos de cada entidad contienen los valores de la imagen. La imagen se obtiene por la combinación de tres canales/capas (red, green, blue) de la siguiente forma:
ch0 = d0[0:1024]
ch1 = d0[1024:2048]
ch2 = d0[2048:]
# Cada canal es una capa del correspondiente color
ch0 = np.reshape(ch0, (32,32)) # red
ch1 = np.reshape(ch1, (32,32)) # green
ch2 = np.reshape(ch2, (32,32)) # blue
# La combinación de ellas da una imagen con los tres colores:
image = np.dstack((ch0, ch1, ch2))
columnaImagenes.append(image)
print(len(columnaEtiquetas))
print(len(columnaImagenes))
df = pd.DataFrame({'etiqueta': columnaEtiquetas, 'imagen': columnaImagenes})
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import MultiLabelBinarizer
X_train = df['imagen']
y_train = df['etiqueta']
# Crear el modelo SVM
svm = SVC()
# Entrenar el modelo SVM
svm.fit(X_train, y_train)
# Crear el modelo de Random Forest
rfc = RandomForestClassifier()
# Entrenar el modelo de Random Forest
rfc.fit(X_train, y_train)
# Crear el modelo de KNN
knn = KNeighborsClassifier()
# Entrenar el modelo de KNN
knn.fit(X_train, y_train)
flatten(), reshape() nothing seems to work
When I do this :
print(len(columnaEtiquetas))
print(len(columnaImagenes))
print(columnaImagenes[0].shape)
df = pd.DataFrame({'etiqueta': columnaEtiquetas, 'imagen': columnaImagenes})
print(df.head())
I obtain this:
7500
7500
(32, 32, 3)
etiqueta imagen
0 b'food_containers' [[[178, 168, 176], [175, 165, 173], [175, 165,...
1 b'trees' [[[254, 254, 254], [255, 255, 255], [255, 255,...
2 b'trees' [[[152, 71, 119], [154, 77, 124], [152, 84, 13...
3 b'trees' [[[153, 157, 168], [155, 160, 167], [162, 168,...
4 b'trees' [[[48, 78, 134], [51, 88, 148], [53, 86, 150],...
This is how my images are made,32x32pixels and 32x32x3 because there is 3 layers of color, I do not know how to fit this data or transform it in a way that the model gets it, tried your line of work to the image but kept giving me same exception.

Indeed, none of the methods you use would work for 32x32x3. If you want to fully utilise the spatial information from the image, you should use convolutions. But this is the more advanced technique.
What you would need is to have the image arrays flattened into an array with 3072 features. I experimented in a shell and if you do columnaImagenes.append(image.flatten().tolist()) this should give you 3072 sized feature arrays.
If it's not working, I'd suggest to take an element from the column and check it's shape with np.shape
But also there's a part of your code where you do the color to grayscale convertion (cvt_color). If I understand correctly you are trying to convert the image as BGR where you are actually concatenating is as RGB, which is a different order and grayscale values would actually differ.

SpaCy - TextCategorizer - Bag Of Words: Is there a way to show the vectorized document?

I just trained and implemented a text categorizer using Space 3.0. Everything went smooth but I'd like to visualize the vectorized document ([13, 0, 0, 120..etc]) in order to better understand what feature (words) drove the bag-of-words (BoW) model to classify the document in a specific class.
nlp = spacy.load('./nlp_single_label_cli/output/model-best')
documents = pd.read_csv(target_directory+'_ocr.csv')
...
test_texts = documents['text'].values
test_docs = [nlp.tokenizer(text) for text in test_texts]
text_categorizer = nlp.get_pipe('textcat')
scores = text_categorizer.predict(test_docs)
predicted_labels = scores.argmax(axis=1)
I've created the model from scratch with just the TextCategorizer layer (following the space 3.0 guidelines). By doing this, the documents have no .vector attribute.
this is my config.cfg
[nlp]
lang = "it"
pipeline = ["textcat"]
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 1000
tokenizer = {"#tokenizers":"spacy.Tokenizer.v1"}
[components]
[components.textcat]
factory = "textcat"
scorer = {"#scorers":"spacy.textcat_scorer.v1"}
threshold = 0.5
[components.textcat.model]
#architectures = "spacy.TextCatEnsemble.v2"
nO = null
[components.textcat.model.linear_model]
#architectures = "spacy.TextCatBOW.v2"
exclusive_classes = true
ngram_size = 1
no_output_layer = false
nO = null
[components.textcat.model.tok2vec]
#architectures = "spacy.Tok2Vec.v2"
...

In SpaCy you can use .vector to get the vector representation of a document.
import spacy
nlp = spacy.load("en_core_web_sm")
doc1 = nlp("I like apples.")
doc2 = nlp("I like bananas.")
doc3 = nlp("The book is about quantum physics.")
print(doc1.vector)
> [-0.28571516 ... 0.77876693]
print(doc2.vector)
> [ 3.14834714e-02 ... 5.22061586e-01]
print(doc3.vector)
> [ 0.08713525 ... 0.2642293 ]
Visualize these vectors for example by using PCA in sklearn to reduce the dimensions to two, then create a basic plot:
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import numpy
pca_ = PCA(n_components=2).fit_transform(numpy.array([doc.vector for doc in [doc1, doc2, doc3]]))
pca_x = tsne_x = [x_[0] for x_ in pca_]
pca_y = [x_[1] for x_ in pca_]
ax = plt.gca()
plt.scatter(pca_x, pca_y, alpha=0.25)
ax.set_xlim([-3, 3])
ax.set_ylim([-3, 3])
for i, name in enumerate(["doc1", "doc2", "doc3"]):
text = ax.annotate(name, (pca_x[i], pca_y[i]))
plt.show()
Which will give you:
I am not sure if this exactly what you meant though, if you do not mean the generic vector representation of each document but the feature vectors or something similar you can do the same thing and use that instead of .vector. Of course by reducing the dimensions you will not be able see the individual vector values in the plot afterwards.

Where is the real fault in my Noverov Alogorithm + Schroedinger Equation?[Physics]

I am solving Schroedinger Equation with help of the Noverov Theorom. So, My initial project question is: Solve s-wave Schroedinger Equation for the ground state and first excited state of the hydrogen atom:
D^2y+2(E-V(r))y=0 , D=d/dr
and potential is V(r)=-1/r
But I want to make the program a program which automatically calculate it's eigen energies and using hatree units and reconsidering it: here is my code :
import numpy as np
import matplotlib.pyplot as plt
dom=[0.001,5]
n=10
r,dr=np.linspace(dom[0],dom[1],n,retstep=True)
def Potential(x,E):
energy=2*(E+(1/x))
return energy
i=0
E=-2
s0i=0.0
s1i=0.0001
si=np.zeros(n)
indi=0
while i==0:
si[0]=s0i
si[1]=s1i
for j in range(0,n-2,1):
f0c=((dr**2)*Potential(x=r[j],E=E))/12.0
f1c=((dr**2)*Potential(x=r[j+1],E=E))/12.0
f2c=((dr**2)*Potential(x=r[j+2],E=E))/12.0
s2i=((2*(1-5*f1c)*s1i)-((1+f0c)*s0i))/(1+f2c)
s1i=s2i
s0i=s1i
si[j+2]=s2i
if abs(s2i)<=1e-18:
i=1
elif s2i>1e-18:
E=E-0.1
elif s2i<-1e-18:
E=E+0.1
elif(indi==500):
i=1
print('loop Count : ',indi,'Energy : ',E,'last Pshi',si[-1])
indi+=1
print(len(r),len(si))
plt.plot(r,si)
plt.show()
I didn't understand where is the issue and I got some hilarious result. So Please help me to do so .

I see you speak dutch. So I hope it is good that I answer in my native language.
Je weet dat er een sympy module is waarmee je het ook kan oplossen? Het is al geschreven code die het voor je oplost.
from sympy.physics.hydrogen import E_nl
from sympy.abc import n, Z
E_nl(n, Z) # output: -Z**2/(2*n**2)
# Z = atoomnummer n = het quantum getal
# Z=1 (hydrogen) bij default kan je veranderen trouwens
E_nl(1) # output: -1/2
E_nl(2) # output: -1/8
E_nl(3) # output: -1/18
E_nl(3, 47) # output: -2209/18
Maar uiteraard is het ook interessant om je eigen code te schrijven.
In je code ontstaat er een overflow
aangezien het deze errror geeft:
RuntimeWarning: overflow encountered in double_scalars
en als je kijkt naar de antwoorden is dat niet heel raar.
loop Count : 78 Energy : -2.1 last Pshi 1.9967193884115876e+306
loop Count : 79 Energy : -2.1 last Pshi NaN
dat betekent dus een 1 met 306 nullen er achter.... erg groot getal, en het wijkt steeds verder van van de verwachte -1/2. Dus ergens loopt ie verkeerd door jouw berekening heen. Zou je de berekening duidelijker kunnen uitschrijven?

Build a correlation circle with Python - Error ValueError: could not broadcast input array from shape (3) into shape (28)

I'm trying to build a correlation circle, basically, it allows to measure to which extend the Eigenvalue / Eigenvector of a variable is correlated to the principal components (dimensions) of a dataset.
Something like this :
Here is my code :
import pandas as pd
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
#chargement données
X = pd.read_excel("mortalitePaysUE.xlsx",sheet_name=0,header=0,index_col=0)
#nombre d'observations
n = X.shape[0]
#nombre de variable
p = X.shape[0]
print(p)
#transformation - centrage/réduction
sc = StandardScaler()
Z = sc.fit_transform(X)
print(Z)
print("-------------")
#moyenne
print("Moyenne : ")
print(np.mean(X,axis=0))
print("-------------")
#ecart-type
print("Ecart type : ")
print(np.std(X,axis=1,ddof=0))
print("-------------")
#acp
acp = PCA(svd_solver='full')
coord = acp.fit_transform(Z)
eigval = (n-1)/n*acp.explained_variance_
print(eigval)
#screen plot
#plt.plot(np.arange(1,p+1),eigval)
#plt.title("Décès en 1990 selon le genre")
#plt.xlabel("Numéro de facteur")
#plt.ylabel("Valeur propre")
#plt.show()
#positionnement des individus dans le premier plan
fig, axes = plt.subplots(figsize=(12,12))
axes.set_xlim(-6,6) #même limites en abscisse
axes.set_ylim(-6,6) #et en ordonnée
#placement des étiquettes des observations
for i in range(n):
plt.annotate(X.index[i],(coord[i,0],coord[i,1]))
#ajouter les axes
plt.plot([-6,6],[0,0],color='silver',linestyle='-',linewidth=1)
plt.plot([0,0],[-6,6],color='silver',linestyle='-',linewidth=1)
#affichage
plt.show()
#racine carrée des valeurs propres
sqrt_eigval = np.sqrt(eigval)
#corrélation des variables avec les axes
corvar = np.zeros((p,p))
for k in range(p):
corvar[:,k] = acp.components_[k,:] * sqrt_eigval[k]
#afficher la matrice des corrélations variables x facteurs
#print(corvar)
#cercle des corrélations
fig, axes = plt.subplots(figsize=(8,8))
axes.set_xlim(-1,1)
axes.set_ylim(-1,1)
#affichage des étiquettes (noms des variables)
for j in range(p):
plt.annotate(X.columns[j],(corvar[j,0],corvar[j,1]))
#ajouter les axes
plt.plot([-1,1],[0,0],color='silver',linestyle='-',linewidth=1)
plt.plot([0,0],[-1,1],color='silver',linestyle='-',linewidth=1)
#ajouter un cercle
cercle = plt.Circle((0,0),1,color='blue',fill=False)
axes.add_artist(cercle)
The problem is that i got an error and i can't display the circle. And i can't resolve the error
corvar[:,k] = acp.components_[k,:] * sqrt_eigval[k]
ValueError: could not broadcast input array from shape (3) into shape (28)
Can anyone help me to fix this please :) Thanks in advance !

Estimation of number of Clusters via gap statistics and prediction strength

I am trying to translate the R implementations of gap statistics and prediction strength http://edchedch.wordpress.com/2011/03/19/counting-clusters/ into python scripts for the estimation of number of clusters in iris data with 3 clusters. Instead of getting 3 clusters, I get different results on different runs with 3 (actual number of clusters) hardly estimated. Graph shows estimated number to be 10 instead of 3. Am I missing something? Can anyone help me locate the problem?
import random
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
def dispersion (data, k):
if k == 1:
cluster_mean = np.mean(data, axis=0)
distances_from_mean = np.sum((data - cluster_mean)**2,axis=1)
dispersion_val = np.log(sum(distances_from_mean))
else:
k_means_model_ = KMeans(n_clusters=k, max_iter=50, n_init=5).fit(data)
distances_from_mean = range(k)
for i in range(k):
distances_from_mean[i] = int()
for idx, label in enumerate(k_means_model_.labels_):
if i == label:
distances_from_mean[i] += sum((data[idx] - k_means_model_.cluster_centers_[i])**2)
dispersion_val = np.log(sum(distances_from_mean))
return dispersion_val
def reference_dispersion(data, num_clusters, num_reference_bootstraps):
dispersions = [dispersion(generate_uniform_points(data), num_clusters) for i in range(num_reference_bootstraps)]
mean_dispersion = np.mean(dispersions)
stddev_dispersion = float(np.std(dispersions)) / np.sqrt(1. + 1. / num_reference_bootstraps)
return mean_dispersion
def generate_uniform_points(data):
mins = np.argmin(data, axis=0)
maxs = np.argmax(data, axis=0)
num_dimensions = data.shape[1]
num_datapoints = data.shape[0]
reference_data_set = np.zeros((num_datapoints,num_dimensions))
for i in range(num_datapoints):
for j in range(num_dimensions):
reference_data_set[i][j] = random.uniform(data[mins[j]][j],data[maxs[j]][j])
return reference_data_set
def gap_statistic (data, nthCluster, referenceDatasets):
actual_dispersion = dispersion(data, nthCluster)
ref_dispersion = reference_dispersion(data, nthCluster, num_reference_bootstraps)
return actual_dispersion, ref_dispersion
if __name__ == "__main__":
data=np.loadtxt('iris.mat', delimiter=',', dtype=float)
maxClusters = 10
num_reference_bootstraps = 10
dispersion_values = np.zeros((maxClusters,2))
for cluster in range(1, maxClusters+1):
dispersion_values_actual,dispersion_values_reference = gap_statistic(data, cluster, num_reference_bootstraps)
dispersion_values[cluster-1][0] = dispersion_values_actual
dispersion_values[cluster-1][1] = dispersion_values_reference
gaps = dispersion_values[:,1] - dispersion_values[:,0]
print gaps
print "The estimated number of clusters is ", range(maxClusters)[np.argmax(gaps)]+1
plt.plot(range(len(gaps)), gaps)
plt.show()

Your graph is showing the correct value of 3. Let me explain a bit
As you increase the number of clusters, your distance metric will certainly decrease. Therefore you are assuming that the correct value is 10. If you increase it to beyond 10, the distance metric will further decrease. But this should not be our decision making criteria
We need to find the inflection point ( here marked in RED ). It is the point where the slope smoothens out. You might want to take a look at elbow curves
Based on the above 2 points, the inflection point is 3 ( which is also the correct solution )
Hope this helps

you could take a look on this code and you could change your output plot format
[![# coding: utf-8
# Implémentation de K-means clustering python
#Chargement des bibliothèques
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from sklearn import datasets
#chargement de jeu des données Iris
iris = datasets.load_iris()
#importer le jeu de données Iris dataset à l'aide du module pandas
x = pd.DataFrame(iris.data)
x.columns = \['Sepal_Length','Sepal_width','Petal_Length','Petal_width'\]
y = pd.DataFrame(iris.target)
y.columns = \['Targets'\]
#Création d'un objet K-Means avec un regroupement en 3 clusters (groupes)
model=KMeans(n_clusters=3)
#application du modèle sur notre jeu de données Iris
model.fit(x)
#Visualisation des clusters
plt.scatter(x.Petal_Length, x.Petal_width)
plt.show()
colormap=np.array(\['Red','green','blue'\])
#Visualisation du jeu de données sans altération de ce dernier (affichage des fleurs selon leur étiquettes)
plt.scatter(x.Petal_Length, x.Petal_width,c=colormap\[y.Targets\],s=40)
plt.title('Classification réelle')
plt.show()
#Visualisation des clusters formés par K-Means
plt.scatter(x.Petal_Length, x.Petal_width,c=colormap\[model.labels_\],s=40)
plt.title('Classification K-means ')
plt.show()][1]][1]
Output 1