We Keep Coding

CNN in python. Array of images

I was trying to deeply search for a solution online before posting here, but I couldn't find it. My problem arises during the reading of images in a training of a convolutional neural network. Basically, I decided to create a function creating the in values and the out values out of a series of images. I want to read all the images of the set, but not all at the same time, to avoid running out of memory, so I create the next function:
def readImages(strSet = 'Train', nIni = 1, nFin = 20):
if strSet not in ('Train','Test'):
return None
#
# Inicializamos los arrays de salida: las imágenes y las etiquetas.
arrImages = []
arrLabels = []
#
# Recorremos todos y cada uno de los directorios dentro del set elegido
for strDir in os.listdir(data_dir+'/' + strSet + '/'):
# Nombre de la clase que estamos tratando.
strClass = strDir[strDir.find('-')+1:]
# Número y nombre de los ficheros, por si es menor que el número n indicado.
arrNameFiles = os.listdir(data_dir+'/' + strSet + '/'+strDir)
nFiles = len(os.listdir(data_dir+'/' + strSet + '/'+strDir))
#
# Cogemos los ficheros desde el nIni al nFin. De esta forma nos aseguramos los cogemos todos en cada directorio.
#print('nImagesClase(',strSet,',',strClass,'):',nImagesClase(strSet, strClass))
if (nIni == -1):
# Si el valor es -1, cogemos todas las imágenes del directorio.
listChosenFiles = arrNameFiles
#print('Todos: ', len(listChosenFiles))
else:
if (nImagesClase(strSet, strClass)<nFin):
# Si ya hemos dado la vuelta a todos los ficheros del grupo, los cogemos al azar.
listChosenFiles = random.sample(arrNameFiles, min(nFiles, nFin-nIni))
#print('Fin del directorio ',nFin,'>',nImagesClase(strSet,strClass),': ', len(listChosenFiles))
else:
# Si no, seguimos.
listChosenFiles = arrNameFiles[nIni-1:min(nFin,nImagesClase(strSet, strClass))-1]
#print('Seguimos ',nIni,'-',nFin,': ', len(listChosenFiles))
#
for file in listChosenFiles:
# Lectura del fichero.
image = plt.imread(data_dir+'/'+strSet+'/'+strDir+'/'+file)
#print('Original Shape: ',image.shape)
#plt.imshow(image)
image = cv2.resize(image, (crop_width, crop_height), interpolation=cv2.INTER_NEAREST)
#image = image.reshape((image_height,image_width,num_channels))
#print('Al array de imágenes: ',image.shape)
arrImages.append(image)
# Añadimos etiquetas.
arrLabel = np.zeros(n_classes)
arrLabel[array_classes.index(strClass)] = 1
arrLabels.append(arrLabel)
#
# Recogemos los valores de entrada y salida en arrays.
y = np.array(arrLabels)
X = np.array(arrImages, dtype=np.uint8)
# Una vez terminado el recorrido por todas las imágenes, reordenamos los índices para que no vayan las imágenes en secuendias de la misma clase.
arrIndexes = np.arange(X.shape[0])
np.random.shuffle(arrIndexes)
X = X[arrIndexes]
y = y[arrIndexes]
#
return X, y
To test the behavior of this function I just execute the following line.
X, y = readImages(strSet = 'Train', nIni = 1, nFin = 5)
Which is ok, until the moment nIni and nFin reach some values (101-105, for example). In that moment, I receive the following error:
ValueError Traceback (most recent call last)
<ipython-input-125-8a690256a1fc> in <module>
----> 1 X, y = readImages(strSet = 'Train', nIni = 101, nFin = 105)
<ipython-input-123-9e9ebc660c33> in readImages(strSet, nIni, nFin)
50 # Recogemos los valores de entrada y salida en arrays.
51 y = np.array(arrLabels)
---> 52 X = np.array(arrImages, dtype=np.uint8)
53 # Una vez terminado el recorrido por todas las imágenes, reordenamos los índices para que no vayan las imágenes en secuendias de la misma clase.
54 arrIndexes = np.arange(X.shape[0])
ValueError: could not broadcast input array from shape (28,28,3) into shape (28,28)
I put some print traces in the reading of the images, and everyone of the read images has a shape of (28,28,3), so I don't really understand from where do I have this (28,28) shape pointed out in the error trace.
Do you know what could be the problem? Did you face this problem earlier?
Thanks in advance.

Some of your images have single channels. Use cv2.imread instead of plt.imread
image = cv2.imread(data_dir+'/'+strSet+'/'+strDir+'/'+file)

sort txt list by element

i have an issue with a .txt list,
the list contains the next:
Numero de permutaciones de la forma (2,4,3) = 1260
Numero de permutaciones de la forma (7,2,0) = 36
Numero de permutaciones de la forma (5,3,1) = 504
Numero de permutaciones de la forma (4,5,0) = 126
Numero de permutaciones de la forma (1,8,0) = 9
Numero de permutaciones de la forma (0,7,2) = 36
Numero de permutaciones de la forma (0,6,3) = 84
...
my code is this:
with open('resultado_lista_original2.txt', 'r') as r:
for line in sorted(r):
print(line,end='')
but i need to sort that list by the element next to right the "=" to get this order
Numero de permutaciones de la forma (1,8,0) = 9
Numero de permutaciones de la forma (0,7,2) = 36
Numero de permutaciones de la forma (7,2,0) = 36
Numero de permutaciones de la forma (0,6,3) = 84
Numero de permutaciones de la forma (4,5,0) = 126
Numero de permutaciones de la forma (5,3,1) = 504
Numero de permutaciones de la forma (2,4,3) = 1260
...
I appreciate very much who can help me / guide me

Pass in the key argument to decide how to sort your iterable. Most people do this using a lambda function:
for line in sorted(r, key=lambda x: int(x.split('=')[1])):
Or if you prefer to define the function yourself:
def sort_my_txt_lines(line):
digits_after_equal_sign = line.split('=')[1]
return int(digits_after_equal_sign)
# ...
for line in sorted(r, key=sort_my_txt_lines):
If you're not used to think about sorting using a key function, think about it like this: when trying to sort the input sequence, only look at the numerical part beyond the = sign (the function's return value) of each line (the function's input parameter).

The sorted method has an attribute key where you can put a lambda or a method which will be called with each item of the list that you want to sort.
The result of that will be used for to sort the list instead of the item itself.

Late answer, but you can also use:
import re
with open("input.txt")as f:
s = sorted(f, key=lambda x:int(re.search(r"(\d+)$", x).group(1)))
Demo

Try this:
for line in sorted(r, key=lambda x: int(x.split(' = ').strip())):

Errors with pickling objects

We're making an exercise with pickle, and this code doesn't work as it is supposed to. Please help me, this is the code (some words are in Spanish because I'm from America):
import pickle
class persona:
def __init__(self, nombre, genero, edad):
self.nombre = nombre
self.genero = genero
self.edad = edad
print("se ha creado una persona nueva con el nombre de: ", self.nombre)
def __str__(self):
return "{} {} {}".format(self.nombre, self.genero, self.edad)
class listaPersonas:
personas = []
def __init__(self):
listaDePersonas = open("ficheroExterno", "ab+")
listaDePersonas.seek(0)
try:
self.personas = pickle.load(listaDePersonas)
print("Se cargaron {} personas del fichero externo".format(len(self.personas)))
except:
print("El fichero está vacío")
finally:
listaDePersonas.close()
del(listaDePersonas)
def agregarPersonas(self, p):
self.personas.append(p)
self.guardarPersonasEnFicheroExterno()
def mostrarPersonas(self):
for p in self.personas:
print(p)
def guardarPersonasEnFicheroExterno(self):
listaDePersonas = open("ficheroExterno", "wb")
pickle.dump(self.personas, listaDePersonas)
listaDePersonas.close()
del(listaDePersonas)
def mostrarInfoFicheroExterno(self):
print("La información sle fichero externo es la siguiente: ")
for p in self.personas:
print(p)
miLista = listaPersonas()
persona = persona("Sandra", "Femenino", 29)
miLista.agregarPersonas(persona)
miLista.mostrarInfoFicheroExterno()
and it throws that:
El fichero está vacío
se ha creado una persona nueva con el nombre de: Sandra
Traceback (most recent call last):
File "Guardado_permanente.py", line 54, in <module>
miLista.agregarPersonas(persona)
File "Guardado_permanente.py", line 34, in agregarPersonas
self.guardarPersonasEnFicheroExterno()
File "Guardado_permanente.py", line 42, in guardarPersonasEnFicheroExterno
pickle.dump(self.personas, listaDePersonas)
_pickle.PicklingError: Can't pickle <class '__main__.persona'>: it's not the same object as __main__.persona
***Repl Closed***
I have like 1 1/2 hour that I'm seeing this code and I'm trying to guess which is the problem, but the code is the same as my teacher's code. please, help me. I'm using Sublime text to code.

In this line, you have replaced your class persona with an instance of the class:
persona = persona("Sandra", "Femenino", 29)
pickle is trying to find the class definition for persona but can't, because it doesn't have a name anymore.
Don't try to use the same name for two things; only the last assignment counts. Standard style is to use CamelCase names for clasess, so you could name your class Persona instead.

ValueError: 0 is not in list when running this code

I'm currently working on a little side project yet having multiple issues. I'm reading a file within the folder where the project is that holds data for 10 users.
As for the code itself...it's pretty big.
def ouvrir_fichier(nomFichier):
""" Ne pas oublier les docstring
"""
try:
fp = open(nomFichier, 'r')
return fp
except:
return print("Le fichier n'existe pas. Veuillez réessayer.")
def lire_fichier(fp):
""" Ne pas oublier les docstring"""
# Lis n et initialise une liste vide --
# où il y a une liste vide pour chaque usager du réseau
# ensuite lis le reste du fichier et ajouter l'information à reseau
liste1 = fp.readlines()
n = int(liste1[0])
liste2 = liste1[1:]
reseau = [[] for i in range(n)]
for i in liste2:
i = i.replace("\n", "")
data = i.split(" ")
valeur1 = int(data[0])
valeur2 = int(data[1])
reseau[valeur1].append(valeur2)
reseau[valeur2].append(valeur1)
fp.close()
return reseau
def trouver_nombre_elements_communs_entre_listes(liste1, liste2):
""" Ne pas oublier les docstring"""
compteur_amis_commun = 0
for element in liste1:
if element in liste2:
compteur_amis_commun = compteur_amis_commun + 1
return compteur_amis_commun
def initialiser_matrice(n):
"""
Crée une matrice nxn, initialisée avec des zéros et retourne la matrice.
Args:
n (int): dimension de la matrice nxn
Returns:
matrice (list): matrice initialisée
"""
matrice = []
for ligne in range(n): # pour chacune des lignes dans n
matrice.append([]) # créer une ligne (liste) et l'initialiser à 0
for colonne in range(n):
matrice[ligne].append(0) # ajouter un 0 pour chaque n colonne
return matrice
def calculer_scores_similarite(reseau):
""" Ne pas oublier les docstring"""
n = len(reseau)
matrice_similarite = initialiser_matrice(n)
liste1 = []
liste2 = []
compteur_liste1 = 0
compteur_liste2 = 0
for element_liste1 in reseau:
liste1 = element_liste1
for element_liste2 in reseau:
liste2 = element_liste2
compteur_amis_commun = trouver_nombre_elements_communs_entre_listes(liste1, liste2)
matrice_similarite[compteur_liste1][compteur_liste2] = compteur_amis_commun
compteur_liste2 = compteur_liste2 + 1
compteur_liste1 = compteur_liste1 + 1
compteur_liste2 = 0
return matrice_similarite
def recommander(id_usager,reseau,matrice_similarite):
""" Ne pas oublier les docstring"""
usager_matrice = matrice_similarite.index(id_usager)
ami_recommande = matrice_similarite.index(max(usager_matrice))
max_value = max(matrice_similarite.index(usager_matrice))
if ami_recommande == id_usager:
max_value = max_value - 1
ami_recommande = matrice_similarite.index(max_value)
while True:
if ami_recommande == reseau.index(ami_recommande):
ami_recommande = reseau.index(max_value, ami_recommande + 1)
return True
return ami_recommande
def main():
nomFichier = input("Nom du fichier contenant le réseau: ")
reseau = lire_fichier(ouvrir_fichier(nomFichier))
n = len(reseau)
matrice_similarite = calculer_scores_similarite(reseau)
while True:
while True:
id_usager = int(input("Entrer l'ID de l'usager pour lequel vous voulez une recommandation (entre 0 et {}):".format(n)))
if 0 <= id_usager and id_usager < n:
calculer_scores_similarite(reseau)
print("Pour la personne" , id_usager , ", nous recommandons l'ami" , recommander(id_usager, reseau, matrice_similarite))
continue
else:
print("Erreur: l'usager doit être un nombre entier entre ", 0, "et", n - 1, "inclusivement.\n")
autreRecommandation = input("Voulez-vous une autre recommandation (oui/non)?")
if autreRecommandation.lower() == "oui":
return True
else:
print("Merci d'avoir utiliser le programme de recommandation d'amis.")
break
if __name__ == "__main__":
main()
Most of the content seems to be working fine until I get to part where I need to recommend a user identification. I'll try to work on the doc string as well in the meantime but I could totally use a little bit of help as to debug this. I tested most of the code on another .py project until I hit the function recommander
First Edit:
I did forget to apply the return. I changed it and it is now in the def. Now however...I seem to be having this error.
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3.2\helpers\pydev\pydevd.py", line 1668, in <module>
main()
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3.2\helpers\pydev\pydevd.py", line 1662, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3.2\helpers\pydev\pydevd.py", line 1072, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3.2\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/101136/PycharmProjects/tp2/TP2.py", line 132, in <module>
main()
File "C:/Users/101136/PycharmProjects/tp2/TP2.py", line 119, in main
print("Pour la personne" , id_usager , ", nous recommandons l'ami" , recommander(id_usager, reseau, matrice_similarite))
File "C:/Users/101136/PycharmProjects/tp2/TP2.py", line 89, in recommander
usager_matrice = matrice_similarite.index(id_usager)
ValueError: 0 is not in list
Also I'm wondering about a little something. The part where I ask for an input to define id_usager in the def main(), I was wondering if there was a way to also treat characters as well as integers. Integers, I got it covered with my code but if somebody wants to write boy instead of a number, it'll fail.

accessing element of list is done by list_name[index], list_name.index(element) returns the index of that element,if they are two in the list it returns the index of the first one, if the element is not there,it throws a ValueError: "element" is not in the list.
in your case
change
usager_matrice = matrice_similarite.index(id_usager)
to
usager_matrice =matrice_similarite[id_usager]
and other places you want to access element of a list using index
in the scond question from the comment, check if its numeric like this
id_usager =input("Entrer l'ID de l'usager pour lequel vous voulez une recommandation (entre 0 et {}):".format(n))
if not id_usager.isdigit():
continue
id_user=int(id_usager)
note that this will convert also floats into int

Python, database results are saved with ('result')

I select some urls from my database using a simple code
def crearArchivo(self):
#Se conecta a la base de datos
db = MySQLdb.connect("localhost","root","","proyectoacademias" )
cursor = db.cursor()
#Selecciona la lista de valores que no hayan sido procesados
sql = "SELECT url_paper_web FROM acw_papers_web WHERE finalizado = 0"
cursor.execute(sql)
#Recibe todos los datos
datos = cursor.fetchall()
#Crea el archivo
archivo = open('urls.txt','w')
archivo.close()
#Lo abre
archivo = open('urls.txt','w')
#Establece un contador para determinar el numero de filas
contador=0;
#Para cada fila la guarda en el archivo y si no es la ultima agrega un salto de linea
for columna in datos:
contador+=1;
archivo.write(str(columna))
if(contador!=len(datos)):
archivo.write("\n")
#Se cierra todo
archivo.close()
db.close()
This method places those links in a file... the problem is that those links instead of being saved like http:// google.com they are saved like ('http://google.com')

Each columna variable is actualy a row, not a single column. Simply select the first value in that row:
archivo.write(columna[0])

That is because columna is a tuple. It looks like you want to write the first (and in this case, the only element):
archivo.write(columna[0])