I'm working on the implementation of several algorithms to compute shortest paths on graphs.
I have managed to implement Dijkstra's algorithm sequentially and I'm now trying to optimize my algorithm through the multiprocessing module of Python.
As a whole the code works. What I am trying to do here is :
First to check how many cpus I can work on with nb_cpu = mp.cpu_count()
Then dividing all the nodes I have in my graph accordingly
Finally calling the method subprocess_dijkstra that should compute the dijkstra algorithm for each of the nodes it is given as an argument (the idea being that each process only has to compute the algorithm for a smaller part of the graph).
When I run my script (called from a main.py file where I just format the data to suit my needs), I have 4 processes launched as I should.
However, they do not seem to execute the for node in nodes loop defined in subprocess_dijkstra.
Each process only computes the code once and then they go on hold indefinitely...
It is my first attempt at multiprocessing under Python so I may have missed a detail. Does anybody have an idea ?
When I interrupt the script, python tells me that the interruption takes place on the p.join() line.
Thanks to anyone helping me :)
Here is my code :
import multiprocessing as mp
def subprocess_dijkstra(do_print, nodes, tab_contenu, tab_distances):
tab_dist_initial = dict(tab_distances)
tab_dist = dict()
for node in nodes:
visited_nodes = list()
tab_dist = dict(tab_dist_initial)
dmin = -1
resultat = ""
filename = "dijkstra"+str(node)+".txt"
if do_print:
dt = open(filename, 'w')
tab_dist[node] = 0
"""Ligne de résultat initiale"""
for valeur in tab_dist.values():
resultat += str(valeur)
resultat += " "
resultat += "\n"
dt.write(resultat)
while len(visited_nodes) != len(tab_contenu):
""" On se place sur le noeud non visité qui a la distance minimale de notre départ """
for cle, valeur in tab_dist.items():
if cle not in visited_nodes:
if dmin ==-1 or valeur<dmin:
dmin = valeur
node = cle
""" On vérifie que le noeud n'a pas déjà été visité """
if (node not in visited_nodes):
""" On regarde les fils de ce noeud et la longueur des arcs"""
for cle,valeur in tab_contenu[node].items():
tab_dist[cle] = min(tab_dist[cle], tab_dist[node]+valeur)
visited_nodes.append(node)
if do_print:
resultat = ""
""" Ligne de résultat """
for valeur in tab_dist.values():
resultat += str(valeur)
resultat += " "
resultat += "\n"
dt.write(resultat)
if do_print:
dt.close()
def main(do_print,donnees):
tab_contenu = donnees[1]
nb_nodes = int(donnees[0])
tab_distances = {x: float('inf') for x in range(nb_nodes)}
args=[(do_print, x, tab_contenu, tab_distances) for x in range(nb_nodes)]
nb_cpu = mp.cpu_count()
pool = mp.Pool(processes = nb_cpu)
pool.starmap(subprocess_dijkstra, args)
pool.close()
pool.join()
I have found the source of my problems.
The tab_dist[node] = 0 was misplaced and should have been put before the if do_print: statement.
All is now working.
Related
I made a script for organizing torrents. I download tv shows and then move files to another disk to a folder like /series name/season xx/
It was working with some issues until I've added a couple of ifs. I saw the problem of case sensitivity. For example, If i download mr robot.mkv and the folder Mr Robot already existed it created a different folder as mr robot. I did the same with season not to double move Season if season already exists.
Apparently my script is trying to create the folder continuosly and I don't see why.
#!/usr/bin/env python3
import sys, glob, re, os, shutil
from termcolor import colored
#enconding: utf-8
dir_series = "/home/user/series/series/"
buscar = "*[sS][0-9][0-9]*"
series = [s for s in glob.glob(buscar) if s.endswith(('.mp4', '.srt', '.avi', '.mkv'))]
if series:
arch_encontrados = len(series)
print(colored("\nArchivos encontrados:",'red', attrs=['bold'] ), colored(arch_encontrados, 'red', attrs=['bold'] ),'\n')
print(*series, sep = "\n")
for serie in series:
#Extraer el nombre de la serie
nombre = re.findall(r'.*[\. ][sS]\d', serie)[0]
nombre_final = re.sub(r'[\. ][sS]\d','',nombre).replace('.',' ')
#Extraer el número de la temporada
season = re.findall(r'[\. ][sS]\d\d', serie)[0]
season_final_numero = re.sub(r'[\. ][sS]','',season)
season_final = ('Season ' + season_final_numero)
#Armar el directorio final
for series_path in os.listdir(dir_series): #lista el contenido de /home/user/series/series/
if nombre_final.lower() == series_path.lower(): #compara el listado con la salida del nombre de la serie sin importar mayúsculas y minúsculas
for season_path in os.listdir(dir_series + series_path):
if season_final == season_path: #compara el listado de seasons contra season_final que tiene mayuscula
path = os.path.join(dir_series, series_path, season_final)
print(path)
else:
path = os.path.join(dir_series, series_path, 'season ', season_final_numero)
else:
print(colored("\n\n*****************************************",'cyan', attrs=['bold']))
print(colored("** Directorio no encontrado, creándolo **",'cyan', attrs=['bold']))
print(colored("*****************************************\n",'cyan', attrs=['bold']))
path = os.path.join(dir_series, nombre_final, season_final)
print(path)
os.makedirs(path)
#Mover el archivo
print(colored('\nCopiando','green'), serie, colored('a', 'green'), path + '/' + serie)
shutil.move(serie,path)
else:
print(colored('\nNo hay archivos para organizar.','green', attrs=['bold']))
input(colored("\n\nPresione Enter para continuar ...", attrs=['blink', 'bold']))
I'm not seeing an infinite loop, but I think I do see a bug that is causing the same directory to get made many times.
You are calling os.makedirs in the else of your inner for loop, which means that you will make the same directory once for each file in os.listdir(dir_series) that does NOT match nombre_final.lower().
I think the issue might just be that you (or your IDE) accidentally indented the os.makedirs(path) call two levels too deep when you added the if/else. I think it needs to be outside of the inner loop entirely.
You probably also need to add a break in the case where it does match, and maybe also a guard to stop it from making a new directory in the case where a match was found?
I was looking to organize my torrents once they're downloaded. I wrote this script that checks for series name and seaons and moves files to other disk where I keep my tv shows.
I want some processes to be printed as they run. I'm having a problem with one of them.
I want first to print "Archivos encontrados" (which means Files founded) and then print the variable with all the files from the directory (in this case the variable is called "series")
The problem is that as I wrote it, it print one Archivos encontrados for each file it finds.
I've tried as you can see in line 21 to check for end of file but it doesn't work.
Also, the else at the end that should run if it can't find any of the extensions I've declared is not working.
Thanks in advance
#!/usr/bin/env python3
import sys, glob, re, os, shutil
from termcolor import colored
#enconding: utf-8
dir_series = "/home/user/series/series/"
buscar = "*[sS][0-9][0-9]*"
for serie in glob.glob(buscar):
if serie.endswith(('.mp4', '.srt', '.avi', '.mkv')):
#Extraer el nombre de la serie
nombre = re.findall(r'.*[\. ][sS]\d', serie)[0]
nombre_final = re.sub(r'[\. ][sS]\d','',nombre).replace('.',' ')
#Extraer el número de la temporada
season = re.findall(r'[\. ][sS]\d\d', serie)[0]
season_final = re.sub(r'[\. ][sS]','',season)
#if serie == serie[-1]:
print(colored("Archivos encontrados: ",'red'))
print(serie)
#Armar el directorio final
path = os.path.join(dir_series, nombre_final, ('Season '+ season_final))
#Chequear si el directorio existe
if not os.path.exists(path):
print(colored("\nDirectorio no encontrado, creándolo",'cyan'))
os.makedirs(path)
#Mover el archivo
shutil.move(serie,path)
print(colored('\nCopiando:','green'), serie, colored('a', 'green'), path + '/' + serie)
else:
print('No hay archivos para organizar.\n')
input("\n\nPresione Enter para continuar ...")
Your if serie == serie[-1] check doesn't work because instead of checking "if the series is the last one in the list", you check "if the series is also the last character of the same series".
Consider using something like this instead:
series = [s for s in glob.glob(buscar) if s.endswith(('.mp4', '.srt', '.avi', '.mkv'))]
if series:
print(colored("Archivos encontrados: ",'red'))
for serie in series:
print(serie)
...
else:
print('No hay archivos para organizar.\n')
I'm currently working on a little side project yet having multiple issues. I'm reading a file within the folder where the project is that holds data for 10 users.
As for the code itself...it's pretty big.
def ouvrir_fichier(nomFichier):
""" Ne pas oublier les docstring
"""
try:
fp = open(nomFichier, 'r')
return fp
except:
return print("Le fichier n'existe pas. Veuillez réessayer.")
def lire_fichier(fp):
""" Ne pas oublier les docstring"""
# Lis n et initialise une liste vide --
# où il y a une liste vide pour chaque usager du réseau
# ensuite lis le reste du fichier et ajouter l'information à reseau
liste1 = fp.readlines()
n = int(liste1[0])
liste2 = liste1[1:]
reseau = [[] for i in range(n)]
for i in liste2:
i = i.replace("\n", "")
data = i.split(" ")
valeur1 = int(data[0])
valeur2 = int(data[1])
reseau[valeur1].append(valeur2)
reseau[valeur2].append(valeur1)
fp.close()
return reseau
def trouver_nombre_elements_communs_entre_listes(liste1, liste2):
""" Ne pas oublier les docstring"""
compteur_amis_commun = 0
for element in liste1:
if element in liste2:
compteur_amis_commun = compteur_amis_commun + 1
return compteur_amis_commun
def initialiser_matrice(n):
"""
Crée une matrice nxn, initialisée avec des zéros et retourne la matrice.
Args:
n (int): dimension de la matrice nxn
Returns:
matrice (list): matrice initialisée
"""
matrice = []
for ligne in range(n): # pour chacune des lignes dans n
matrice.append([]) # créer une ligne (liste) et l'initialiser à 0
for colonne in range(n):
matrice[ligne].append(0) # ajouter un 0 pour chaque n colonne
return matrice
def calculer_scores_similarite(reseau):
""" Ne pas oublier les docstring"""
n = len(reseau)
matrice_similarite = initialiser_matrice(n)
liste1 = []
liste2 = []
compteur_liste1 = 0
compteur_liste2 = 0
for element_liste1 in reseau:
liste1 = element_liste1
for element_liste2 in reseau:
liste2 = element_liste2
compteur_amis_commun = trouver_nombre_elements_communs_entre_listes(liste1, liste2)
matrice_similarite[compteur_liste1][compteur_liste2] = compteur_amis_commun
compteur_liste2 = compteur_liste2 + 1
compteur_liste1 = compteur_liste1 + 1
compteur_liste2 = 0
return matrice_similarite
def recommander(id_usager,reseau,matrice_similarite):
""" Ne pas oublier les docstring"""
usager_matrice = matrice_similarite.index(id_usager)
ami_recommande = matrice_similarite.index(max(usager_matrice))
max_value = max(matrice_similarite.index(usager_matrice))
if ami_recommande == id_usager:
max_value = max_value - 1
ami_recommande = matrice_similarite.index(max_value)
while True:
if ami_recommande == reseau.index(ami_recommande):
ami_recommande = reseau.index(max_value, ami_recommande + 1)
return True
return ami_recommande
def main():
nomFichier = input("Nom du fichier contenant le réseau: ")
reseau = lire_fichier(ouvrir_fichier(nomFichier))
n = len(reseau)
matrice_similarite = calculer_scores_similarite(reseau)
while True:
while True:
id_usager = int(input("Entrer l'ID de l'usager pour lequel vous voulez une recommandation (entre 0 et {}):".format(n)))
if 0 <= id_usager and id_usager < n:
calculer_scores_similarite(reseau)
print("Pour la personne" , id_usager , ", nous recommandons l'ami" , recommander(id_usager, reseau, matrice_similarite))
continue
else:
print("Erreur: l'usager doit être un nombre entier entre ", 0, "et", n - 1, "inclusivement.\n")
autreRecommandation = input("Voulez-vous une autre recommandation (oui/non)?")
if autreRecommandation.lower() == "oui":
return True
else:
print("Merci d'avoir utiliser le programme de recommandation d'amis.")
break
if __name__ == "__main__":
main()
Most of the content seems to be working fine until I get to part where I need to recommend a user identification. I'll try to work on the doc string as well in the meantime but I could totally use a little bit of help as to debug this. I tested most of the code on another .py project until I hit the function recommander
First Edit:
I did forget to apply the return. I changed it and it is now in the def. Now however...I seem to be having this error.
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3.2\helpers\pydev\pydevd.py", line 1668, in <module>
main()
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3.2\helpers\pydev\pydevd.py", line 1662, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3.2\helpers\pydev\pydevd.py", line 1072, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3.2\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/101136/PycharmProjects/tp2/TP2.py", line 132, in <module>
main()
File "C:/Users/101136/PycharmProjects/tp2/TP2.py", line 119, in main
print("Pour la personne" , id_usager , ", nous recommandons l'ami" , recommander(id_usager, reseau, matrice_similarite))
File "C:/Users/101136/PycharmProjects/tp2/TP2.py", line 89, in recommander
usager_matrice = matrice_similarite.index(id_usager)
ValueError: 0 is not in list
Also I'm wondering about a little something. The part where I ask for an input to define id_usager in the def main(), I was wondering if there was a way to also treat characters as well as integers. Integers, I got it covered with my code but if somebody wants to write boy instead of a number, it'll fail.
accessing element of list is done by list_name[index], list_name.index(element) returns the index of that element,if they are two in the list it returns the index of the first one, if the element is not there,it throws a ValueError: "element" is not in the list.
in your case
change
usager_matrice = matrice_similarite.index(id_usager)
to
usager_matrice =matrice_similarite[id_usager]
and other places you want to access element of a list using index
in the scond question from the comment, check if its numeric like this
id_usager =input("Entrer l'ID de l'usager pour lequel vous voulez une recommandation (entre 0 et {}):".format(n))
if not id_usager.isdigit():
continue
id_user=int(id_usager)
note that this will convert also floats into int
I am implement a simple simulator of soccer with python using threads and lock, the app works fine but I have doubts in the way that implement the thread it seems to me that the first team has an advantage because is executing first.
def jugar(Equipo1, Equipo2):
# Busco las probabilidades de encajar por cada equipo
prob_encajar_eq1 = Equipo1.probabilidad_encajar()
prob_encajar_eq2 = Equipo2.probabilidad_encajar()
def jugar_equipo1(defensa_rival):
semaforo.acquire()
if Equipo1.hacer_pases():
Equipo1.shoot(defensa_rival)
semaforo.release()
def jugar_equipo2(defensa_rival):
semaforo.acquire()
if Equipo2.hacer_pases():
Equipo2.shoot(defensa_rival)
semaforo.release()
hilo_equipo1 = threading.Thread(name = 'hilo_eq1', target = jugar_equipo1, args = (prob_encajar_eq2,))
hilo_equipo2 = threading.Thread(name = 'hilo_eq2', target = jugar_equipo2, args = (prob_encajar_eq1,))
hilo_equipo1.start()
hilo_equipo2.start()
hilo_equipo1.join()
hilo_equipo2.join()
to make several attempts both teams, I do a cicle for a few seconds and inside the function jugar() which is the one that does the work with threads but here is were I have the doubts, because every time that jugar is executing the threads are declared again.
if __name__ == '__main__':
cargar_informacion()
eqA = Equipo(equipoA, ranking_eqA)
eqB = Equipo(equipoB, ranking_eqB)
probabilidades = porcenajes_ranking(ranking_eqA)
eqA.cargar_probabilidades(probabilidades)
probabilidades = porcenajes_ranking(ranking_eqB)
eqB.cargar_probabilidades(probabilidades)
starttime=time.time()
tiempo = 0
# creo la barra de progreso
bar = progressbar.ProgressBar(widgets=[
progressbar.Percentage(),
progressbar.Bar(),
], max_value=100).start()
# hacemos que el juego tarde aproximadamente 10seg en simularse.
while tiempo < 10:
time.sleep(0.3 - ((time.time() - starttime) % 0.3))
jugar(eqA,eqB)
tiempo = time.time() - starttime
bar += 2.8
bar.finish() # Para que finalice la barra de progreso
resultados_finales(eqA, eqB) # Mostramos el resultado final del partido.
I've got a problem with Python 2.7.3-32bits on Windows. I put this code to see if anyone can help me out with this error. The comments are in Spanish but it don't affect the code.
import gtk
import numpy
import math
import os
#Pedimos el nombre de la imagen de origen
nombreFich = input("Por favor, introduzca el nombre de la imagen de origen:")
#Validar que existe el fichero
imagen1 = gtk.Image()
imagen1.set_from_file('C:\\Users\\xxx\\Desktop\\xxxx.png')
pb1 = imagen1.get_pixbuf()
pm1 = pb1.get_pixels_array()
#Hacemos una copia de la imagen
pm2 = pm1.copy()
#Validamos los puntos de distorsion hasta que sean validos
puntos = " "
arrayPuntos = " "
while(puntos == " " and len(arrayPuntos) < 4):
print"Por favor, introduzca los puntos de distorsión xc yc r e:"
puntos= raw_input()
arrayPuntos = puntos.split(" ")
#Sacamos los puntos separando la cadena por el caracter espacio
xc =(puntos[0])
yc =(puntos[1])
r =(puntos[2])
e =(puntos[3])
#función que calcula el grado de distorsión
def grado(self,z,e):
if(z>1):
return 1
elif(e<0):
return (1/z)**(-e/(1-e))
else:
return z**e
#Distorsionamos la imagen
def distors(xc,yc,r,e,x,y):
d = math.sqrt(x**2+y**2)#Sacamos la distancia
z = d/r
if(z!=0):
g=grado(z,e)
xm=x*g
ym=y*g
return xm,ym
else:
xm=x
ym=y
return xm,ym
def recorrido (pm1, xc, yc, r, e):
pm2 = pm1.copy()
x= str(--r)
y= str(--r)
while (y <= r):
while (x <= r):
xm, ym = mover(xc, yc, r, e, x, y)
pm2[yc+y][xc+x] = pm1[yc+ym][xc+xm]
x = x+1
y= y+1
x= -r
return pm2
pm2 = recorrido(pm1, xc, yc, r, e)
#Guardamos los cambios
pb2 = gtk.gdk.pixbuf_new_from_array(pm2,pb1.get_colorspace(),pb1.get_bits_per_sample())
nomfich2 = nombreFich+"_copia"
ext = os.path.splitext("C:\\Users\xxx\Desktop\xxxx.png_copia")[1][1:].lower()
pb2.save("C:\\Users\xxx\Desktop\xxxx.png_copia",ext)
print"FINISH"
When I run the python code I get the following error:
Traceback (most recent call last):
File "F:\Dropbox\Práctica Pitón\Práctica3.0.py", line 71, in <module>
pm2 = recorrido(pm1, xc, yc, r, e)
File "F:\Dropbox\Práctica Pitón\Práctica3.0.py", line 59, in recorrido
x= str(--r)
TypeError: bad operand type for unary -: 'str'
The error message is telling you that r is a string. You can't negate a string.
Why is it a string? Well, it seems to come from here:
# ...
puntos= raw_input()
arrayPuntos = puntos.split(" ")
# ...
r =(puntos[2])
The split method on a string returns a list of strings.
So, how do you solve this? Well, if r is, say, the string "22", then float(r) is the float 22.0, and int(r) is the integer 22. One of those is probably what you want.
Once you add, say, r=int(r), your --r will no longer be an exception.
But it probably isn't what you want. In Python, --r just means the negation of the negation of r—in other words, it's the same as -(-(r)), which is just r. You're probably looking for the equivalent of the C prefix operator--, which decrements the variable and returns the new value. There is no such operator in Python; in fact, there are no operators that modify a variable and then return the value.
This is part of a larger issue. Python is designed to make statements and expressions as distinct as possible, to avoid confusion. C is designed to make as many things as possible expressions, to save typing. So, you often can't just translate one into the other line by line.
In this case, you have to do it in two steps, as Thanasis Petsas shows:
r -= 1
x = str(r)
Increment ++ and decrement -- operators are not supported in python.
You can use instead this:
r -= 1
x = str(r)