I wrote (copied and adapted) a python program to do some backups. It basically reads the files from a source location and copies or creates hard links in a destination. It is multi-threaded and that part works ok. The problem is that the loop that is called in the threads never ends. The relevant code is below. I read that it might be one of the threads raising an error and hanging but don't know how to check that. The program reads from a file queue, process the files, then is supposed to end. It reads and processes the files till the queue is empty and then just hangs. Any suggestions on what to try or to look at? hopefully i pasted enough code to help
print ("beginning backup")
backup(path_to_source, path_to_full, dest_path, backup_type)
print ("Backup finished")
# --- clean full backup
if backup_type == "Full":
print ("Cleaning up full backup - deleting deleted files")
clean_backup()
print ("Backup has been cleaned. Done")
#--------------------------------------------------
def backup(path_to_source, path_to_full, dest_path, backup_type):
threadWorkerCopyInc(source_files)
print ("All Done") #(never executes)
def threadWorkerCopyInc(fileNameList):
global num_threads
#fileQueue = queue.queue()
for i in range(num_threads):
t = threading.Thread(target=IncCopyWorker)
t.daemon = True
t.start()
#threads.append(t)
for fileName in fileNameList:
fileQueue.put(fileName)
fileQueue.join()
#for i in range(len(threads)):
# fileQueue.put('None')
#for t in threads:
# t.join()
print('done with threadworkercopyinc') #(never executes)
def IncCopyWorker():
print ('Starting IncCopyWorker. ') #executes
while True:
filename = fileQueue.get()
if filename == 'None':
print("\nFilename is none") #executes when fileQueue.qsize() = 0
with threadlock: processedfiles +=1
print ("Files to go: %d processed files: %d newfiles: %d hardlinks: %d not copied: %d dfsr: %d " %(fileQueue.qsize(), processedfiles, newfiles, hardlinks, notcopied, dfsr), end = '\r')
is_dfsr = filename.upper().find("DFSR")
if (is_dfsr == -1):
#do main processing here
else:
with threadlock: dfsr+=1
fileQueue.task_done()
print ("done with while true loop") #never Executes
So it looks like my problem was that i included continue statements in my try/except blocks to try and speed up execution. When there is a continue or break statement the loop never ended. When i took out the continue statements it finished the way it was supposed to. I don't know the reason -just the solution
Related
I want to write a python script that will read all output of other scripts and save it to a log file. This is my simple test that doesn't work.
#parent.py
def writer():
i = 10
while i:
print(i)
i -= 1
print('Start of parent process \n')
process = subprocess.Popen(['python3', 'reader.py'], stdin=sys.stdout)
print('Start of writer')
writer()
#reader.py
def reader():
count = 10
log = open('log', 'w')
while count:
print(f'count = {count}')
i = input()
print(i)
log.write(i)
count -= 1
log.close()
print('Second process started')
reader()
print('Second process finish')
I need to set the input of the child process, but not send over some channels. I already have few scripts, they use simple echo for printing, and my reader should log that.
First of all I'm quite new to programming so sorry for the badly formatted/dodgy code
I am trying to make a program that tracks the amount of time certain programs is running. It sees if a program is running and then adds 1 to a variable which is then written to a file. The problem is I cannot run multiple loops at once to check if those programs are running. This leads to my code only checking if the first one is running.
My question is, is it possible to make several infinite loops at once to see if a program is running all at once. If so how do I do this. If there is another alternate solution that can accomplish the same thing please let me know down below.
I didnt know how to check if a program is running so i took the code used from here
The code I have right now is:
import time
import psutil
def programrunning(processName):
'''
Check if there is any running process that contains the given name processName.
'''
# Iterate over the all the running process
for proc in psutil.process_iter():
try:
# Check if process name contains the given name string.
if processName.lower() in proc.name().lower():
return True
except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):
pass
return False;
def write(file, var):
store_thing = open(file, "w+")
store_thing.write(var)
store_thing.close()
def create(filename):
open(filename, "x")
def read(filename):
read_p = open(filename, "r")
read_p.mode == "r"
content = read_p.read()
return content
def whilerun(program):
while programrunning(program) == True:
var = read(program + ".txt")
programvar = var
programvar = int(programvar) + int(1)
time.sleep(1)
write(program + ".txt", str(programvar))
while programrunning(program) == False:
var = read(program + ".txt")
programvar = var
programvar = int(programvar) + int(1)
time.sleep(1)
write(program + ".txt", str(programvar))
whilerun("program1")
whilerun("program2")
whilerun("program3")
I'd put a loop for your programs.
programs = ["program1", "program2", "program3"]
def whilerun(programs):
while True:
any_alive = False # flag for inf loop prevention
for program in programs:
if programrunning(program):
# -- do sth for program running --
any_alive = True
else:
# -- do sth for program NOT running
pass
if not any_alive: # stop the loop if all processes are dead
break
I have made a thread that is supposed to show the seconds passing. Unfortunately, when I use getstr from the curses module the whole script stops, including the thread. I have to use a thread lock to stop random characters being printed out because of overlapping orders.
Any suggestions on how to fix this or an alternative would be great!
In the below example window and window2 are already set-up...
lock = threaing.Lock()
def main_function():
#starts thread
t1 = threading.Thread(target=time_calc,args=(window2,))
t1.start()
#Gets user input
while True:
data = window1.addstr(y,x,"Type something in!")
data = window1.getstr(y,x,5)
lock.acquire()
window1.erase()
txt = "You said: "+data
window1.addstr(y,x,txt)
lock.release()
def time_calc(window2):
current_count = 0
while True:
time += 1
text = "Time Count: "+str(time)
lock.acquire()
window2.erase()
window2.addstr(y,x,text)
lock.release()
time.sleep(1)
The problem with my code
I figured out the problem with my code. You can't run a thread inside a thread for some reason and I originally had my main function called to be considered a thread. I guess I should have stated this in my question. Sorry
There is probably a way to run a thread in a thread, but this did not work for me.
My Updated Code
lock = threading.Lock()
def call_threads():
t1 = threading.Thread(target=main_function,args=(window1,))
t1.start()
t2 = threading.Thread(target=time_calc,args=(window2,))
t1.start()
def main_function(window1):
#Gets user input
while True:
data = window1.addstr(y,x,"Type something in!")
data = window1.getstr(y,x,5)
lock.acquire()
window1.erase()
txt = "You said: "+data
window1.addstr(y,x,txt)
lock.release()
def time_calc(window2):
current_count = 0
while True:
time += 1
text = "Time Count: "+str(time)
lock.acquire()
window2.erase()
window2.addstr(y,x,text)
lock.release()
time.sleep(1)
If there is anything else that may have caused this error, please comment it!
I've been struggling with this program for some time.
I wrote this program to learn how to run the same program in different CPU's at the same time and to reduce processing time.
By itself, the program is not complex, just detect if there are some screenshots in different directories, but as I said before my final goal is not to detect those files, but to learn multiprocessing.
To be easier to test, instead of reading files or reading user inputs, I just set those variables so you won't have to be typing those things. I'll be faster to test this way.
Here's the code:
from multiprocessing import Array, Lock, Pool
def DetectSCREENSHOT(file, counter, total):
# To check if this function is ever accessed
print 'Entered in DetectSCREENSHOT function'
if file.startswith('Screenshot') == False:
print ' (%s %%) ERROR: the image is not a screenshot ' %(round((float(counter)/total)*100, 1))
return 0
else:
print ' (%s %%) SUCCESS: the image is a screenshot' %(round((float(counter)/total)*100, 1))
return 1
def DetectPNG(file_list, counter_success_errors, lock):
# To check if this function is ever accessed
print 'Entered in DetectPNG function'
for i in range(len(file_list)):
file = file_list[i]
# If the file is an .png image:
if file.endswith('.png'):
lock.acquire()
counter_success_errors[0] = counter_success_errors[0] + 1
lock.release()
if DetectSCREENSHOT(file, counter_success_errors[0], len(file_list)) == 1:
lock.acquire()
counter_success_errors[1] = counter_success_errors[1] + 1
lock.release()
else:
lock.acquire()
counter_success_errors[2] = counter_success_errors[2] + 1
lock.release()
def Main():
# file_list = # List of lists of files in different directories
file_list = [['A.png', 'B.png', 'C.txt', 'Screenshot_1.png'], ['D.txt', 'Screenshot_2.png', 'F.png', 'Screenshot_3.png']]
# Array where the first cell is a counter, the second one, the number of successes and the third one, the number of errors.
counter_success_errors = Array('i', 3)
lock = Lock()
counter_success_errors[0] = 0
counter_success_errors[1] = 0
counter_success_errors[2] = 0
# Number of CPUS's to be used (will set the number of different processes to be run)
# CPUs = raw_input('Number of CPUs >> ')
CPUs = 2
pool = Pool(processes=CPUs)
for i in range(CPUs):
pool.apply_async(DetectPNG, [file_list[i], counter_success_errors, lock])
pool.close()
pool.join()
success = int(counter_success_errors[1])
errors = int(counter_success_errors[2])
print(' %s %s %s successfully. %s %s.' %('Detected', success, 'screenshot' if success == 1 else 'screenshots', errors, 'error' if errors == 1 else 'errors'))
#################################################
if __name__ == "__main__":
Main()
When I execute it, I don't get any errors, but it looks like it doesn't even access to both DetectPNG and DetectSCREENSHOT functions.
What's broken in the code?
Output: Detected 0 screenshots successfully. 0 errors.
I'm new to Python and have been googling for a couple of days and read all I can find on this forum. Might be that I don't understand it all but I haven't found a solution to my problem yet. Ask for forgiveness already if there's an answer already to my problem, then I haven't understood it.
I want to make a Pause function for my program Tennismatch. The program will when it's being run print the score of a tennis match like this: "15-0, 15-15 etc ongoing till the match ends. It will print the score line by line.
I want the user to be able to pause after x number of balls, games, etc. So I don't know when the user wants to pause and after the user has paused I want the user to be able to resume the tennismatch where it was.
Have seen the time.sleep() but as I have understood it you must know when you want to pause to use this and it also ain't an indefinetie pause like I want. With input() it's the same.
Am going to make a GUI later on when the code is finished. Happy for anything that leads me to solving my problem.
I use Windows and Python 3.42 and run the program in Shell.
A piece of the code (haven't written it all yet, it's more of a general situation when something is being printed line after line for some time and want to be able do pause in the CIL:
#self.__points = [0,0]
def playGame(self):
if self.server == True: #self.server is either True or False when someone calls playGame()
server = self.player_1.get_win_serve() #self.player_1 = an object of a class Player():
else:
server = self.player_2.get_win_serve() #get_win_serve() method returns the probability to win his serv (1-0)
while (0 < self.__points[0] - self.__points[1] >= 2 or 0 < self.__points[1] - self.__points[0] >= 2) and (self.__points[1] >= 4 or self.__points[0] >= 4):
x = random.uniform(0,1)
if x > 0 and x < server:
self.__points[0] += 1
else:
self.__points[1] += 1
# print('The score, by calling a score() function that I haven't written yet')
For dealing with events in main loop you need to make a separated thread which capture input or any other event.
import sys
from sys import stdin
from time import sleep
from threading import Thread
from Queue import Queue, Empty
def do_something():
sleep(1)
print 42
def enqueue_output(queue):
while True:
# reading line from stdin and pushing to shared queue
input = stdin.readline()
print "got input ", input
queue.put(input)
queue = Queue()
t = Thread(target=enqueue_output, args=(queue,))
t.daemon = True
t.start()
pause = False
try:
while True:
try:
command = queue.get_nowait().strip()
print 'got from queue ', command
except Empty:
print "queue is empty"
command = None
if command:
if command == 'p':
pause = True
if command == 'u':
pause = False
if not pause:
print pause
do_something()
except KeyboardInterrupt:
sys.exit(0)
I came up with the following.
while True:
try:
## Keep doing something here
## your regular code
print '.',
except KeyboardInterrupt:
## write or call pause function which could be time.sleep()
print '\nPausing... (Hit ENTER to continue, type quit to exit.)'
try:
response = raw_input()
if response.lower() == 'quit':
break
print 'Quitting...'
except KeyboardInterrupt:
print 'Resuming...'
continue
The Event loop might as well be the code I wrote with.
I don't see any user input so I assume that x emulates it. To pause the game if x < 0.1 and to unpause(/resume) it if x > 0.9, you could:
while your_condition(self.__points):
x = random.random()
if x < 0.1: # pause
self.pause()
elif x > 0.9: # resume
self.resume()
if self.is_paused:
continue # do nothing else only wait for input (`x`)
# assume your_condition() has no side-effects
# here's what the resumed version does:
print("...")
# change self.__points, etc
where pause(), resume(), is_paused() methods could be implemented as:
def __init__(self):
self.is_paused = False
def pause(self):
self.is_paused = True
def resume(self):
self.is_paused = False
as you can see the implementation is very simple.