I have a process that gets a files from a directory and puts them in a list. It then iterates that list in a loop. The last line of the loop being where it should update my gui display, then it begins the loop again with the next item in the list.
My problem is that it does not actually update the gui until the entire process is complete, which depending on the size of the list could be 30 seconds to over a minute. This gives the feeling of the program being 'hung'
What I wanted it to do was to process one line in the list, update the gui and then continue. Where did I go wrong? The line to update the list is # Populate listview with drive contents. The print statements are just for debug.
def populateList(self):
print "populateList"
sSource = self.txSource.Value
sDest = self.txDest.Value
# re-intialize listview and validated list
self.listView1.DeleteAllItems()
self.validatedMove = None
self.validatedMove = []
#Create list of files
listOfFiles = getList(sSource)
#prompt if no files detected
if listOfFiles == []:
self.lvActions.Append([datetime.datetime.now(),"Parse Source for .MP3 files","No .MP3 files in source directory"])
#Populate list after both Source and Dest are chosen
if len(sDest) > 1 and len(sDest) > 1:
print "-iterate listOfFiles"
for file in listOfFiles:
sFilename = os.path.basename(file)
sTitle = getTitle(file)
sArtist = getArtist(file)
sAlbum = getAblum(file)
# Make path = sDest + Artist + Album
sDestDir = os.path.join (sDest, sArtist)
sDestDir = os.path.join (sDestDir, sAlbum)
#If file exists change destination to *.copyX.mp3
sDestDir = self.defineDestFilename(os.path.join(sDestDir,sFilename))
# Populate listview with drive contents
self.listView1.Append([sFilename,sTitle,sArtist,sAlbum,sDestDir])
#populate list to later use in move command
self.validatedMove.append([file,sDestDir])
print "-item added to SourceDest list"
else:
print "-list not iterated"
Create a worker thread/process that does your processing in the background and updates the GUI after the processing is done, maybe reporting progress during work.
Have a look at the threading or multiprocessing modules.
This is a common problem with GUI programs. Controls don't get updated until a "repaint" command is received and processed, and that won't happen until your function returns.
You can force a control to repaint at any time by calling its Update method, as shown in the answer to this question: How do you force refresh of a wx.Panel?
I might suggest you try wx.lib.delayedresult. It is somehow the simplified multithread workaround. You can put your business logic into worker function and other logics (including GUI appending, updating) in consumer function. The worker function runs in another thread while the consumer function is guaranteed to run after the finish of worker function in main thread.
Related
I use the right click menu to launch a program that moves files to a folder and do some work on them. The problem is that when i do this with multiple files, it starts multiple instances of the program. If i have 50 files, it will launch the app 50 times.
I need to have only one window not multiple windows. So far i manage to make it work sometimes with this code, but what i need is to make it work a 100% of time:
# ========================================================
#this only catches the file adress that was right clicked and launch with the app. Then it moves it to the installer folder
try:
# THIS CATCHES THE SELECTED FILE LINK TO BE MOVED TO THE INSTALL FOLDER IF IT IS SUCCESSFULL
variable= sys.argv[1]
index = -1
for i in variable:
index = index + 1
if "\\" in i:
sum= index
# MOVE TO INSTALLER DIRECTORY
shutil.move(variable, Install_folder + f"\Installer\files\{str(variable[sum+ 1::])}")
except:
print('FILE NOT ADDED THROUGH THE MOVE TOOL')
#=========================================================
processlist = list()
time.sleep(int(time.process_time())*random.randint(1,3))
for process in psutil.process_iter():
processlist.append(process.name())
if processlist.count("program.exe") >= 4:
sys.exit()
My guess is that the programs start activating at the same time and that could be why they are closing instead of letting only one active window remain. I only have 2 months of python so hope you could help me. Thank you in advance for reading.
My other alternate solution to this is separate the programs into two. One for moving the files and another for doing the work with the files. But this solution it is not the desired one.
Solution:
try:
variable = sys.argv[1]
index = -1
# THIS CATCHES THE SELECTED FILE LINK TO BE MOVED TO THE HANDY INSTALL FOLDER IF IT IS SUCCESSFULL
for i in chetita:
index = index + 1
if "\\" in i:
sum = index
# CHOOSES DIRECTORY WHERE THE FILES TO BE INSTALLED ARE
shutil.move(variable, f"{Install_folder}\\program\\InstallFolder\{str(variable[sum + 1::])}")
os.makedirs(f"{Install_folder}\\program\\Queue")
tiger = []
serpent = []
while True:
time.sleep(1)
serpent = os.listdir(f"{Install_folder}\\program\\InstallFolder")
time.sleep(1)
tiger = os.listdir(f"{Install_folder}\\program\\InstallFolder")
if tiger == serpent:
break
except:
if os.path.exists(f"{Install_folder}\\program\\Queue"):
sys.exit()
print('application executed directly through the .exe').
Basically when all the windows open, the first window on compliting the task of moving a file. Creates a file called Queue and enters a while loop that is active until the content of the folder match 2 variables(meaning all the other windows finish their work and are closed). The other programs closes because they get an execption when they tried to create that folder. When all the other programs are closed, the waiting windows will leave the while loop and start working on the files.
Author your python app so it behaves in this way.
from pathlib import Path
QUEUE_FILE = Path("~/queue.txt").expanduser()
...
if __name__ == "__main__":
if QUEUE_FILE.exists():
... # do the old app behavior in an interactive window
else:
append_filename(QUEUE_FILE, sys.argv)
So there are two modes of operation.
That second if clause will very quickly service any right-click requests.
It does almost no work, merely writing a line of text to a central queue file.
The first if clause mostly behaves the same as your current app,
and it keeps a single window open while you're interactively working with it.
The difference is that, instead of accepting filename(s) in sys.argv,
it accepts most of those fifty filenames via the central queue file.
Upon exiting, it must delete the queue file.
That sets us up for a subsequent interaction.
I have a function that watches a csv file. It iterates through all the lines in a csv file and returns each line, waits for the file to be updated, and if it is, returns that new value. This is how it looks like:
def follow():
file_path = filedialog.askopenfilename()
with open(file_path) as csvDataFile:
csvReader = csv.reader(csvDataFile)
csvDataFile.seek(0,0)
while True:
line = csvDataFile.readline()
if not line:
time.sleep(0.5)
continue
yield line
I am doing this for a gui. I have a button that, when pressed, loads the csv file and calls this function. It saves each line to a list that is a member variable.
def browseForFile(self):
line = pull_csv_data.follow()
for item in line:
self.list.append(item)
When another button is pressed, it iterates through that list and displays the information on the gui:
def listItems(self):
for i in self.list:
time.sleep(0.1)
item = QListWidgetItem(i)
self.prev_scenes.addItem(item)
The problem is, whenever I click the button to display the data in the list, it stops responding until I kill the python script that has the follow() function. For clarification, the follow function is in a separate file that I include in my "main" file.
Essentially.. the follow function does not allow for other processes to run at the same time, I think. Is that true? is there a workaround? Or is there a better way to do this?
The follow function is in the same process as your main function. Because of this, when your follow function sleeps, so does your entire program. If you want this to be run asynchronously, you'll have to make it do so explicitly.
You should probably use something like https://pythonhosted.org/watchdog/index.html or https://github.com/seb-m/pyinotify/wiki instead of trying to write your own file handling function.
I need to conform some maya scenes we receive from a client to make them compatible to our pipeline. I'd like to batch that action, obviously, and I'm asked to launch the process from within Maya.
I've tried two methods already (quite similar to each other), which both work, but the problem is that the Maya GUI freezes until the process is complete. I'd like for the process to be completely transparent for the user so that they can keep workind, and only a message when it's done.
Here's what I tried and found until now:
This tutorial here : http://www.toadstorm.com/blog/?p=136 led me to write this and save it:
filename = sys.argv[1]
def createSphere(filename):
std.initialize(name='python')
try:
mc.file(filename, open=True, pmt=False, force=True)
sphere = mc.polySphere() [0]
mc.file(save=True, force=True)
sys.stdout.write(sphere)
except Exception, e:
sys.stderr.write(str(e))
sys.exit(-1)
if float(mc.about(v=True)) >= 2016.0:
std.uninitialize()
createSphere(filename)
Then to call it from within maya that way:
mayapyPath = 'C:/Program Files/Autodesk/Maya2016/bin/mayapy.exe'
scriptPath = 'P:/WG_MAYA_Users/lbouet/scripts/createSphere.py'
filenames = ['file1', 'file2', 'file3', 'file4']
def massCreateSphere(filenames):
for filename in filenames:
maya = subprocess.Popen(mayapyPath+' '+scriptPath+' '+filename,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
out,err = maya.communicate()
exitcode = maya.returncode
if str(exitcode) != '0':
print(err)
print 'error opening file: %s' % (filename)
else:
print 'added sphere %s to %s' % (out,filename)
massCreateSphere(filenames)
It works fine, but like I said, freezes Maya GUI until the process is over. And it's just for creating a sphere, so not nearly close to all the actions I'll actually have to perform on the scenes.
I've also tried to run the first script via a .bat file calling mayabatch and running the script, same issue.
I found this post (Running list of cmd.exe commands from maya in Python) who seems to be exactly what I'm looking for, but I can't see how to adapt it to my situation ?
From what I understand the issue might come from calling Popen in a loop (i.e. multiple times), but I really can't see how to do otherwise... I'm thinking maybe saving the second script somewhere on disk too and calling that one from Maya ?
In this case subprocess.communicate() will block until the child process is done, so it is not going to fix your problem on its own.
If you just want to kick off the processes and not wait for them to complete -- 'fire and forget' style -- you can just use threads, starting off a new thread for each process. However you'll have to be very careful about reporting back to the user -- if you try to touch the Maya scene or GUI from an outside thread you'll get mysterious, undebuggable errors. print() is usually ok but maya.cmds() is not. If you're only printing messages you can probably get away with maya.utils.executeDeferred() which is discussed in this question and in the docs.
I'd like to ask if there's any way we could watch a directory in python and parse the latest text file being generated on the directory.
Tho i have this start up code which parse a certain text file.
import time
def follow(thefile):
thefile.seek(0,2)
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
continue
yield line
if __name__ == '__main__':
logfile = open(r'\\some directory\files.txt',"r")
loglines = follow(logfile)
for line in loglines:
print line,
See the bold files.txt i need that to be dynamic by watching the directory for newly generated text files and switch to the latest text file and parse it.
It will run on Windows XP service Pack 3
I'm using Python 2.7
Directory i'm watching is also using windows XP
Thank you.
To check for new files, repeatedly get a list of files currently in the directory with os.listdir('directory'). Save the entries in a set and calculate the difference of the set with the previous set.
# Initialize before an event loop:
old_entries = set()
# You need a loop that calls two handlers, each handler returning soon.
# Inside your loop, check for a "new file" event this way:
now_entries = os.listdir(r'\\some directory')
now_entries.symmetric_difference_update(old_entries)
for new_entry in now_entries:
handle_new_file(new_entry)
Your program needs to listen for two events:
New file in the directory.
New line in the old file.
You call follow(), which is like an event handler that never returns. I think you want that handler to return to one main event loop that checks for each kind of event. Your follow() function never returns because it continues within the while True infinite loop unless a new line is added to the file for it to yield. It will never yield if no more lines are getting added to that file.
Take a look into the FindFirstChangeNotification API
http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html
The approach here is to use the MS FindFirstChangeNotification API, exposed via the pywin32 win32file module. It needs a little explanation: you get a change handle for a directory (optionally with its subdirectories) for certain kinds of change. You then use the ubiquitous WaitForSingleObject call from win32event, which fires when something's changed in one of your directories.
Essentially because the Windows OS is responsible for managing creation/modification of files, you can ask it to let you know immediately when a file is changed/created.
I am trying to make a program(in python) that as I write it writes to a file and opens to a certain window that I have already created.I have looked allarund for a vaible soution bt it would seem that multi-threading may be the only option.
I was hoping that when option autorun is "activated" it will:
while 1:
wbuffer = textview.get_buffer()
text = wbuffer.get_text(wbuffer.get_start_iter(), wbuffer.get_end_iter())
openfile = open(filename,"w")
openfile.write(text)
openfile.close()
I am using pygtk and have a textview window, but when I get the buffer it sits forever.
I am thinking that I need to multi-thread it and queue it so one thread will be writing the buffer while it is being queued.
my source is here. (I think the statement is at line 177.)
any help is much appreciated. :)
and here is the function:
def autorun(save):
filename = None
chooser = gtk.FileChooserDialog("Save File...", None,
gtk.FILE_CHOOSER_ACTION_SAVE,
(gtk.STOCK_CANCEL, gtk.RESPONSE_CANCEL,
gtk.STOCK_SAVE, gtk.RESPONSE_OK))
response = chooser.run()
if response == gtk.RESPONSE_OK: filename = chooser.get_filename()
filen = filename
addr = (filename)
addressbar.set_text("file://" + filename)
web.open(addr)
chooser.destroy()
wbuffer = textview.get_buffer()
while 1:
text = wbuffer.get_text(wbuffer.get_start_iter(), wbuffer.get_end_iter())
time.sleep(1)
openfile = open(filename,"w")
openfile.write(text)
openfile.close()
Though not too easy to see exactly what your GTK-stuff not included here is doing, the main problem is that the control needs to be returned to the gtk main-loop. Else the program will hang.
So if you have a long process (like this eternal one here), then you need to thread it. The problem is that you need the thread to exit nicely when the main program quits, so you'll have to redesign a bit around that. Also, threading with gtk needs to be initialized correctly (look here).
However, I don't think you need threading, instead you could connect the changed signal of your TextBuffer to a function that writes the buffer to the target-file (if the user has put the program in autorun-mode). A problem with this is if the buffer gets large or program slow, in which case, you should consider threading the callback of the changed signal. So this solution requires to make sure you don't get into the situation where save-requests get stacked on top of each other because the user is faster at typing than the computer is saving. Takes some design thought.
So, finally, the easier solution: you may not want the buffer to save for every button-press. In which case, you could have the save-function (which could look like your first code-block without the loop) on a timeout instead. Just don't make the time-out too short.