Edit: The main part of this question before this revision/update was how to terminate a QThread. That has been solved, the question is being revised to How to kill a requests rest object that is in progress.
http://docs.python-requests.org/en/v0.10.4/user/advanced/#asynchronous-requests
It appears using an Asynchronous Request still blocks - the user can't cancel the post while its in progress.
Basically this is the functionality that is needed:
When the user presses Stop Uploading, the uploading must stop instantly, I can stop the thread using stop() however it is only checked if it should stop once the loop has looped over again.
So basically, it should be possible to use an asynchronous request that would let me check if it should be cancelled during the request, however, I don't know how.
Any suggestions? The previous part of the post is still relevant so it's below.
Please note that the initial question of how to terminate a QThread has been solved, and as such the code below isn't too important, it's just for context, the thing I need help with now is what I just described.
I've been writing a program, it's a photo uploader, I've made a thread
in which I upload the files. I can't figure out how to exit the
thread. I've tried suggestions i've read from here:
1) I've tried a bool flag, wrapping it both around the method and the
for statement that does the work.
2) I've use a 'with' and then tried to set an exception.
I want to be able to cancel uploading, preferably quickly. I've read a
lot that it's always recommended to "clean up" the thread before
terminating it, I honestly don't know what 'clean it up' means.
But, I think I should be able to just kill the thread - since all it's
doing is sending the binary data of an image to the TUMBLR api. It
shouldn't matter if the request is cancelled early, because It will
just cancel the upload in the api too.
Anyway, here is my thread:
class WorkThread(QtCore.QThread):
def __init__(self):
QtCore.QThread.__init__(self)
global w
w = WorkThread
def __del__(slf):
self.wait()
def run(self):
url = 'https://www.tumblr.com/api/write'
files = os.listdir(directory)
for file in files:
file_path = os.path.join(directory + '\\' + file)
file_path = str(file_path)
if file_path[-3:] in ['bmp', 'gif', 'jpg', 'png', 'thm', 'tif', 'yuv']:
self.emit(QtCore.SIGNAL('update(QString)'), "Uploading %s ..." % file_path)
print smart_str(password)
data = {'email': email, 'password': password, 'type': 'photo'}
with open(file_path, 'rb') as photo:
r = requests.post(url, data=data, files={'data': photo})
print r.content
self.emit(QtCore.SIGNAL('update(QString)'), 'All finished! Go check out your stuff on tumblr :)\n')
return
This is how im calling it.
def upload(self):
self.doneBox.clear()
self.workThread = WorkThread()
self.connect( self.workThread, QtCore.SIGNAL("update(QString)"), self.startUploading )
self.workThread.start()
Can anyone suggest in a way that I can terminate the thread quickly
and quietly? OR if you think that's not good, a way to stop it safely.
HOWEVER, If I do not kill it instantly, and it goes through the for
loop again in the run() method, It will upload the photo that it was
uploading WHEN the user presses "Stop Uploading". I
wouldn't want it to do that, I'd prefer to have it stop uploading the
current photo the second the user presses "Stop Uploading".
Thanks.
I'm not sure what you are doing with that global w; w = WorkThread but it seems pointless since you aren't doing anything with it.
In your __init__() you want a flag:
def __init__(self):
...
self._isRunning = False
In your run() method you just check for this flag and exit when needed:
def run(self):
self._isRunning = True
while True:
if not self._isRunning:
# clean up or close anything here
return
# do rest of work
And maybe add a simple convenience method:
def stop(self):
self._isRunning = False
self.wait()
When documentation refers to cleaning up they are talking about closing down resources that you may have opened, undoing things that might be partially started, or anything else that you might want to signal or handle before just killing the thread object. In the case of your uploader, you want to check if the thread should exit before you start each upload process. This will give you the chance to stop the thread and have it exit before it continues with another bit of work.
Try terminating it:
self.workThread.terminate()
Just be careful while doing this. Since you are performing a network operation, you can't stop it, but usually you do this:
# Logic...
if self.running:
# Continue logic...
You set self.running from outside of the thread.
Related
I'm having trouble figuring out how to receive the WM_ENDSESSION window manager message in Tkinter, that is supposed to be sent to the top-level window during system shutdown. I am aware of everything that comes with trying to run extra code during shutdown, however in my case it'll only be a simple file flush down to disk and close, so that my program has a chance to save it's state.
My simple test code:
import tkinter as tk
def on_close(*args):
print("User closed the window")
# I want this to run during shutdown
def on_shutdown(*args):
print("User is shutting down their PC")
root = tk.Tk()
root.protocol("WM_DELETE_WINDOW", on_close)
root.mainloop()
A code example, or at least a pointer towards which functions or methods to use, would be appreciated.
I would suggest using a message spying too like "Spy++" (spyxx.exe) to check which messages are actually sent to your window.
In some places it is suggested that wireshark can also do this, but I have not found concrete evidence for that.
Tk gives your program "a chance to save it's state" with the aptly named WM_SAVE_YOURSELF protocol. It is hooked up to WM_QUERYENDSESSION on Windows. The return value seems to be ignored so you cannot prevent WM_ENDSESSION from being dispatched, and therefore it shouldn't be used for long blocking work, so it really depends on how much you have to write if things will get weird or interrupted.
Hook up your callback with root.protocol("WM_SAVE_YOURSELF", on_close) and let me know if that works!
As you, I found out (with respect for the issue) tk.Tk().protocol is pretty useless and your desired behavior is a wish since 1999. In the link is a patch you could try out.
I think the easiest way to go for you will be SetConsoleCtrlHandler:
This function provides a similar notification for console application
and services that WM_QUERYENDSESSION provides for graphical
applications with a message pump. You could also use this function
from a graphical application, but there is no guarantee it would
arrive before the notification from WM_QUERYENDSESSION.
You could find an exampel on StackOverflow or go with Pywin32. The disadvantage over the message approach is that you can't delay the shutdown further more. But the User would have this option anyway, after the timeout of 5 seconds is expired and you would have with SetConsoleHandler the same amount of time.
A word of caution, flushing a bigger amount of data is explicitly discouraged:
Avoid disk flushes, for example, those initiated through calling
FlushFileBuffers API. Flushing causes the disk stack to delete its
caches and is supposed to force the hard drive to write out data in
its RAM buffers. Typically, this operation is very costly and does not
guarantee data consistency since the hard drives often ignore the
request.
...
It is recommended that applications save their data and state
frequently; for example, automatically save data between save
operations
In addition there is an example with tkinter and pywin32 that handles WM_ENDSESSION, if you intrested.
After many struggles of trying to find information online and failing to get my code to work, this is what eventually ended up working for me in the end. It's based on the https://stackoverflow.com/a/58622778/13629335 answer, suggested by #furas and #Thingamabobs. I've added helpful comments explaining what each section does. Feel free to adjust it up to your own needs.
import sys
import time
import ctypes
import logging
import tkinter as tk
from tkinter import ttk
import win32con
import win32gui
import win32api
# NOTE: this is used in the root.mainloop() replacement at the bottom.
exit_requested = False
def quit():
global exit_requested
exit_requested = True
# Received when the user tries to close the window via X in the top right corner.
def close_window(*args, **kwargs):
logging.info("WM_CLOSE received")
quit()
# 0 tells Windows the message has been handled
return 0
# Received when the system is shutting down.
def end_session(*args, **kwargs):
logging.info("WM_ENDSESSION received")
quit()
# Returning immediately lets Windows proceed with the shutdown.
# You can run some shutdown code here, but there's a 5 seconds maximum timeout,
# before your application is killed by Windows.
return 0
# Received when the system is about to shutdown, but the user can
# cancel this action. Return 0 to tell the system to wait until
# the application exits first. No timeout.
def query_end_session(*args, **kwargs):
logging.info("WM_QUERYENDSESSION received")
quit()
# 1 means you're ready to exit, and you'll receive a WM_ENDSESSION immediately afterwards.
# 0 tells Windows to wait before proceeding with the shutdown.
return 0
# Simple logging setup to catch all logging messages into a file.
file_handler = logging.FileHandler("shutdown_test.log", encoding="utf8")
file_handler.setFormatter(logging.Formatter("%(asctime)s: %(message)s"))
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
root_logger.addHandler(file_handler)
logging.info("starting shutdown test")
# Start of your application code
root = tk.Tk()
root.title("Shutdown test")
main_frame = ttk.Frame(root, padding=20)
main_frame.grid(column=0, row=0)
ttk.Label(
main_frame, text="Shutdown test in progress...", padding=50
).grid(column=0, row=0, sticky="nsew")
# End of your application code
# This is crucial - a root.update() after all application setup is done,
# is very needed here, otherwise Tk won't properly set itself up internally,
# leading to not being able to catch any messages later.
root.update()
# NOTE: These two lines below can be used for basic message handling instead.
# Return value from WM_SAVE_YOURSELF is ignored, so you're expected to
# finish all of the closing sequence before returning. Note that Windows will wait
# for you up to 5 seconds, and then proceed with the shutdown anyway.
# root.protocol("WM_DELETE_WINDOW", close_window)
# root.protocol("WM_SAVE_YOURSELF", query_end_session)
root_handle = int(root.wm_frame(), 16)
message_map = {
win32con.WM_CLOSE: close_window,
win32con.WM_ENDSESSION: end_session,
win32con.WM_QUERYENDSESSION: query_end_session,
}
def wnd_proc(hwnd, msg, w_param, l_param):
"""
This function serves as a message processor for all messages sent to your
application by Windows.
"""
if msg == win32con.WM_DESTROY:
win32api.SetWindowLong(root_handle, win32con.GWL_WNDPROC, old_wnd_proc)
if msg in message_map:
return message_map[msg](w_param, l_param)
return win32gui.CallWindowProc(old_wnd_proc, hwnd, msg, w_param, l_param)
# This hooks up the wnd_proc function as the message processor for the root window.
old_wnd_proc = win32gui.SetWindowLong(root_handle, win32con.GWL_WNDPROC, wnd_proc)
if old_wnd_proc == 0:
raise NameError("wndProc override failed!")
# This works together with WM_QUERYENDSESSION to provide feedback to the user
# in terms of what's preventing the shutdown from proceeding.
# NOTE: It's sort-of optional. If you don't include it, Windows will use
# a generic message instead. However, your application can fail to receive
# a WM_QUERYENDSESSION if it's window is minimized (via iconify/withdraw)
# when the message happens - if you also need to be able to handle that case,
# then you'll need it.
retval = ctypes.windll.user32.ShutdownBlockReasonCreate(
root_handle, ctypes.c_wchar_p("I'm still saving data!")
)
if retval == 0:
raise NameError("shutdownBlockReasonCreate failed!")
# NOTE: this replaces root.mainloop() to allow for a loop exit
# without closing any windows - root.quit() apparently does so.
while not exit_requested:
root.update()
time.sleep(0.05)
# Your shutdown sequence goes here.
logging.info("shutdown start")
time.sleep(10)
logging.info("shutdown finished")
root.destroy()
sys.exit(0)
I am trying to using python download a batch of files, and I use requests module with stream turned on, in other words, I retrieve each file in 200K blocks.
However, sometimes, the downloading may stop as it just gets stuck (no response) and there is no error. I guess this is because the connection between my computer and server was not stable enough. Here is my question, how to check this kind of stop and make a new connection?
You probably don't want to detect this from outside, when you can just use timeouts to have requests fail instead of stopping is the server stops sending bytes.
Since you didn't show us your code, it's hard to show you how to change it… but I'll show you how to change some other code:
# hanging
text = requests.get(url).text
# not hanging
try:
text = requests.get(url, timeout=10.0).text
except requests.exceptions.Timeout:
# failed, do something else
# trying until success
while True:
try:
text = requests.get(url, timeout=10.0).text
break
except requests.exceptions.Timeout:
pass
If you do want to detect it from outside for some reason, you'll need to use multiprocessing or similar to move the requests-driven code to a child process. Ideally you'll want it to post updates on some Queue or set and notify some Condition-protected shared flag variable every 200KB, then the main process can block on the Queue or Condition and kill the child process if it times out. For example (pseudocode):
def _download(url, q):
create request
for each 200kb block downloaded:
q.post(buf)
def download(url):
q = multiprocessing.Queue()
with multiprocessing.Process(_download, args=(url, q)) as proc:
try:
return ''.join(iter(functools.partial(q.get, timeout=10.0)))
except multiprocessing.Empty:
proc.kill()
# failed, do something else
Below is a function I am using. When I call the command the spawn a putty session, it completely freezes up my gui program until I close the putty session. Is there any way to get around this, so that it calls the to the command, and then just moves along? (there is more I am passing to the command, but I have removed it to clean it up.
def SpawnSessionrmt(self,event):
if "SSH" in self.protormt:
subprocess.call('C:/bin/putty.exe ',shell=True)
elif "RDP" in self.protormt:
subprocess.call('C:/bin/rdp.exe)
else:
print "Not that you will see this...but that isn't a valid protocol"
The problem is that, as the docs say, call will:
Run the command described by args. Wait for command to complete, then return the returncode attribute.
If you don't want to wait for the command to complete, don't use call. Just create a Popen instance (and, ideally, wait for it later on, maybe at exit time, or in a background thread).
For example:
def SpawnSessionrmt(self,event):
if "SSH" in self.protormt:
self.children.append(subprocess.Popen('C:/bin/putty.exe ',shell=True))
elif "RDP" in self.protormt:
self.children.append(subprocess.Popen('C:/bin/rdp.exe'))
else:
print "Not that you will see this...but that isn't a valid protocol"
def WaitForChildren(self):
for child in self.children:
child.wait()
For a GUI app, I think the simplest thing to do might be to just put the subprocess.call into a background thread. That way, you can update the GUI when the actual work is done.
Unfortunately, each GUI framework has a different way of doing that—some let you do GUI stuff from any thread, some have a run_on_main_thread function, some let you post events to the main thread to be picked up by its event loop, some require you to build your own inter-thread communication system, etc. And you didn't tell us which GUI framework you're using.
So, here's an example with one framework I picked at random, wx:
def SpawnSessionrmt(self,event):
if "SSH" in self.protormt:
cmd = 'C:/bin/putty.exe'
elif "RDP" in self.protormt:
cmd = 'C:/bin/rdp.exe'
else:
print "Not that you will see this...but that isn't a valid protocol"
return
def background_function():
result = subprocess.call(cmd)
event = wx.CommandEvent(MY_UPDATE_ID)
event.SetInt(result)
self.GetEventHandler().AddPendingEvent(event)
t = threading.Thread(target=background_function)
t.daemon = True
t.start()
(PS, I hate my random number generator, because I hate wx… but at least it didn't pick Tkinter, which would have forced me to write my own cross-thread communication around a Queue or Condition…)
I'm trying to figure out how to properly close an asynchronous tweepy stream.
The tweepy streaming module can be found here.
I start the stream like this:
stream = Stream(auth, listener)
stream.filter(track=['keyword'], async=True)
When closing the application, I try to close the stream as simple as:
stream.disconnect()
This method seems to work as intended but it seems to have one problem:
the stream thread is still in the middle of the loop (waiting/handling tweets) and is not killed until the next loop, so when the stream receives a tweet even after the app has closed, it still tries to call the listener object (this can be seen with a simple print syntax on the listener object). I'm not sure if this is a bad thing or if it can simply be ignored.
I have 2 questions:
Is this the best way to close the stream or should I take a different approach?
Shouldn't the async thread be created as a daemon thread?
I had the same problem. I fixed it with restarting the script. Tweepy Stream doesn't stop until the next incoming tweet.
Example:
import sys
import os
python=sys.executable
time.sleep(10)
print "restart"
os.execl(python,python,*sys.argv)
I didn't find another solution.
I am not positive that it applies to your situation, but in general you can have applicable entities clean up after themselves by putting them in a with block:
with stream = Stream(auth, listener):
stream.filter(track=['keyword'], async=True)
# ...
# Outside the with-block; stream is automatically disposed of.
What "disposed of" actually means, it that the entities __exit__ function is called.
Presumably tweepy will have overridden that to Do The Right Thing.
As #VooDooNOFX suggests, you can check the source to be sure.
This is by design. Looking at the source, you will notice that disconnect has no immediate termination option.
def disconnect(self):
if self.running is False:
return
self.running = False
When calling disconnect(), it simply sets self.running = False, which is then checked on the next loop of the _run method
You can ignore this side effect.
Instead of restarting the script, as #burkay suggests, I finally deleted the Stream object and started a new one. In my example, someone wants to add a new user to be followed, so I update the track list this way:
stream.disconnect() # that should wait until next tweet, so let's delete it
del stream
# now, create a new object
stream = tweepy.Stream( auth=api.auth, listener=listener )
stream.userstream( track=all_users(), async=True )
I am stuck reading a file in /sys/ which contains the light intensity in Lux of the ambient light sensor on my Nokia N900 phone.
See thread on talk.maemo.org here
I tried to use pyinotify to poll the file but this looks some kind of wrong to me since the file is alway "process_IN_OPEN", "process_IN_ACCESS" and "process_IN_CLOSE_NOWRITE"
I basically want to get the changes ASAP and if something changed trigger an event, execute a class...
Here's the code I tried, which works, but not as I expected (I was hoping for process_IN_MODIFY to be triggered):
#!/usr/bin/env python
import os, time, pyinotify
import pyinotify
ambient_sensor = '/sys/class/i2c-adapter/i2c-2/2-0029/lux'
wm = pyinotify.WatchManager() # Watch Manager
mask = pyinotify.ALL_EVENTS
def action(self, the_event):
value = open(the_event.pathname, 'r').read().strip()
return value
class EventHandler(pyinotify.ProcessEvent):
...
def process_IN_MODIFY(self, event):
print "MODIFY event:", action(self, event)
...
#log.setLevel(10)
notifier = pyinotify.ThreadedNotifier(wm, EventHandler())
notifier.start()
wdd = wm.add_watch(ambient_sensor, mask)
wdd
time.sleep(5)
notifier.stop()
Update 1:
Mmmh, all I came up without having a clue if there is a special mechanism is the following:
f = open('/sys/class/i2c-adapter/i2c-2/2-0029/lux')
while True:
value = f.read()
print value
f.seek(0)
This, wrapped in a own thread, could to the trick, but does anyone have a smarter, less CPU-hogging and faster way to get the latest value?
Since the /sys/file is a pseudo-file which just presents a view on an underlying, volatile operating system value, it makes sense that there would never be a modify event raised. Since the file is "modified" from below it doesn't follow regular file-system semantics.
If a modify event is never raised, using a package like pinotify isn't going to get you anywhere. 'twould be better to look for a platform-specific mechanism.
Response to Update 1:
Since the N900 maemo runtime supports GFileMonitor, you'd do well to check if it can provide the asynchronous event that you desire.
Busy waiting - as I gather you know - is wasteful. On a phone it can really drain a battery. You should at least sleep in your busy loop.
Mmmh, all I came up without having a clue if there is a special mechanism is the following:
f = open('/sys/class/i2c-adapter/i2c-2/2-0029/lux')
while True:
value = f.read()
print value
f.seek(0)
This, wrapped in a own thread, could to the trick, but does anyone have a smarter, less CPU-hogging and faster way to get the latest value?
Cheers
Bjoern