python-running-autobahnpython-asyncio-websocket-server-in-a-separate-subproce
can-an-asyncio-event-loop-run-in-the-background-without-suspending-the-python-in
Was trying to solve my issue with this two links above but i have not.
I have the following error : RuntimeError: There is no current event loop in thread 'Thread-1'.
Here the code sample (python 3):
from autobahn.asyncio.wamp import ApplicationSession
from autobahn.asyncio.wamp import ApplicationRunner
from asyncio import coroutine
import time
import threading
class PoloniexWebsocket(ApplicationSession):
def onConnect(self):
self.join(self.config.realm)
#coroutine
def onJoin(self, details):
def on_ticker(*args):
print(args)
try:
yield from self.subscribe(on_ticker, 'ticker')
except Exception as e:
print("Could not subscribe to topic:", e)
def poloniex_worker():
runner = ApplicationRunner("wss://api.poloniex.com:443", "realm1")
runner.run(PoloniexWebsocket)
def other_worker():
while True:
print('Thank you')
time.sleep(2)
if __name__ == "__main__":
polo_worker = threading.Thread(None, poloniex_worker, None, (), {})
thank_worker = threading.Thread(None, other_worker, None, (), {})
polo_worker.start()
thank_worker.start()
polo_worker.join()
thank_worker.join()
So, my final goal is to have 2 threads launched at the start. Only one need to use ApplicationSession and ApplicationRunner. Thank you.
A separate thread must have it's own event loop. So if poloniex_worker needs to listen to a websocket, it needs its own event loop:
def poloniex_worker():
asyncio.set_event_loop(asyncio.new_event_loop())
runner = ApplicationRunner("wss://api.poloniex.com:443", "realm1")
runner.run(PoloniexWebsocket)
But if you're on a Unix machine, you will face another error if you try to do this. Autobahn asyncio uses Unix signals, but those Unix signals only work in the main thread. You can simply turn off Unix signals if you don't plan on using them. To do that, you have to go to the file where ApplicationRunner is defined. That is wamp.py in python3.5 > site-packages > autobahn > asyncio on my machine. You can comment out the signal handling section of the code like so:
# try:
# loop.add_signal_handler(signal.SIGTERM, loop.stop)
# except NotImplementedError:
# # signals are not available on Windows
# pass
All this is a lot of work. If you don't absolutely need to run your ApplicationSession in a separate thread from the main thread, it's better to just run the ApplicationSession in the main thread.
Related
I have a function that downloads videos from specific URLs and I launch this function through a thread to avoid GUI freezing, but I want a function to stop or pause the download. How to do this?
Here is the code:
def download_videos(self):
ydl1 = youtube_dl.YoutubeDL(self.get_opts())
if self.get_Urls().__len__() > 0:
ydl1.download(self.get_Urls())
def downloadVideoThrd(self):
self.t1 = threading.Thread(target=self.download_videos())
self.t1.start()
You can use these two options. Use multiprocessing library instead threading or raise SystemExit exception in the subthread
Note: When i tested youtube-dl, it is resume downloading from where it is left. That's why when we start to download same url, youtube-dl will resume, but downloaded file need to be in the filesystem
Here is the first option, in this solution, we not use thread, we use subprocess because we can send any signal to the subprocess and handle this signal whatever we want.
import multiprocessing
import time
import ctypes,signal,os
import youtube_dl
class Stop_Download(Exception): # This is our own special Exception class
pass
def usr1_handler(signum,frame): # When signal comes, run this func
raise Stop_Download
def download_videos(resume_event):
signal.signal(signal.SIGUSR1,usr1_handler)
def download(link):
ydl1 = youtube_dl.YoutubeDL()
ydl1.download([link])
try:
download(link)
except Stop_Download: #catch this special exception and do what ever you want
print("Stop_Download exception")
resume_event.wait() # wait for event to resume or kill this process by sys.exit()
print("resuming")
download(link)
def downloadVideoThrd():
resume_event=multiprocessing.Event()
p1 = multiprocessing.Process(target=download_videos,args=(resume_event,))
p1.start()
print("mp start")
time.sleep(5)
os.kill(p1.pid,signal.SIGUSR1) # pause the download
time.sleep(5)
down_event.set() #resume downlaod
time.sleep(5)
os.kill(p1.pid,signal.SIGUSR2) # stop the download
downloadVideoThrd()
Here is the second solution, you can also check this for more detail about killing the thread. We will raise a SystemExit exception in the subthread by main thread. We can both stop or pause thread. To pause thread you can use threading.Event() class. To stop the thread(not process) you can use sys.exit()
import ctypes,time, threading
import youtube_dl,
def download_videos(self):
try:
ydl1 = youtube_dl.YoutubeDL(self.get_opts())
if self.get_Urls().__len__() > 0:
ydl1.download(self.get_Urls())
except SystemExit:
print("stopped")
#do whatever here
def downloadVideoThrd(self):
self.t1 = threading.Thread(target=self.download_videos)
self.t1.start()
time.sleep(4)
"""
raise SystemExit exception in self.t1 thread
"""
ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(self.t1.ident),
ctypes.py_object(SystemExit))
I've read a lot of questions on SO and elsewhere on this topic but can't get it working. Perhaps it's because I'm using Windows, I don't know.
What I'm trying to do is download a bunch of files (whose URLs are read from a CSV file) in parallel. I've tried using multiprocessing and concurrent.futures for this with no success.
The main problem is that I can't stop the program on Ctrl-C - it just keeps running. This is especially bad in the case of processes instead of threads (I used multiprocessing for that) because I have to kill each process manually every time.
Here is my current code:
import concurrent.futures
import signal
import sys
import urllib.request
class Download(object):
def __init__(self, url, filename):
self.url = url
self.filename = filename
def perform_download(download):
print('Downloading {} to {}'.format(download.url, download.filename))
return urllib.request.urlretrieve(download.url, filename=download.filename)
def main(argv):
args = parse_args(argv)
queue = []
with open(args.results_file, 'r', encoding='utf8') as results_file:
# Irrelevant CSV parsing...
queue.append(Download(url, filename))
def handle_interrupt():
print('CAUGHT SIGINT!!!!!!!!!!!!!!!!!!!11111111')
sys.exit(1)
signal.signal(signal.SIGINT, handle_interrupt)
with concurrent.futures.ThreadPoolExecutor(max_workers=args.num_jobs) as executor:
futures = {executor.submit(perform_download, d): d for d in queue}
try:
concurrent.futures.wait(futures)
except KeyboardInterrupt:
print('Interrupted')
sys.exit(1)
I'm trying to catch Ctrl-C in two different ways here but none of them works. The latter one (except KeyboardInterrupt) actually gets run but the process won't exit after calling sys.exit.
Before this I used the multiprocessing module like this:
try:
pool = multiprocessing.Pool(processes=args.num_jobs)
pool.map_async(perform_download, queue).get(1000000)
except Exception as e:
pool.close()
pool.terminate()
sys.exit(0)
So what is the proper way to add ability to terminate all worker threads or processes once you hit Ctrl-C in the terminal?
System information:
Python version: 3.6.1 32-bit
OS: Windows 10
You are catching the SIGINT signal in a signal handler and re-routing it as a SystemExit exception. This prevents the KeyboardInterrupt exception to ever reach your main loop.
Moreover, if the SystemExit is not raised in the main thread, it will just kill the child thread where it is raised.
Jesse Noller, the author of the multiprocessing library, explains how to deal with CTRL+C in a old blog post.
import signal
from multiprocessing import Pool
def initializer():
"""Ignore CTRL+C in the worker process."""
signal.signal(SIGINT, SIG_IGN)
pool = Pool(initializer=initializer)
try:
pool.map(perform_download, dowloads)
except KeyboardInterrupt:
pool.terminate()
pool.join()
I don't believe the accepted answer works under Windows, certainly not under current versions of Python (I am running 3.8.5). In fact, it won't run at all since SIGINT and SIG_IGN will be undefined (what is needed is signal.SIGINT and signal.SIG_IGN).
This is a know problem under Windows. A solution I have come up with is essentially the reverse of the accepted solution: The main process must ignore keyboard interrupts and we initialize the process pool to initially set a global flag ctrl_c_entered to False and to set this flag to True if Ctrl-C is entered. Then any multiprocessing worker function (or method) is decorated with a special decorator, handle_ctrl_c, that firsts tests the ctrl_c_entered flag and only if False does it run the worker function after re-enabling keyboard interrupts and establishing a try/catch handler for keyboard interrups. If the ctrl_c_entered flag was True or if a keyboard interrupt occurs during the execution of the worker function, the value returned is an instance of KeyboardInterrupt, which the main process can check to determine whether a Ctrl-C was entered.
Thus all submitted tasks will be allowed to start but will immediately terminate with a return value of a KeyBoardInterrupt exception and the actual worker function will never be called by the decorator function once a Ctrl-C has been entered.
import signal
from multiprocessing import Pool
from functools import wraps
import time
def handle_ctrl_c(func):
"""
Decorator function.
"""
#wraps(func)
def wrapper(*args, **kwargs):
global ctrl_c_entered
if not ctrl_c_entered:
# re-enable keyboard interrups:
signal.signal(signal.SIGINT, default_sigint_handler)
try:
return func(*args, **kwargs)
except KeyboardInterrupt:
ctrl_c_entered = True
return KeyboardInterrupt()
finally:
signal.signal(signal.SIGINT, pool_ctrl_c_handler)
else:
return KeyboardInterrupt()
return wrapper
def pool_ctrl_c_handler(*args, **kwargs):
global ctrl_c_entered
ctrl_c_entered = True
def init_pool():
# set global variable for each process in the pool:
global ctrl_c_entered
global default_sigint_handler
ctrl_c_entered = False
default_sigint_handler = signal.signal(signal.SIGINT, pool_ctrl_c_handler)
#handle_ctrl_c
def perform_download(download):
print('begin')
time.sleep(2)
print('end')
return True
if __name__ == '__main__':
signal.signal(signal.SIGINT, signal.SIG_IGN)
pool = Pool(initializer=init_pool)
results = pool.map(perform_download, range(20))
if any(map(lambda x: isinstance(x, KeyboardInterrupt), results)):
print('Ctrl-C was entered.')
print(results)
I'm working with asynchronous programming and wrote a small wrapper class for thread-safe execution of co-routines based on some ideas from this thread here: python asyncio, how to create and cancel tasks from another thread. After some debugging, I found that it hangs when calling the Thread class's join() function (I overrode it only for testing). Thinking I made a mistake, I basically copied the code that the OP said he used and tested it to find the same issue.
His mildly altered code:
import threading
import asyncio
from concurrent.futures import Future
import functools
class EventLoopOwner(threading.Thread):
class __Properties:
def __init__(self, loop, thread, evt_start):
self.loop = loop
self.thread = thread
self.evt_start = evt_start
def __init__(self):
threading.Thread.__init__(self)
self.__elo = self.__Properties(None, None, threading.Event())
def run(self):
self.__elo.loop = asyncio.new_event_loop()
asyncio.set_event_loop(self.__elo.loop)
self.__elo.thread = threading.current_thread()
self.__elo.loop.call_soon_threadsafe(self.__elo.evt_start.set)
self.__elo.loop.run_forever()
def stop(self):
self.__elo.loop.call_soon_threadsafe(self.__elo.loop.stop)
def _add_task(self, future, coro):
task = self.__elo.loop.create_task(coro)
future.set_result(task)
def add_task(self, coro):
self.__elo.evt_start.wait()
future = Future()
p = functools.partial(self._add_task, future, coro)
self.__elo.loop.call_soon_threadsafe(p)
return future.result() # block until result is available
def cancel(self, task):
self.__elo.loop.call_soon_threadsafe(task.cancel)
async def foo(i):
return 2 * i
async def main():
elo = EventLoopOwner()
elo.start()
task = elo.add_task(foo(10))
x = await task
print(x)
elo.stop(); print("Stopped")
elo.join(); print("Joined") # note: giving it a timeout does not fix it
if __name__ == "__main__":
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
assert isinstance(loop, asyncio.AbstractEventLoop)
try:
loop.run_until_complete(main())
finally:
loop.close()
About 50% of the time when I run it, It simply stalls and says "Stopped" but not "Joined". I've done some debugging and found that it is correlated to when the Task itself sent an exception. This doesn't happen every time, but since it occurs when I'm calling threading.Thread.join(), I have to assume it is related to the destruction of the loop. What could possibly be causing this?
The exception is simply: "cannot join current thread" which tells me that the .join() is sometimes being run on the thread from which I called it and sometimes from the ELO thread.
What is happening and how can I fix it?
I'm using Python 3.5.1 for this.
Note: This is not replicated on IDE One: http://ideone.com/0LO2D9
The following program hangs the terminal such that it ignores Ctrl+C. This is rather annoying since I have to restart the terminal every time one of the threads hang.
Is there any way to catch the KeyboardInterrupt while waiting on an event?
import threading
def main():
finished_event = threading.Event()
startThread(finished_event)
finished_event.wait()#I want to stop the program here
print('done!')
def startThread(evt):
"""Start a thread that will trigger evt when it is done"""
#evt.set()
if __name__ == '__main__':
main()
If you want to avoid polling, you can use the pause() function of the signal module instead of finished_event.wait(). signal.pause() is a blocking function and gets unblocked when a signal is received by the process. In this case, when ^C is pressed, SIGINT signal unblocks the function. Note that the function does not work on Windows according to the documentation. I've tried it on Linux and it worked for me.
I came across this solution in this SO thread.
Update: On the current Python 3 finished_event.wait() works on my Ubuntu machine (starting with Python 3.2). You don't need to specify the timeout parameter, to interrupt it using Ctrl+C. You need to pass the timeout parameter on CPython 2.
Here's a complete code example:
#!/usr/bin/env python3
import threading
def f(event):
while True:
pass
# never reached, otherwise event.set() would be here
event = threading.Event()
threading.Thread(target=f, args=[event], daemon=True).start()
try:
print('Press Ctrl+C to exit')
event.wait()
except KeyboardInterrupt:
print('got Ctrl+C')
There could be bugs related to Ctrl+C. Test whether it works in your environment.
Old polling answer:
You could try to allow the interpreter to run the main thread:
while not finished_event.wait(.1): # timeout in seconds
pass
If you just want to wait until the child thread is done:
while thread.is_alive():
thread.join(.1)
You could also patch the Event.wait() function in the following manner:
def InterruptableEvent():
e = threading.Event()
def patched_wait():
while not e.is_set():
e._wait(3)
e._wait = e.wait
e.wait = patched_wait
return e
>>> event = InterruptableEvent()
>>> try:
... event.wait()
... except KeyboardInterrupt:
... print "Received KeyboardInterrupt"
...
^CReceived KeyboardInterrupt
This works because wait() with a timeout argument will raise a KeyboardInterrupt.
Based on #Pete's answer, but with subclassing and using the actual Event.wait method, just with smaller timeouts to allow handling of KeyboardInterrupts and such in between:
class InterruptableEvent(threading.Event):
def wait(self, timeout=None):
wait = super().wait # get once, use often
if timeout is None:
while not wait(0.01): pass
else:
wait(timeout)
I have following code which compares user input
import thread,sys
if(username.get_text() == 'xyz' and password.get_text()== '123' ):
thread.start_new_thread(run,())
def run():
print "running client"
start = datetime.now().second
while True:
try:
host ='localhost'
port = 5010
time = abs(datetime.now().second-start)
time = str(time)
print time
client = socket.socket()
client.connect((host,port))
client.send(time)
except socket.error:
pass
If I just call the function run() it works but when I try to create a thread to run this function, for some reason the thread is not created and run() function is not executed I am unable to find any error..
Thanks in advance...
you really should use the threading module instead of thread.
what else are you doing? if you create a thread like this, then the interpreter will exit no matter if the thread is still running or not
for example:
import thread
import time
def run():
time.sleep(2)
print('ok')
thread.start_new_thread(run, ())
--> this produces:
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
where as:
import threading
import time
def run():
time.sleep(2)
print('ok')
t=threading.Thread(target=run)
t.daemon = True # set thread to daemon ('ok' won't be printed in this case)
t.start()
works as expected. if you don't want to keep the interpreter waiting for the thread, just set daemon=True* on the generated Thread.
*edit: added that in example
thread is a low level library, you should use threading.
from threading import Thread
t = Thread(target=run, args=())
t.start()