I have a code that only runs on Linux with asyncio, but I need it to run on Windows, my knowledge is little in multiprocessing, what would be the way to run on Windows? I saw that there is a limitation of asyncio to run on windows: https://docs.python.org/3/library/asyncio-platforms.html#asyncio-windows-subprocess
My code:
import sys
import asyncio
scriptspy = [
'scrp1.py',
'scrp2.py',
'scrp3.py',
'scrp4.py',
'scrp5.py',
'scrp6.py',
'scrp7.py',
'scrp8.py',
'scrp9.py',
'scrp10.py',
'scrp11.py',
'scrp12.py',
]
async def main():
tarefas_rodando = set()
while scriptspy:
# start up to 6 scripts
while len(tarefas_rodando) < 6 and scriptspy:
script = scriptspy.pop()
p = await asyncio.create_subprocess_exec(sys.executable, script)
tarefa = asyncio.create_task(p.wait())
tarefas_rodando.add(tarefa)
# wait for one of the scripts to end
tarefas_rodando, finalizadas = await asyncio.wait(tarefas_rodando, return_when=asyncio.FIRST_COMPLETED)
# finished, wait for the rest to finish
await asyncio.wait(tarefas_rodando, return_when=asyncio.ALL_COMPLETED)
asyncio.run(main())
This code runs fine on linux but not windows. I need to run it on windows. https://docs.python.org/3/library/asyncio-platforms.html#asyncio-windows-subprocess
Exception run in Windows:
Traceback (most recent call last):
File "scripts.py", line 53, in <module>
asyncio.run(main())
File "C:\Python\Python37\lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Python\Python37\lib\asyncio\base_events.py", line 584, in run_until_complete
return future.result()
File "scripts.py", line 44, in main
p = await asyncio.create_subprocess_exec(sys.executable, script)
File "C:\Python\Python37\lib\asyncio\subprocess.py", line 217, in create_subprocess_exec
stderr=stderr, **kwds)
File "C:\Python\Python37\lib\asyncio\base_events.py", line 1533, in subprocess_exec
bufsize, **kwargs)
File "C:\Python\Python37\lib\asyncio\base_events.py", line 463, in
_make_subprocess_transport
raise NotImplementedError NotImplementedError
According to the documentation, to run subprocesses on Windows, you need to switch to the proactor event loop:
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
Related
I am trying to make some API calls to firebase using the google.cloud.firestore.AsyncClient module. Fire2 is just a wrapper for this object.
The traceback suggests its a problem with asyncio, but if I make a dummy await it actually runs just fine. What is the problem and why does it occur?
Sample code:
import Fire2
class Test:
def __init__(self):
doc = asyncio.run(self.wait())
async def wait(self):
doc = await Fire2("test1").get(g1) # gives the error
# doc = await asyncio.sleep(1) # runs without error
return doc
def test(self):
x = Test2(p1)
class Test2:
def __init__(self, p):
doc = asyncio.run(self.run(p))
print(doc.to_dict())
async def run(self, p):
doc = await Fire2('test2').get(p)
return doc
p1 = 'foo'
g1 = 'bar'
h = Test()
h.test()
Traceback:
Traceback (most recent call last):
File "<project path>\scratch.py", line 137, in <module>
h.test()
File "<project path>\scratch.py", line 123, in test
x = Test2(p1)
File "<project path>\scratch.py", line 127, in __init__
doc = asyncio.run(self.run(p))
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\runners.py", line 44, in run
return loop.run_until_complete(main)
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 642, in run_until_complete
return future.result()
File "<project path>\scratch.py", line 131, in run
doc = await Fire2('test2').get(p)
File "<project path>\<Fire2 file>", line 422, in get
res = await self._collection.document(doc_id).get()
File "<project path>\venv\lib\site-packages\google\cloud\firestore_v1\async_document.py", line 364, in get
response_iter = await self._client._firestore_api.batch_get_documents(
File "<project path>\venv\lib\site-packages\google\api_core\grpc_helpers_async.py", line 171, in error_remapped_callable
call = callable_(*args, **kwargs)
File "<project path>\venv\lib\site-packages\grpc\aio\_channel.py", line 165, in __call__
call = UnaryStreamCall(request, deadline, metadata, credentials,
File "<project path>\venv\lib\site-packages\grpc\aio\_call.py", line 553, in __init__
self._send_unary_request_task = loop.create_task(
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 431, in create_task
self._check_closed()
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
sys:1: RuntimeWarning: coroutine 'UnaryStreamCall._send_unary_request' was never awaited
Process finished with exit code 1
You are making two separate calls to asyncio.run. We can see from the stack trace that the error is coming from inside the grpc library. I suspect that the same grpc connection is being used both times, and that during its initialization it integrates itself with the async event loop in order to maintain the connection in the background between calls.
The problem is that every time you call asyncio.run, you will create a new event loop, and every time it returns that event loop will be closed. So by the time you call this function again, and the grpc layer tries to do its work, it realizes that the original event loop has been closed and it returns this error.
To fix this, You can have a single call to asyncio.run, typically calling a "main" function, or similar. This would then make calls to async functions which perform the actual work.
If the problem persists then there might be some issues in the “Fire2”class.
For more reference you can check this link which outlines high-level asyncio APIs to work with coroutines and Tasks with the help of various examples.
Use pytest with using fixture with scope="session" in conftest.py
#pytest.fixture(scope="session")
def event_loop():
loop = get_event_loop()
yield loop
loop.close()
I am trying to run a simple asyncio loop that I want to run continuously until Control + C is pressed, which should stop the main function and run a function to shutdown the loop. I am using the following code:
import asyncio
async def run():
while True:
print("Running")
async def shutdown():
print("Starting to shutdown")
await asyncio.sleep(1)
print("Finished sleeping")
try:
loop = asyncio.get_event_loop()
loop.create_task(run())
loop.run_forever()
except KeyboardInterrupt:
loop.stop()
loop.run_until_complete(asyncio.gather(*[shutdown()]))
When running this code and pressing Control + C the script doesn't shutdown gracefully as I expected (also based on this stackoverflow answer but I get the following error:
Traceback (most recent call last):
File "test.py", line 25, in <module>
loop.run_until_complete(asyncio.gather(*[shutdown()]))
File "/usr/lib/python3.7/asyncio/base_events.py", line 585, in run_until_complete
raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<run() done, defined at test.py:8> exception=KeyboardInterrupt()>
Traceback (most recent call last):
File "test.py", line 22, in <module>
loop.run_forever()
File "/usr/lib/python3.7/asyncio/base_events.py", line 541, in run_forever
self._run_once()
File "/usr/lib/python3.7/asyncio/base_events.py", line 1786, in _run_once
handle._run()
File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "test.py", line 10, in run
print("Running")
KeyboardInterrupt
Can anyone tell me where I am going wrong and how I should change my code to solve the issue?
There are lots of coroutines in my production code, which are stuck at unknown position while processing request. I attached gdb with Python support extension to the process, but it doesn't show the exact line in the coroutine where the process is stuck, only primary stack trace. Here is a minimal example:
import asyncio
async def hello():
await asyncio.sleep(30)
print('hello world')
asyncio.run(hello())
(gdb) py-bt
Traceback (most recent call first):
File "/usr/lib/python3.8/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
File "/usr/lib/python3.8/asyncio/base_events.py", line 2335, in _run_once
File "/usr/lib/python3.8/asyncio/base_events.py", line 826, in run_forever
None, getaddr_func, host, port, family, type, proto, flags)
File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
self.run_forever()
File "/usr/lib/python3.8/asyncio/runners.py", line 299, in run
File "main.py", line 7, in <module>
GDB shows a trace that ends on line 7, but the code is obviously stuck on line 4. How to make it show a more complete trace with nested coroutines?
You can use the aiodebug.log_slow_callbacks.enable(0.05)
Follow for more : https://pypi.org/project/aiodebug/
I just simply use the tornado application together with threading as the following code:
def MakeApp():
return tornado.web.Application([(r"/websocket", EchoWebSocket), ])
def run_tornado_websocket():
app = MakeApp()
http_server = tornado.httpserver.HTTPServer(app, ssl_options={
"certfile": os.path.join(os.path.abspath("."), "server.crt"),
"keyfile": os.path.join(os.path.abspath("."), "server_no_passwd.key"),
})
http_server.listen(options.port)
tornado.ioloop.IOLoop.current().start()
if __name__ == '__main__':
threads = []
t = threading.Thread(target=run_tornado_websocket, args=())
threads.append(t)
for t in threads:
t.start()
It works fine on python3.5.But it fails on python3.6 and the lasted tornado.It gets the error:
Exception in thread Thread-1:
Traceback (most recent call last):
File "D:\python3\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "D:\python3\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "D:\ssl\ws_server.py", line 49, in run_tornado_websocket
http_server.listen(options.port)
File "D:\python3\lib\site-packages\tornado\tcpserver.py", line 145, in listen
self.add_sockets(sockets)
File "D:\python3\lib\site-packages\tornado\tcpserver.py", line 159, in add_sockets
sock, self._handle_connection)
File "D:\python3\lib\site-packages\tornado\netutil.py", line 219, in add_accept_handler
io_loop = IOLoop.current()
File "D:\python3\lib\site-packages\tornado\ioloop.py", line 282, in current
loop = asyncio.get_event_loop()
File "D:\python3\lib\asyncio\events.py", line 694, in get_event_loop
return get_event_loop_policy().get_event_loop()
File "D:\python3\lib\asyncio\events.py", line 602, in get_event_loop
% threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Thread-1'.
I think there are some changes in IOLOOP in python3.6.But I don't know how to fix this and really want to know the reason.
Starting in Tornado 5.0, the asyncio event loop is used by default. asyncio has some extra restrictions because starting event loops on threads other than the main thread is an uncommon pattern and is often a mistake. You must tell asyncio that you want to use an event loop in your new thread with asyncio.set_event_loop(asyncio.new_event_loop()), or use asyncio.set_event_loop_policy(tornado.platform.asyncio.AnyThreadEventLoopPolicy()) to disable this restriction.
first: I am an absolute beginner in python, I used to write PHP before, so if I am getting something complitly wrong please let me know.
I am writing an app. It should serve its information via websockets. I choosed flask-socketio for this. In the background I want to process the data. Because I would like to have the app small I decided against a solution like Celery.
I have shortened the code to:
# -*- coding: utf8 -*-
from flask import Flask, jsonify, abort, make_response, url_for, request, render_template
from flask.ext.socketio import SocketIO, emit
from multiprocessing import Pool
from multiprocessing.managers import BaseManager
import time
import os
def background_stuff(args):
while True:
try:
print args
time.sleep(1)
except Exception as e:
return e
thread = None
_pool = None
app = Flask(__name__)
app.debug = True
socketio = SocketIO(app)
#app.route('/', methods=['GET'])
def get_timers():
return 'timer'
if __name__=='__main__':
_pool = Pool(1)
if os.environ.get('WERKZEUG_RUN_MAIN') == 'true':
workers = _pool.apply_async(
func=background_stuff,
args=('do background stuff',),
)
socketio.run(app)
# app.run()
When starting this, i get the following messages:
python test/multitest.py
* Running on http://127.0.0.1:5000/
* Restarting with stat
do background stuff
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 336, in _handle_tasks
for taskseq, set_length in iter(taskqueue.get, None):
File "/usr/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File "/usr/lib/python2.7/threading.py", line 340, in wait
waiter.acquire()
File "gevent/_semaphore.pyx", line 112, in gevent._semaphore.Semaphore.acquire (gevent/gevent._semaphore.c:3386)
File "/home/phil/work/ttimer/server/local/lib/python2.7/site-packages/gevent/hub.py", line 338, in switch
return greenlet.switch(self)
LoopExit: This operation would block forever
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
127.0.0.1 - - [2015-09-30 00:06:23] "GET / HTTP/1.1" 200 120 0.001860
do background stuff
do background stuff
do background stuff
do background stuff
^CProcess PoolWorker-1:
Process PoolWorker-1:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 113, in worker
File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
result = (True, func(*args, **kwds))
File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
File "test/multitest.py", line 14, in background_stuff
KeyboardInterrupt
time.sleep(1)
KeyboardInterrupt
return recv()
KeyboardInterrupt
So the background process is working and it answers http requests (127.0.0.1 - - [2015-09-30 00:06:23] "GET / HTTP/1.1" 200 120 0.001860). But just ignoring an error because it seems to work does not seem to be a solution for me. Does anyone can tell my what I am doing wrong here?
If you say I can't do it that way can you tell me why? I would like to learn and understand what I am doing wrong.
I read something about monkepatching, but everything suggested threw just more or other errors. I think it is better to work on the first error instead of blindly trying fixes.
python -V
Python 2.7.9
Greetings
update
I added the 2 lines for monkeypatching, this is what I got:
$python multitest2.py
^CProcess PoolWorker-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
return recv()
KeyboardInterrupt
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 380, in _handle_results
task = get()
KeyboardInterrupt
* Running on http://127.0.0.1:5000/
* Restarting with stat
^CProcess PoolWorker-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 380, in _handle_results
task = get()
KeyboardInterrupt
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
return recv()
KeyboardInterrupt
do background stuff
FAILED to start flash policy server: [Errno 98] Address already in use: ('127.0.0.1', 10843)
$do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
On start there is no output at all. After hittinc ctrl-c several times, i get the background stuff output. This continues until i kill the python process via SIGKILL
update 2
what I expect to see is
* Running on http://127.0.0.1:5000/
* Restarting with stat
do background stuff
do background stuff
do background stuff
right after the running of the script. But before I press ctrl-c nothing is happening.
First of all, you need to be aware that the version of Flask-SocketIO that you are using requires gevent, which is a coroutine framework. Using the asynchronous coroutines of gevent with a multiprocessing pool is a strange combination. You are using gevent, so what would make the most sense is to use the gevent pool functionality so that everything is consistent.
Now regarding the problem, I think it is likely due to not having the standard library monkey patched at an early stage. I recommend that you add the following lines at the very top of your script (above your imports, make them lines 1 and 2):
from gevent import monkey
monkey.patch_all()
These will ensure that any calls into standard library for things such as threads, semaphores, etc. go to the gevent implementations.
Update: I tried your example. The original version, without monkey-patching, works fine for me, I do not see the LoopExit error that you reported. Adding the monkey patching prevents the background processes from running, as you reported.
In any case, I converted your script to use gevent.pool and that works reliably for me. Here is the edited script:
from flask import Flask, jsonify, abort, make_response, url_for, request, render_template
from flask.ext.socketio import SocketIO, emit
from gevent.pool import Pool
import time
import os
def background_stuff(args):
while True:
try:
print args
time.sleep(1)
except Exception as e:
return e
thread = None
_pool = None
app = Flask(__name__)
app.debug = True
socketio = SocketIO(app)
#app.route('/', methods=['GET'])
def get_timers():
return 'timer'
if __name__=='__main__':
_pool = Pool(1)
workers = _pool.apply_async(
func=background_stuff,
args=('do background stuff',),
)
socketio.run(app)
Hope this helps!
I read a tutorial about gevent and fount a solution which is simple and clean for my needs:
# -*- coding: utf8 -*-
from flask import Flask
from flask.ext.socketio import SocketIO
import gevent
import os
def background_stuff():
while True:
try:
print 'doing background work ... '
gevent.sleep(1)
except Exception as e:
return e
app = Flask(__name__)
app.debug = True
socketio = SocketIO(app)
#app.route('/', methods=['GET'])
def get_timers():
return 'timer'
if __name__=='__main__':
if os.environ.get('WERKZEUG_RUN_MAIN') == 'true':
gevent.spawn(background_stuff)
socketio.run(app)
The tutorial can be found here: http://sdiehl.github.io/gevent-tutorial/#long-polling
It even tells about problems with gevent and multiprocesing: http://sdiehl.github.io/gevent-tutorial/#subprocess , but because I found a simple solution fitting to my needs I didn't try anything else.