I would like to learn how to call a function asynchronously in Python3. I think Tornado can do this. Currently, my code is returning nothing on the command line:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
async def count(end):
"""Print message when start equals end."""
start = 0
while True:
if start == end:
print('start = {0}, end = {1}'.format(start, end))
break
start = start + 1
def main():
# Start counting.
yield count(1000000000)
# This should print while count is running.
print('Count is running. Async!')
if __name__ == '__main__':
main()
Thanks
To call an async function, you need to provide an event loop to handle it. If you have a Tornado app, it provides such a loop, which allows you to make your handlers asynchronous:
from tornado.web import RequestHandler, url
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
async def do_something_asynchronous():
# e.g. call another service, read from database etc
return {'something': 'something'}
class YourAsyncHandler(RequestHandler):
async def get(self):
payload = await do_something_asynchronous()
self.write(payload)
application = web.Application([
url(r'/your_url', YourAsyncHandler, name='your_url')
])
http_server = HTTPServer(application)
http_server.listen(8000, address='0.0.0.0')
IOLoop.instance().start()
Outside of a Tornado app you can get the event loop from any number of providers, including the built-in asyncio library:
import asyncio
event_loop = asyncio.get_event_loop()
try:
event_loop.run_until_complete(do_something_asynchronous())
finally:
event_loop.close()
Related
The cryptofeed is a python library that uses asyncio library to get real-time prices of different Crypto Exchanges.
In this short program, we try to run the cryptofeed FeedHandler in an independent thread. An example of the code is shown below:
import functools as fct
from cryptofeed import FeedHandler
from cryptofeed.defines import BID, ASK, L2_BOOK
from cryptofeed.exchanges import Kraken
from datetime import datetime
import threading
async def bookfunc(params , orderbooks, feed, symbol, book, timestamp, receipt_timestamp):
print(f'Timestamp: {timestamp} Cryptofeed Receipt: {receipt_timestamp} Feed: {feed} Symbol: {symbol}'
f' Book Bid Size is {len(book[BID])} Ask Size is {len(book[ASK])}')
orderbooks = filter_orderbook(orderbooks, book, symbol, params['orderbook']['depth'])
def func():
# Parameters
params = {'orderbook': {'depth': 2}, 'price_model':{}, 'trade_model': {}}
config = {'log': {'filename': 'logs/demo.log', 'level': 'INFO'}}
orderbooks = {}
f = FeedHandler(config=config)
f.add_feed(Kraken(checksum_validation=True, subscription={L2_BOOK: ['BTC-USD', 'ETH-USD', 'LINK-USD', 'LTC-USD', 'ADA-USD']},
callbacks={L2_BOOK: fct.partial(bookfunc, params, orderbooks)})) # This way passes the orderbooks inside the callback
f.run()
if __name__ == '__main__':
thread = threading.Thread(target=func, args=())
thread.start()
When the code is executed, the get the following error:
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-1'.
Any idea how to solve this problem?
EDIT:
This is the solution for different questions in stackoverflow. An example is the following Question:
Running cryptofeed (asyncio library) in a new thread
(As of cryptofeed==1.9.2) there are two important things to make it run from a thread:
FeedHandler.run sets up signal handlers by default, which must
be done from the main thread. To avoid that, there's
install_signal_handlers argument on the method.
Initialiser of FeedHandler set's uvloop's policy, but
doesn't call uvloop.install() (assuming it's the application's
responsibility I guess). Without it asyncio.set_event_loop
doesn't have effect. Alternatively you can set 'uvloop': False
in the feed handler config (as shown below), or just uninstall
uvloop.
import asyncio
import threading
from cryptofeed import FeedHandler
from cryptofeed.defines import BID, ASK, L2_BOOK
from cryptofeed.exchanges import Kraken
async def bookfunc(**kwargs):
print('bookfunc', kwargs)
def run_feed_handler_forever():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
config = {
'uvloop': False,
'log': {'filename': 'log.log', 'level': 'DEBUG'},
}
l2_book = ['BTC-USD', 'ETH-USD', 'LINK-USD', 'LTC-USD', 'ADA-USD']
feed = Kraken(
subscription={L2_BOOK: l2_book}, callbacks={L2_BOOK: bookfunc}
)
fh = FeedHandler(config)
fh.add_feed(feed)
fh.run(install_signal_handlers=False)
if __name__ == '__main__':
thread = threading.Thread(target=run_feed_handler_forever)
thread.start()
thread.join()
You have to manually set an event loop for your thread through asyncio. You can do it by wrapping your function in a sort of loop setter:
def create_loop(func):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(func())
if __name__ == "__main__":
thread = threading.Thread(target=create_loop, args=(func,))
thread.start()
I'm trying to build a web server to collect "commands" via AJAX and then distribute the commands to clients via long-polling.
The goal is that someone POSTs some data to /add-command.
Another client implements a long-polling client hitting /poll waiting for a command to execute.
I think a queue is the right data structure to use to hold commands waiting for attention. I'd like the commands to essentially be distributed immediately to any long-polling client but held if no client is currently polling.
Here's my python script.
import os
import time
import tornado.httpserver
import tornado.ioloop
import tornado.web
import tornado.gen
import Queue
import multiprocessing.pool
import mysql.connector
import urlparse
import uuid
import json
_commandQueue = Queue.Queue()
_commandPollInterval = 0.2
_commandPollTimeout = 10
class HomeHandler(tornado.web.RequestHandler):
def get(self):
self.render("home.htm")
class AddCommandHandler(tornado.web.RequestHandler):
def post(self):
d = urlparse.parse_qs(self.request.body)
_commandQueue.put(d)
self.write(str(True))
class PollHandler(tornado.web.RequestHandler):
#tornado.gen.coroutine
def get(self):
self.write("start")
d = 1
d = yield self.getCommand()
self.write(str(d))
self.write("end")
self.finish()
#tornado.gen.coroutine
def getCommand(self):
start = time.time()
while (time.time() - start) < _commandPollTimeout * 1000:
if not _commandQueue.empty:
return _commandQueue.get()
else:
time.sleep(_commandPollInterval)
return None
def main():
application = tornado.web.Application(
[
(r"/", HomeHandler),
(r"/add-command", AddCommandHandler),
(r"/poll", PollHandler),
],
debug=True,
template_path=os.path.join(os.path.dirname(__file__), "templates"),
static_path=os.path.join(os.path.dirname(__file__), "static"),
)
tornado.httpserver.HTTPServer(application).listen(int(os.environ.get("PORT", 5000)))
tornado.ioloop.IOLoop.instance().start()
if __name__ == "__main__":
main()
The AddCommandHandler works fine to put items in the _commandQueue.
The PollHandler request just times out. If I call the PollHandler, it seems to lock the _commandQueue and I can't put or get from it.
I suspect I need to join the queue, but I can't seem to find the right time to do that in the code.
UPDATE -- Here's my final code thanks to the answers
import os
import time
import datetime
import tornado.httpserver
import tornado.ioloop
import tornado.web
import tornado.gen
import tornado.queues
import urlparse
import json
_commandQueue = tornado.queues.Queue()
_commandPollInterval = 0.2
_commandPollTimeout = 10
class HomeHandler(tornado.web.RequestHandler):
def get(self):
self.render("home.htm")
class AddCommandHandler(tornado.web.RequestHandler):
def get(self):
cmd = urlparse.parse_qs(self.request.body)
_commandQueue.put(cmd)
self.write(str(cmd))
def post(self):
cmd = urlparse.parse_qs(self.request.body)
_commandQueue.put(cmd)
self.write(str(cmd))
class PollHandler(tornado.web.RequestHandler):
#tornado.gen.coroutine
def get(self):
cmd = yield self.getCommand()
self.write(str(cmd))
#tornado.gen.coroutine
def getCommand(self):
try:
cmd = yield _commandQueue.get(
timeout=datetime.timedelta(seconds=_commandPollTimeout)
)
raise tornado.gen.Return(cmd)
except tornado.gen.TimeoutError:
raise tornado.gen.Return()
def main():
application = tornado.web.Application(
[
(r"/", HomeHandler),
(r"/add-command", AddCommandHandler),
(r"/poll", PollHandler),
],
debug=True,
template_path=os.path.join(os.path.dirname(__file__), "templates"),
static_path=os.path.join(os.path.dirname(__file__), "static"),
)
tornado.httpserver.HTTPServer(application).listen(int(os.environ.get("PORT", 5000)))
tornado.ioloop.IOLoop.instance().start()
if __name__ == "__main__":
main()
In async model you should omit blocking operation, time.sleep is evil in your code. Moreover, I think that the best way is to use tornado's (in async interface) queue - tornado.queue.Queue and use async get:
import datetime
import tornado.gen
import tornado.queues
_commandQueue = tornado.queues.Queue()
# ...rest of the code ...
#tornado.gen.coroutine
def getCommand(self):
try:
# wait for queue item if cannot obtain in timeout raise exception
cmd = yield _commandQueue.get(
timeout=datetime.timedelta(seconds=_commandPollTimeout)
)
return cmd
except tornado.gen.Timeout:
return None
Note: Module tornado.queues si available since Tornado 4.x, if you use older one, Toro will help.
You can NOT use sleep in listener, since it blocks reading from input stream. time.sleep(_commandPollInterval). What you should use is yield gen.sleep(_commandPollInterval)
python-running-autobahnpython-asyncio-websocket-server-in-a-separate-subproce
can-an-asyncio-event-loop-run-in-the-background-without-suspending-the-python-in
Was trying to solve my issue with this two links above but i have not.
I have the following error : RuntimeError: There is no current event loop in thread 'Thread-1'.
Here the code sample (python 3):
from autobahn.asyncio.wamp import ApplicationSession
from autobahn.asyncio.wamp import ApplicationRunner
from asyncio import coroutine
import time
import threading
class PoloniexWebsocket(ApplicationSession):
def onConnect(self):
self.join(self.config.realm)
#coroutine
def onJoin(self, details):
def on_ticker(*args):
print(args)
try:
yield from self.subscribe(on_ticker, 'ticker')
except Exception as e:
print("Could not subscribe to topic:", e)
def poloniex_worker():
runner = ApplicationRunner("wss://api.poloniex.com:443", "realm1")
runner.run(PoloniexWebsocket)
def other_worker():
while True:
print('Thank you')
time.sleep(2)
if __name__ == "__main__":
polo_worker = threading.Thread(None, poloniex_worker, None, (), {})
thank_worker = threading.Thread(None, other_worker, None, (), {})
polo_worker.start()
thank_worker.start()
polo_worker.join()
thank_worker.join()
So, my final goal is to have 2 threads launched at the start. Only one need to use ApplicationSession and ApplicationRunner. Thank you.
A separate thread must have it's own event loop. So if poloniex_worker needs to listen to a websocket, it needs its own event loop:
def poloniex_worker():
asyncio.set_event_loop(asyncio.new_event_loop())
runner = ApplicationRunner("wss://api.poloniex.com:443", "realm1")
runner.run(PoloniexWebsocket)
But if you're on a Unix machine, you will face another error if you try to do this. Autobahn asyncio uses Unix signals, but those Unix signals only work in the main thread. You can simply turn off Unix signals if you don't plan on using them. To do that, you have to go to the file where ApplicationRunner is defined. That is wamp.py in python3.5 > site-packages > autobahn > asyncio on my machine. You can comment out the signal handling section of the code like so:
# try:
# loop.add_signal_handler(signal.SIGTERM, loop.stop)
# except NotImplementedError:
# # signals are not available on Windows
# pass
All this is a lot of work. If you don't absolutely need to run your ApplicationSession in a separate thread from the main thread, it's better to just run the ApplicationSession in the main thread.
I'm trying to run a Tornado server with Couchbase 4.0 Developer preview.
import tornado.web
import tornado.httpserver
import tornado.options
import tornado.ioloop
import tornado.websocket
import tornado.httpclient
from tornado import gen
import os.path
from tornado.options import define, options, parse_command_line
import time
#from couchbase.bucket import Bucket
from twisted.internet import reactor
from txcouchbase.bucket import Bucket
from couchbase.n1ql import N1QLQuery, N1QLError
from pprint import pprint
server = "x.x.x.x"
bucketname = "zips"
Connection = "couchbase://" + server + "/" + bucketname
bkt = Bucket(Connection)
class IndexHandler(tornado.web.RequestHandler):
#tornado.web.asynchronous
def get(self):
print "entered"
query = "SELECT * FROM `zips` where pincode= '632014'"
q = N1QLQuery(query)
#self.bkt = bkt
t0 = time.time()
res = bkt.n1qlQueryAll(q)
res.addCallback(self.on_ok)
reactor.run()
t1 = time.time()
print t1-t0
self.write("Hello World")
def on_ok(self,response):
print "LOl"
for each in res:
print each
reactor.stop()
self.finish()
handlers = [
(r'/',IndexHandler),
]
if __name__ == "__main__":
parse_command_line()
# template path should be given here only unlike handlers
app = tornado.web.Application(handlers, template_path=os.path.join(os.path.dirname(__file__), "templates"),
static_path=os.path.join(os.path.dirname(__file__), "static"), cookie_secret="61oETzKXQAGaYdkL5gEmGeJJFuYh7EQnp2XdTP1o/Vo=", debug=True)
http_server = tornado.httpserver.HTTPServer(app)
http_server.listen(8888, address='0.0.0.0')
tornado.ioloop.IOLoop.instance().start()
After I run this, for some reason the callback function is never called. I could not find any proper documentation for this, and had to go through the source code to write this. I'm still confused as I'm new to asynchronous programming. Can someone please tell me where I'm going wrong and if there is a better way of doing this?
In asynchronous programming, you only want to start an event loop (like IOLoop.start() or reactor.run()) once, at the top of your program. You're calling IOLoop.start(), so instead of calling reactor.run() you want to tell Twisted to use the Tornado IOLoop as its reactor. Before the import of reactor, do
import tornado.platform.twisted
tornado.platform.twisted.install()
from twisted.internet import reactor
See http://www.tornadoweb.org/en/stable/twisted.html#twisted-on-tornado for more.
Once you've done this, you can call twisted libraries without having to start and stop the reactor.
I want to stream a long database result set through Tornado.
I obviously need a server cursor since its not feasible to load the whole query in memory.
So I have the following code:
class QueryStreamer(RequestHandler):
def get(self):
cursor.execute("Select * from ...")
chunk = cursor.fetch(1000)
while chunk:
self.write(chunk)
self.flush()
chunk = cursor.fetch(1000)
self.finish()
cursor.close()
If someone does not read my request till the end? (i.e. curl ... |head),
The get method keeps happily streaming my data to nowhere. I would expect to get SIGPIPE at some point and close database cursor (without running it to the end for nothing).
How can I catch write errors in Tornado?
Update: Following suggestion in the answer I've tried the following:
import tornado.ioloop
import tornado.web
import time
class PingHandler(tornado.web.RequestHandler):
def get(self):
for i in range(600):
self.write("pong\n")
self.flush()
time.sleep(1)
print "pong"
self.finish()
print "ponged"
def on_connection_close(self):
print "closed"
if __name__ == "__main__":
application = tornado.web.Application([ ("/ping", PingHandler), ])
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()
I'm running this file in terminal 1 and in terminal 2 I invoke:
curl -s http://localhost:8888/ping
and after first response I hit CTRL-C. But in terminal 1 I see that it happily keeps "pong"-ing and on_connection_close never gets called.
Bottom line - still does not work.
You need to make the handler asynchronous and use ioloop.add_timeout instead of time.sleep, because that blocks the loop:
import tornado.ioloop
import tornado.web
import tornado.gen
class PingHandler(tornado.web.RequestHandler):
connection_closed = False
def on_connection_close(self):
print "closed"
self.connection_closed = True
#tornado.gen.coroutine # <= async handler
def get(self):
for i in range(600):
if self.connection_closed:
# `on_connection_close()` has been called,
# break out of the loop
break
self.write("pong %s\n" % i)
self.flush()
# Add a timeout. Similar to time.sleep(1), but non-blocking:
yield tornado.gen.Task(
tornado.ioloop.IOLoop.instance().add_timeout,
tornado.ioloop.IOLoop.instance().time() + 1,
)
self.finish()
print "finished"
if __name__ == "__main__":
application = tornado.web.Application([("/ping", PingHandler), ])
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()
Implement the on_connection_close method and have it stop that write loop in your get handler.