In Tornado's official documentation (https://www.tornadoweb.org/en/stable/guide/async.html) there is an example of synchronous fetch, but when I put it in get method of my IndexHandler it returns following error:
File "/home/stefan/.local/lib/python3.6/site-packages/tornado/web.py", line 1697, in _execute
result = method(*self.path_args, **self.path_kwargs)
File "ex1.py", line 19, in get
client = tornado.httpclient.HTTPClient()
File "/home/stefan/.local/lib/python3.6/site-packages/tornado/httpclient.py", line 107, in __init__
self._async_client = self._io_loop.run_sync(make_client)
File "/home/stefan/.local/lib/python3.6/site-packages/tornado/ioloop.py", line 526, in run_sync
self.start()
File "/home/stefan/.local/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 148, in start
self.asyncio_loop.run_forever()
File "/usr/lib/python3.6/asyncio/base_events.py", line 428, in run_forever
'Cannot run the event loop while another loop is running')
RuntimeError: Cannot run the event loop while another loop is running
My code looks like:
import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
import tornado.httpclient
from tornado.options import define, options
define("port", default=8000, help="run on the given port", type=int)
class IndexHandler(tornado.web.RequestHandler):
def get(self):
client = tornado.httpclient.HTTPClient()
response = client.fetch('https://www.google.com')
print(response.body)
if __name__ == "__main__":
tornado.options.parse_command_line()
app=tornado.web.Application(handlers=[(r"/", IndexHandler)],debug=True)
http_server = tornado.httpserver.HTTPServer(app)
http_server.listen(options.port)
tornado.ioloop.IOLoop.instance().start()
I believe I should make changes in main in order to make it work.
Because as you can see in your traceback, a "synchronous" fetch starts it's own ioloop under the hood. The easiest to fix is to rewrite your def get as a coroutine (either as async def get or with a #gen.coroutine decorator) and use asynchronous fetch
class IndexHandler(tornado.web.RequestHandler):
async def get(self):
client = tornado.httpclient.HTTPClient()
response = await client.fetch('https://www.google.com')
print(response.body)
Related
I am trying to make an API call from within my service and I am running into event loop issues. Can someone help me understand what I am doing wrong?
Basically I want to make a service that does some calculations based on data pulled from a different service.
I can call this code below from a cli, but not when I start up a web app (i.e.) hitting http://127.0.0.1:8080/add
loop = asyncio.get_event_loop()
data = loop.run_until_complete(run_fetch(loop, 'http://google.com'))
Sample code:
from aiohttp import web
import aiohttp
import asyncio
async def add(request):
loop = asyncio.get_event_loop()
data = loop.run_until_complete(run_fetch(loop, 'http://google.com'))
return web.json_response(data)
async def fetch(client, url):
async with client.get(url) as resp:
assert resp.status == 200
return await resp.text()
async def run_fetch(loop, url):
async with aiohttp.ClientSession(loop=loop) as client:
html = await fetch(client, url)
return html
app = web.Application()
app.router.add_get('/add', add)
web.run_app(app, host='127.0.0.1', port=8080)
Exception:
Error handling request
Traceback (most recent call last):
File ".../aiohttp/web_protocol.py", line 417, in start
resp = yield from self._request_handler(request)
File ".../aiohttp/web.py", line 289, in _handle
resp = yield from handler(request)
File ".../sample.py", line 11, in add data = loop.run_until_complete(run_fetch(loop, 'http://google.com'))
File ".../python3.6/asyncio/base_events.py", line 454, in run_until_complete
self.run_forever()
File ".../python3.6/asyncio/base_events.py", line 408, in run_forever
raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
The run_until_complete is the way to run some async code from the sync context. Under the hood it adds provided future to the given ioloop and calls run_forever then returns result or throws exception (resolved future).
Actually you need to await run_fetch(loop, 'http://google.com'), since the caller function is asynchronous.
I'm attempting to use asyncio to make an asynchronous client/server setup.
For some reason I'm getting AssertionError: yield from wasn't used with future when running the client.
Searching for this error didn't turn up much.
What does this error mean and what's causing it?
#!/usr/bin/env python3
import asyncio
import pickle
import uuid
port = 9999
class ClientProtocol(asyncio.Protocol):
def __init__(self, loop):
self.loop = loop
self.conn = None
self.uuid = uuid.uuid4()
self.other_clients = []
def connection_made(self, transport):
print("Connected to server")
self.conn = transport
m = "hello"
self.conn.write(m)
def data_received(self, data):
print('Data received: {!r}'.format(data))
def connection_lost(self, exc):
print('The server closed the connection')
print('Stop the event loop')
self.loop.stop()
# note that in my use-case, main() is called continuously by an external game engine
client_init = False
def main():
# use a global here only for the purpose of providing example code runnable outside of aforementioned game engine
global client_init
if client_init != True:
loop = asyncio.get_event_loop()
coro = loop.create_connection(lambda: ClientProtocol(loop), '127.0.0.1', port)
task = asyncio.Task(coro)
transport, protocol = loop.run_until_complete(coro)
client_init = True
# to avoid blocking the execution of main (and of game engine calling it), only run one iteration of the event loop
loop.stop()
loop.run_forever()
if transport:
transport.write("some data")
if __name__ == "__main__":
main()
Traceback:
Traceback (most recent call last):
File "TCPclient.py", line 57, in <module>
main()
File "TCPclient.py", line 45, in main
transport, protocol = loop.run_until_complete(coro)
File "/usr/lib/python3.5/asyncio/base_events.py", line 337, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/usr/lib/python3.5/asyncio/base_events.py", line 599, in create_connection
yield from tasks.wait(fs, loop=self)
File "/usr/lib/python3.5/asyncio/tasks.py", line 341, in wait
return (yield from _wait(fs, timeout, return_when, loop))
File "/usr/lib/python3.5/asyncio/tasks.py", line 424, in _wait
yield from waiter
File "/usr/lib/python3.5/asyncio/futures.py", line 359, in __iter__
assert self.done(), "yield from wasn't used with future"
AssertionError: yield from wasn't used with future
The problem seems to be that you create a task from your coroutine, but then pass the coroutine to run_until_complete instead:
coro = loop.create_connection(lambda: ClientProtocol(loop), '127.0.0.1', port)
task = asyncio.Task(coro)
transport, protocol = loop.run_until_complete(coro)
Either pass the task:
coro = loop.create_connection(lambda: ClientProtocol(loop), '127.0.0.1', port)
task = asyncio.Task(coro)
transport, protocol = loop.run_until_complete(task)
Or don't create the task and pass the coroutine. run_until_complete will create a task for you
coro = loop.create_connection(lambda: ClientProtocol(loop), '127.0.0.1', port)
transport, protocol = loop.run_until_complete(coro)
In addition, you need to ensure the strings you are writing are byte strings. String literals in Python 3 default to unicode. You can either encode these, or just write byte strings in the first place
transport.write("some data".encode('utf-8'))
transport.write(b"some data")
EDIT It's not clear to me why this is an issue, however the source for run_until_complete has this to say:
WARNING: It would be disastrous to call run_until_complete()
with the same coroutine twice -- it would wrap it in two
different Tasks and that can't be good.
I suppose creating a task and then passing in the coroutine (which causes a task to be created) has the same effect.
I use tornado (4.2.1) + momoko (2.2.0) + psycopg2 (2.6.1) for small web application and it works ok until PostgreSQL server close connection. Then after every db.execute() command I receive an error message:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\tornado-4.2.1-py2.7-win32.egg\tornado\web.py", line 1415, in _execute
result = yield result
File "C:\Python27\lib\site-packages\tornado-4.2.1-py2.7-win32.egg\tornado\gen.py", line 870, in run
value = future.result()
File "C:\Python27\lib\site-packages\tornado-4.2.1-py2.7-win32.egg\tornado\concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "C:\Python27\lib\site-packages\tornado-4.2.1-py2.7-win32.egg\tornado\gen.py", line 876, in run
yielded = self.gen.throw(*exc_info)
File "server.py", line 63, in get
cursor = yield self.db.execute(query)
File "C:\Python27\lib\site-packages\tornado-4.2.1-py2.7-win32.egg\tornado\gen.py", line 870, in run
value = future.result()
File "C:\Python27\lib\site-packages\tornado-4.2.1-py2.7-win32.egg\tornado\concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "D:\work\program-stat\momoko\connection.py", line 453, in when_available
future_or_result = method(conn, *args, **kwargs)
File "D:\work\program-stat\momoko\connection.py", line 743, in execute
cursor.execute(operation, parameters)
File "C:\Python27\lib\site-packages\psycopg2\extras.py", line 288, in execute
return super(NamedTupleCursor, self).execute(query, vars)
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Here is a code:
import os
import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
import momoko
from tornadotools.route import Route
from psycopg2.extras import NamedTupleCursor
import environments as env
tornado.options.define("port", default=9999, help="run on the given port", type=int)
tornado.options.define("pgsql_host", default=env.DB_HOST, help="database host")
tornado.options.define("pgsql_database", default=env.DB_DATABASE, help="database name")
tornado.options.define("pgsql_user", default=env.DB_LOGIN, help="database user")
tornado.options.define("pgsql_password", default=env.DB_PASSWORD, help="database password")
class Application(tornado.web.Application):
def __init__(self):
handlers = Route.routes()
settings = dict(
template_path=os.path.join(os.path.dirname(__file__), "templates"),
static_path=os.path.join(os.path.dirname(__file__), "static"),
debug=True,
)
tornado.web.Application.__init__(self, handlers, **settings)
# Have one global connection to DB across all handlers
self.db = momoko.Pool(
dsn='dbname=%s user=%s password=%s '
'host=%s port=5432' % (
tornado.options.options.pgsql_database,
tornado.options.options.pgsql_user,
tornado.options.options.pgsql_password,
tornado.options.options.pgsql_host),
cursor_factory=NamedTupleCursor,
)
self.db.connect()
class BaseHandler(tornado.web.RequestHandler):
#property
def db(self):
return self.application.db
#Route(r"/")
class HomeHandler(BaseHandler):
def get(self):
self.write("<ul>")
self.write("<li><a href='/test'>Test page</a>")
self.write("</ul>")
self.finish()
#Route(r"/test")
class MogrifyHandler(BaseHandler):
#tornado.web.asynchronous
#tornado.gen.coroutine
def get(self):
cursor = yield self.db.execute("SELECT %s as t;", (1,))
self.render("test.html", result=cursor.fetchall())
def main():
tornado.options.parse_command_line()
http_server = tornado.httpserver.HTTPServer(Application())
http_server.listen(tornado.options.options.port)
tornado.ioloop.IOLoop.instance().start()
if __name__ == "__main__":
main()
How could I handle this exception and reconnect to db automatically without restarting app?
Tornado/Momoko/AutoReload
Problem:
If the connection to the database Postgres distructed - momoko can not re-create the connection pool.
To solve this problem you need to restart the application. To automate this process, I wrote a small script that use the built-in tornado update mechanism.
Solution:
Test every 1 second that database connection is alive.
Add this check before you start ioloop.
import reload
check_connect = tornado.ioloop.PeriodicCallback(lambda: reload.reload_if_db_pool_is_dead(application.db), 1000)
check_connect.start()
https://gist.github.com/MaxRV/d7180e63d6b5396d904d446653aa7e88
you can add parameter:
raise_connect_errors=False,
When you create the connection pool:
self.db = momoko.Pool(
dsn='dbname=%s user=%s password=%s '
'host=%s port=5432' % (
tornado.options.options.pgsql_database,
tornado.options.options.pgsql_user,
tornado.options.options.pgsql_password,
tornado.options.options.pgsql_host),
cursor_factory=NamedTupleCursor,
raise_connect_errors=False,
)
But it is not always helpful.
Consider the following Tornado (v 4.0.2) application, which is a little bit modified version of official hello world example:
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.set_status(400)
self.write("Hello, world")
application = tornado.web.Application([
(r"/", MainHandler),
])
if __name__ == "__main__":
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()
As you can see, the only difference here is set_status call in MainHandler. Now, I save this code into app.py, then I open tests.py and I put there this simple unit test:
import tornado.ioloop
from tornado.httpclient import HTTPRequest
from tornado.testing import AsyncHTTPTestCase, gen_test
from app import application
class SimpleTest(AsyncHTTPTestCase):
def get_app(self):
return application
def get_new_ioloop(self):
return tornado.ioloop.IOLoop.instance()
#gen_test
def test_bad_request(self):
request = HTTPRequest(url=self.get_url('/'))
response = yield self.http_client.fetch(request)
self.assertEqual(response.code, 400)
When I run this test with python -m tornado.test.runtests tests I get the following result:
E
======================================================================
ERROR: test_bad_request (tests.SimpleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/testing.py", line 118, in __call__
result = self.orig_method(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tornado/testing.py", line 494, in post_coroutine
timeout=timeout)
File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 418, in run_sync
return future_cell[0].result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 109, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 631, in run
yielded = self.gen.throw(*sys.exc_info())
File "tests.py", line 18, in test_bad_request
response = yield self.http_client.fetch(request)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 628, in run
value = future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 111, in result
raise self._exception
HTTPError: HTTP 400: Bad Request
----------------------------------------------------------------------
Ran 1 test in 0.022s
FAILED (errors=1)
[E 140929 12:55:59 testing:687] FAIL
Obviously this is correct, because the handler sets 400 status code. But how can I test my application for such case? I think 4xx codes are useful, so I don't want to give them up. However I'm new to Tornado and I wasn't able to find a way to test them. Is there any?
Try this:
#gen_test
def test_bad_request(self):
request = HTTPRequest(url=self.get_url('/'))
with self.assertRaises(tornado.httpclient.HTTPError) as context:
yield self.http_client.fetch(request)
self.assertEqual(context.exception.code, 400)
See the documentation for assertRaises.
As part of a project, I've been trying to port a Tornado server to work on the Google App Engine. Since the App Engine doesn't implement the asynchronous functions of normal Tornado, I've been trying to convert the main Application to a WSGIApplication. The normal main code works fine (forgive the imports and formatting, it's a mess from trying to follow other examples):
import wsgiref
import tornado.wsgi
import tornado.web
import tornado.httpserver
import os
import Handlers
from tornado.options import define, options
define("port", default=8000, help="run on the given port", type=int)
def application():
handlers=[(r"/", Handlers.MainHandler),
(r"/Login", Handlers.LoginHandler),
(r"/ViewHistory",Handlers.ViewHistoryHandler),
(r"/UploadFile", Handlers.UploadHandler),
(r"/Index", Handlers.IndexHandler),
(r"/About", Handlers.AboutHandler),
(r"/Profile", Handlers.ProfileHandler)]
settings=dict(template_path=os.path.join(os.path.dirname(__file__), "templates"),
static_path=os.path.join(os.path.dirname(__file__), "static"),
debug=True)
return tornado.web.Application(handlers, **settings)
def main():
tornado.options.parse_command_line()
http_server = tornado.httpserver.HTTPServer(application())
http_server.listen(options.port)
tornado.ioloop.IOLoop.instance().start()
if __name__ == "__main__":
main()
I can access the web pages, browse the site, works great. If I change line 24 to return a tornado.wsgi.WSGIApplication, like thus:
import wsgiref
import tornado.wsgi
import tornado.web
import tornado.httpserver
import os
import Handlers
from tornado.options import define, options
define("port", default=8000, help="run on the given port", type=int)
def application():
handlers=[(r"/", Handlers.MainHandler),
(r"/Login", Handlers.LoginHandler),
(r"/ViewHistory",Handlers.ViewHistoryHandler),
(r"/UploadFile", Handlers.UploadHandler),
(r"/Index", Handlers.IndexHandler),
(r"/About", Handlers.AboutHandler),
(r"/Profile", Handlers.ProfileHandler)]
settings=dict(template_path=os.path.join(os.path.dirname(__file__), "templates"),
static_path=os.path.join(os.path.dirname(__file__), "static"),
debug=True)
return tornado.wsgi.WSGIApplication(handlers, **settings)
def main():
tornado.options.parse_command_line()
http_server = tornado.httpserver.HTTPServer(application())
http_server.listen(options.port)
tornado.ioloop.IOLoop.instance().start()
if __name__ == "__main__":
main()
it also runs just fine. However, when I try to access any of the web pages, it gives me the following error:
[E 140131 10:02:18 iostream:357] Uncaught exception, closing connection.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado-3.2-py2.7-linux-x86_64.egg/tornado/iostream.py", line 354, in wrapper
callback(*args)
File "/usr/local/lib/python2.7/dist-packages/tornado-3.2-py2.7-linux-x86_64.egg/tornado/stack_context.py", line 331, in wrapped
raise_exc_info(exc)
File "/usr/local/lib/python2.7/dist-packages/tornado-3.2-py2.7-linux-x86_64.egg/tornado/stack_context.py", line 302, in wrapped
ret = fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tornado-3.2-py2.7-linux-x86_64.egg/tornado/httpserver.py", line 328, in _on_headers
self.request_callback(self._request)
TypeError: __call__() takes exactly 3 arguments (2 given)
[E 140131 10:02:18 ioloop:491] Exception in callback <functools.partial object at 0xf6e3ecfc>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado-3.2-py2.7-linux-x86_64.egg/tornado/ioloop.py", line 477, in _run_callback
callback()
File "/usr/local/lib/python2.7/dist-packages/tornado-3.2-py2.7-linux-x86_64.egg/tornado/stack_context.py", line 331, in wrapped
raise_exc_info(exc)
File "/usr/local/lib/python2.7/dist-packages/tornado-3.2-py2.7-linux-x86_64.egg/tornado/stack_context.py", line 302, in wrapped
ret = fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tornado-3.2-py2.7-linux-x86_64.egg/tornado/iostream.py", line 354, in wrapper
callback(*args)
File "/usr/local/lib/python2.7/dist-packages/tornado-3.2-py2.7-linux-x86_64.egg/tornado/stack_context.py", line 331, in wrapped
raise_exc_info(exc)
File "/usr/local/lib/python2.7/dist-packages/tornado-3.2-py2.7-linux-x86_64.egg/tornado/stack_context.py", line 302, in wrapped
ret = fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tornado-3.2-py2.7-linux-x86_64.egg/tornado/httpserver.py", line 328, in _on_headers
self.request_callback(self._request)
TypeError: __call__() takes exactly 3 arguments (2 given)
Which I can't make heads or tails of, and Google hasn't turned up anyone with the same problem (except for the guy who recommended mucking around in the Tornado files, which won't help on the App Engine AFAIK). Has anyone else seen this error before, or can spot why the WSGIApplication crashes when the default Application does not?
The name "application" is unfortunately overloaded here - tornado.web.Application and WSGIApplication differ in their interface to the server. tornado.web.Application can be used by a tornado HTTPServer, but WSGIApplication must be run in a WSGI container. In app engine deployment, you'd just mention your WSGIApplication instance directly in the config file, with no mention of tornado.httpserver. To use adapt a WSGIApplication to tornado.httpserver, use a tornado.wsgi.WSGIContainer: HTTPServer(WSGIContainer(application()))