I'm currently using flask-restful (http://flask-restful.readthedocs.io/en/0.3.5/index.html) to deploy resources as endpoints and I'm wondering if there's a way to access the API logger from within then resources classes. I've skimmed through the docs and couldn't find the appropriate answer.
Basically I want to do that :
from flask_restful import Resource
class SomeEndpoint(Resource):
def get(self):
try:
... something throws an exception
except SomeException as se:
... send custom message to API logger <----- Here!
return response
What I though of doing was passing the logger from the API through the constructor of the Resource like that :
App = Flask(__name__)
api = Api(App)
api.add_resource(SomeEndpoint, '/', resource_class_kwargs={'logger': App.logger})
Is this the most appropriate way to access the logger inside flask-restful resource endpoints ?
Thanks a lot
I know the answer has been chosen already, but there is a slightly different approach that also works.
First, import
from flask import current_app as app
in the resource file, and when calling the logger, do:
app.logger.info("This is an info message")
You need to define constructor of Resource. Here an example:
import logging
class SomeEndpoint(Resource):
def __init__(self, **kwargs):
self.logger = kwargs.get('logger')
def get(self):
# self.logger - 'logger' from resource_class_kwargs
return self.logger.name
api.add_resource(SomeEndpoint, '/', resource_class_kwargs={
# any logger here...
'logger': logging.getLogger('my_custom_logger')
})
Open your endpoint. You will see my_custom_logger.
Hope this helps.
Setting debug=False temporarily fixes the problem.
But I dont really know what is the issue when debug is set to True
Related
I just came across rollbar and wanted to include it in my Python project.
This is the standard way in which I am told to implement rollbar from the website.
import rollbar
rollbar.init('KEY')
try:
a = s
except:
rollbar.report_exc_info()
Is there a better way to implement this without going through all my try except blocks and replacing them with rollbar.report_exc_info()
Can there be a decorator implementation for this?
My current project is a Flask app that provides API's to end users.
Here's an example for rollbar integration in Flask apps.
https://github.com/rollbar/rollbar-flask-example/blob/master/hello.py
#app.before_first_request
def init_rollbar():
"""init rollbar module"""
rollbar.init(
# access token for the demo app: https://rollbar.com/demo
'fc316ac1f7404dc28af26d5baed1416c',
# environment name
'flasktest',
# server root directory, makes tracebacks prettier
root=os.path.dirname(os.path.realpath(__file__)),
# flask already sets up logging
allow_logging_basic_config=False)
# send exceptions from `app` to rollbar, using flask's signal system.
got_request_exception.connect(rollbar.contrib.flask.report_exception, app)
I am looking for help with an aiohttp middleware I'm working on that will automatically add trailing slashes to a uri if it's missing. Aiohttp requires that when you define a url, you define two routes for each path in your list of routes, one with a trailing slash and one without. They have an example middleware that addresses this by finding if the uri does not end with a / and adding it if not, but then they use an http 302 redirect to tell the client to go to the new uri. Otherwise a relative uri like /endpoint will 404 while /endpoint/ will work.
This is their middleware:
https://github.com/aio-libs/aiohttp/blob/master/aiohttp/web_middlewares.py
Using their middleware as a basis, I'm trying to accomplish the same thing but without a redirect on the client side. I want the handling to only be on the server. My current attempt at this is shown below and seems to update the request in flight, but I still get a 404:
from aiohttp.web_urldispatcher import SystemRoute
def middleware(f):
f.__middleware_version__ = 1
return f
def trailing_slashes():
#middleware
async def impl(request, handler):
if isinstance(request.match_info.route, SystemRoute):
rel_url = str(request.rel_url)
if '?' in rel_url:
path, query = rel_url.split('?', 1)
query = f'?{query}'
else:
path = rel_url
query = ''
if not path.endswith('/'):
rel_url = f'{path}/{query}'
request = request.clone(rel_url=rel_url)
return await handler(request)
return impl
This is a class that implements the middleware and illustrates the problem.
from aiohttp import web
import slashes_mw
class ClassName(object):
def __init__(self):
self.app = web.Application(middlewares=[slashes_mw.trailing_slashes()])
self.app.add_routes(self.get_routes())
web.run_app(self.app, port=80, host='0.0.0.0')
def get_routes(self):
return [
web.get('/', self.handler),
web.get('/{name}/', self.handler)
]
def handler(self, request):
return web.Response(text='hello')
ClassName()
Any ideas? Thanks for your help.
I think the redirect approach that they propose is quite correct, as the routes are simply not the same. But you might try to just register the second "version" of the route as well:
def get_routes(self):
return [
web.get('/', self.handler),
web.get('/{name}/', self.handler),
web.get('/{name}', self.handler)
]
This should handle your problem server-side (NOT TESTED).
As I understand the framework, your approach does not work because the specific handler is selected BEFORE the different middlewares are called. If you call the handler function in the middleware, there is no "routing" lookup done anymore. Or at least that is how I understand their documentation: https://docs.aiohttp.org/en/stable/web_advanced.html#middlewares
Might be interesting as well: https://docs.aiohttp.org/en/stable/web_quickstart.html#variable-resources
I use tornado 4.5.2 with routing implementation.
My server have two versions of API, let them call base and fancy. So a client is able to use both of them:
GET /base/foo
GET /base/baz
GET /fancy/foo
GET /fancy/baz
However, some fancy handlers may not be implemented; In this case a base one should be used.
In example:
application = web.Application([
(r"/base/foo", handlers.BaseFooHandler, {"some": "settings"}),
(r"/base/baz", handlers.BaseBazHandler, {"some": "settings"}),
(r"/fancy/foo", handlers.FancyFooHandler, {"some": "settings"}),
])
when cilent requests GET /fancy/baz the BaseBazHandler should do the job.
How can I achieve that with tornado routing?
Since you're registering your routes using a decorator, you can create a custom router that will respond to all the unmatched/unregistered /fancy/.* routes. For this to work correctly, you'll have to register your router at the end.
That way your custom router will be matched only if there isn't already a /fancy/... route registered. So, that means the custom router class will need to do these things:
Check if a fallback BaseBazHandler exists or not.
If exists, forward the request to it.
Else, return a 404 error.
Before proceeding any further, you'll have to create a custom class to handle 404 requests. This is necessary because if not handler is found, then this is the easiest way to return a 404 error.
class Handle404(RequestHandler):
def get(self):
self.set_status(404)
self.write('404 Not Found')
Okay, now let's write the custom router:
from tornado.routing import Router
class MyRouter(Router):
def __init__(self, app):
self.app = app
def find_handler(self, request, **kwargs):
endpoint = request.path.split('/')[2] # last part of the path
fallback_handler = 'Base%sHandler' % endpoint.title()
# fallback_handler will look like this - 'BaseBazHandler'
# now check if the handler exists in the current file
try:
handler = globals()[fallback_handler]
except KeyError:
handler = Handle404
return self.app.get_handler_delegate(request, handler)
Finally, after you've added all other routes, you can register your custom router:
from tornado.routing import PathMatches
application.add_handlers(r'.*', # listen for all hosts
[
(PathMatches(r"/fancy/.*"), MyRouter(application)),
]
)
I should point out that MyRouter.find_handler, only check checks for handlers in the current module (file). Modify the code to search for handlers in different modules, if you want.
So I am trying to build a restful API using flask, served up by apache on centos (httpd).
Basic API calls work just fine but I am not making much progress on the more advanced aspects because every time it fails I just get an HTTP 500 response which is completely useless for troubleshooting and I have no server-side logs to look at. I am literally trying to solve this through trial and error and it is making me bang my head against the wall.
In order to make any progress on this project I need to setup some basic error logging, but I do not understand the documentation or existing threads about this. It is completely over my head.
What I want to do is have flask write out all warnings and exceptions generated by my application to a specific file (it can be in the app directory to keep it simple).
I am looking for the simplest, easiest, least mind bendy way of doing this... suggestions?
Here is a very simplified version of my app... it shows the basic structure I am using, so please use that for reference.
from flask import Flask, jsonify, request
from flask_restful import reqparse, abort, Resource, Api
app = Flask(__name__)
api = Api(app)
class fetchTicket(Resource):
def post(self):
request_data = request.get_json(force=True)
r_ticket_id = request_data['ticket_id']
return jsonify(ticket_id=r_ticket_id)
api.add_resource(fetchTicket, '/ticket/fetch')
if __name__ == "__main__":
import logging
from logging.handlers import FileHandler
app.debug = True
file_handler = FileHandler("/var/www/project_folder/error.log")
file_handler.setLevel(logging.DEBUG)
app.logger.addHandler(file_handler)
app.run()
But when I run the above code no error.log file is created. I am not sure what I am doing wrong.
Note: I did set the folder permissions so that the apache user has access to write to the directory where the log file should go, as per http://fideloper.com/user-group-permissions-chmod-apache but it did not help so I don't think it is a permissions issue.
You’ll need to explicitly include the stack trace when logging, using the extra kwarg.
logger.exception('Probably something went wrong', extra={'stack': True})
I'm needing some help setting up unittests for Google Cloud Endpoints. Using WebTest all requests answer with AppError: Bad response: 404 Not Found. I'm not really sure if endpoints is compatible with WebTest.
This is how the application is generated:
application = endpoints.api_server([TestEndpoint], restricted=False)
Then I use WebTest this way:
client = webtest.TestApp(application)
client.post('/_ah/api/test/v1/test', params)
Testing with curl works fine.
Should I write tests for endpoints different? What is the suggestion from GAE Endpoints team?
After much experimenting and looking at the SDK code I've come up with two ways to test endpoints within python:
1. Using webtest + testbed to test the SPI side
You are on the right track with webtest, but just need to make sure you correctly transform your requests for the SPI endpoint.
The Cloud Endpoints API front-end and the EndpointsDispatcher in dev_appserver transforms calls to /_ah/api/* into corresponding "backend" calls to /_ah/spi/*. The transformation seems to be:
All calls are application/json HTTP POSTs (even if the REST endpoint is something else).
The request parameters (path, query and JSON body) are all merged together into a single JSON body message.
The "backend" endpoint uses the actual python class and method names in the URL, e.g. POST /_ah/spi/TestEndpoint.insert_message will call TestEndpoint.insert_message() in your code.
The JSON response is only reformatted before being returned to the original client.
This means you can test the endpoint with the following setup:
from google.appengine.ext import testbed
import webtest
# ...
def setUp(self):
tb = testbed.Testbed()
tb.setup_env(current_version_id='testbed.version') #needed because endpoints expects a . in this value
tb.activate()
tb.init_all_stubs()
self.testbed = tb
def tearDown(self):
self.testbed.deactivate()
def test_endpoint_insert(self):
app = endpoints.api_server([TestEndpoint], restricted=False)
testapp = webtest.TestApp(app)
msg = {...} # a dict representing the message object expected by insert
# To be serialised to JSON by webtest
resp = testapp.post_json('/_ah/spi/TestEndpoint.insert', msg)
self.assertEqual(resp.json, {'expected': 'json response msg as dict'})
The thing here is you can easily setup appropriate fixtures in the datastore or other GAE services prior to calling the endpoint, thus you can more fully assert the expected side effects of the call.
2. Starting the development server for full integration test
You can start the dev server within the same python environment using something like the following:
import sys
import os
import dev_appserver
sys.path[1:1] = dev_appserver._DEVAPPSERVER2_PATHS
from google.appengine.tools.devappserver2 import devappserver2
from google.appengine.tools.devappserver2 import python_runtime
# ...
def setUp(self):
APP_CONFIGS = ['/path/to/app.yaml']
python_runtime._RUNTIME_ARGS = [
sys.executable,
os.path.join(os.path.dirname(dev_appserver.__file__),
'_python_runtime.py')
]
options = devappserver2.PARSER.parse_args([
'--admin_port', '0',
'--port', '8123',
'--datastore_path', ':memory:',
'--logs_path', ':memory:',
'--skip_sdk_update_check',
'--',
] + APP_CONFIGS)
server = devappserver2.DevelopmentServer()
server.start(options)
self.server = server
def tearDown(self):
self.server.stop()
Now you need to issue actual HTTP requests to localhost:8123 to run tests against the API, but again can interact with GAE APIs to set up fixtures, etc. This is obviously slow as you're creating and destroying a new dev server for every test run.
At this point I use the Google API Python client to consume the API instead of building the HTTP requests myself:
import apiclient.discovery
# ...
def test_something(self):
apiurl = 'http://%s/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest' \
% self.server.module_to_address('default')
service = apiclient.discovery.build('testendpoint', 'v1', apiurl)
res = service.testresource().insert({... message ... }).execute()
self.assertEquals(res, { ... expected reponse as dict ... })
This is an improvement over testing with CURL as it gives you direct access to the GAE APIs to easily set up fixtures and inspect internal state. I suspect there is an even better way to do integration testing that bypasses HTTP by stitching together the minimal components in the dev server that implement the endpoint dispatch mechanism, but that requires more research time than I have right now.
webtest can be simplified to reduce naming bugs
for the following TestApi
import endpoints
import protorpc
import logging
class ResponseMessageClass(protorpc.messages.Message):
message = protorpc.messages.StringField(1)
class RequestMessageClass(protorpc.messages.Message):
message = protorpc.messages.StringField(1)
#endpoints.api(name='testApi',version='v1',
description='Test API',
allowed_client_ids=[endpoints.API_EXPLORER_CLIENT_ID])
class TestApi(protorpc.remote.Service):
#endpoints.method(RequestMessageClass,
ResponseMessageClass,
name='test',
path='test',
http_method='POST')
def test(self, request):
logging.info(request.message)
return ResponseMessageClass(message="response message")
the tests.py should look like this
import webtest
import logging
import unittest
from google.appengine.ext import testbed
from protorpc.remote import protojson
import endpoints
from api.test_api import TestApi, RequestMessageClass, ResponseMessageClass
class AppTest(unittest.TestCase):
def setUp(self):
logging.getLogger().setLevel(logging.DEBUG)
tb = testbed.Testbed()
tb.setup_env(current_version_id='testbed.version')
tb.activate()
tb.init_all_stubs()
self.testbed = tb
def tearDown(self):
self.testbed.deactivate()
def test_endpoint_testApi(self):
application = endpoints.api_server([TestApi], restricted=False)
testapp = webtest.TestApp(application)
req = RequestMessageClass(message="request message")
response = testapp.post('/_ah/spi/' + TestApi.__name__ + '.' + TestApi.test.__name__, protojson.encode_message(req),content_type='application/json')
res = protojson.decode_message(ResponseMessageClass,response.body)
self.assertEqual(res.message, 'response message')
if __name__ == '__main__':
unittest.main()
I tried everything I could think of to allow these to be tested in the normal way. I tried hitting the /_ah/spi methods directly as well as even trying to create a new protorpc app using service_mappings to no avail. I'm not a Googler on the endpoints team so maybe they have something clever to allow this to work but it doesn't appear that simply using webtest will work (unless I missed something obvious).
In the meantime you can write a test script that starts the app engine test server with an isolated environment and just issue http requests to it.
Example to run the server with an isolated environment (bash but you can easily run this from python):
DATA_PATH=/tmp/appengine_data
if [ ! -d "$DATA_PATH" ]; then
mkdir -p $DATA_PATH
fi
dev_appserver.py --storage_path=$DATA_PATH/storage --blobstore_path=$DATA_PATH/blobstore --datastore_path=$DATA_PATH/datastore --search_indexes_path=$DATA_PATH/searchindexes --show_mail_body=yes --clear_search_indexes --clear_datastore .
You can then just use requests to test ala curl:
requests.get('http://localhost:8080/_ah/...')
If you don't want to test the full HTTP stack as described by Ezequiel Muns, you can also just mock out endpoints.method and test your API definition directly:
def null_decorator(*args, **kwargs):
def decorator(method):
def wrapper(*args, **kwargs):
return method(*args, **kwargs)
return wrapper
return decorator
from google.appengine.api.users import User
import endpoints
endpoints.method = null_decorator
# decorator needs to be mocked out before you load you endpoint api definitions
from mymodule import api
class FooTest(unittest.TestCase):
def setUp(self):
self.api = api.FooService()
def test_bar(self):
# pass protorpc messages directly
self.api.foo_bar(api.MyRequestMessage(some='field'))
My solution uses one dev_appserver instance for the entire test module, which is faster than restarting the dev_appserver for each test method.
By using Google's Python API client library, I also get the simplest and at the same time most powerful way of interacting with my API.
import unittest
import sys
import os
from apiclient.discovery import build
import dev_appserver
sys.path[1:1] = dev_appserver.EXTRA_PATHS
from google.appengine.tools.devappserver2 import devappserver2
from google.appengine.tools.devappserver2 import python_runtime
server = None
def setUpModule():
# starting a dev_appserver instance for testing
path_to_app_yaml = os.path.normpath('path_to_app_yaml')
app_configs = [path_to_app_yaml]
python_runtime._RUNTIME_ARGS = [
sys.executable,
os.path.join(os.path.dirname(dev_appserver.__file__),
'_python_runtime.py')
]
options = devappserver2.PARSER.parse_args(['--port', '8080',
'--datastore_path', ':memory:',
'--logs_path', ':memory:',
'--skip_sdk_update_check',
'--',
] + app_configs)
global server
server = devappserver2.DevelopmentServer()
server.start(options)
def tearDownModule():
# shutting down dev_appserver instance after testing
server.stop()
class MyTest(unittest.TestCase):
#classmethod
def setUpClass(cls):
# build a service object for interacting with the api
# dev_appserver must be running and listening on port 8080
api_root = 'http://127.0.0.1:8080/_ah/api'
api = 'my_api'
version = 'v0.1'
discovery_url = '%s/discovery/v1/apis/%s/%s/rest' % (api_root, api,
version)
cls.service = build(api, version, discoveryServiceUrl=discovery_url)
def setUp(self):
# create a parent entity and store its key for each test run
body = {'name': 'test parent'}
response = self.service.parent().post(body=body).execute()
self.parent_key = response['parent_key']
def test_post(self):
# test my post method
# the tested method also requires a path argument "parent_key"
# .../_ah/api/my_api/sub_api/post/{parent_key}
body = {'SomeProjectEntity': {'SomeId': 'abcdefgh'}}
parent_key = self.parent_key
req = self.service.sub_api().post(body=body,parent_key=parent_key)
response = req.execute()
etc..
After digging through the sources, I believe things have changed in endpoints since Ezequiel Muns's (excellent) answer in 2014. For method 1 you now need to request from /_ah/api/* directly and use the correct HTTP method instead of using the /_ah/spi/* transformation. This makes the test file look like this:
from google.appengine.ext import testbed
import webtest
# ...
def setUp(self):
tb = testbed.Testbed()
# Setting current_version_id doesn't seem necessary anymore
tb.activate()
tb.init_all_stubs()
self.testbed = tb
def tearDown(self):
self.testbed.deactivate()
def test_endpoint_insert(self):
app = endpoints.api_server([TestEndpoint]) # restricted is no longer required
testapp = webtest.TestApp(app)
msg = {...} # a dict representing the message object expected by insert
# To be serialised to JSON by webtest
resp = testapp.post_json('/_ah/api/test/v1/insert', msg)
self.assertEqual(resp.json, {'expected': 'json response msg as dict'})
For searching's sake, the symptom of using the old method is endpoints raising a ValueError with Invalid request path: /_ah/spi/whatever. Hope that saves someone some time!