I am trying to add some monitoring to a simple REST web service with flask and mongoengine and have come across what I think is a lack of understanding on my part of how imports and mongoengine is working in flask applications.
I'm following pymongo's documentation on monitoring : https://pymongo.readthedocs.io/en/3.7.2/api/pymongo/monitoring.html
I defined the following CommandListener in a separate file:
import logging
from pymongo import monitoring
log = logging.getLogger('my_logger')
class CommandLogger(monitoring.CommandListener):
def started(self, event):
log.debug("Command {0.command_name} with request id "
"{0.request_id} started on server "
"{0.connection_id}".format(event))
monitoring.register(CommandLogger())
I made an application_builder.py file to create my flask App, code looks something like this:
from flask_restful import Api
from flask import Flask
from command_logger import CommandLogger # <----
from db import initialize_db
from routes import initialize_routes
def create_app():
app = Flask(__name__)
api = Api(app)
initialize_db(app)
initialize_routes(api)
return app
The monitoring only seems to works if I import : CommandLogger in application_builder.py. I'd like to understand what is going on here, how does the import affect the monitoring registration?
Also I'd like to extract monitoring.register(CommandLogger()) as a function and call it at a latter stage in my code something like def register(): monitoring.register(CommandLogger())
But this doesn't seem to work, "registration' only works when it is in the same file as the CommandLogger class...
From the MongoEngine's doc, it seems important that the listener gets registered before connecting mongoengine
To use pymongo.monitoring with MongoEngine, you need to make sure that
you are registering the listeners before establishing the database
connection (i.e calling connect)
This worked for me. I'm just initializing/registering it the same way as I did other modules to avoid circular imports.
# admin/logger.py
import logging
from pymongo import monitoring
log = logging.getLogger()
log.setLevel(logging.DEBUG)
logging.basicConfig(level=logging.DEBUG)
class CommandLogger(monitoring.CommandListener):
# def methods...
class ServerLogger(monitoring.ServerListener):
# def methods
class HeartbeatLogger(monitoring.ServerHeartbeatListener):
# def methods
def initialize_logger():
monitoring.register(CommandLogger())
monitoring.register(ServerLogger())
monitoring.register(HeartbeatLogger())
monitoring.register(TopologyLogger())
# /app.py
from flask import Flask
from admin.toolbar import initialize_debugtoolbar
from admin.admin import initialize_admin
from admin.views import initialize_views
from admin.logger import initialize_logger
from database.db import initialize_db
from flask_restful import Api
from resources.errors import errors
app = Flask(__name__)
# imports requiring app
from resources.routes import initialize_routes
api = Api(app, errors=errors)
# Logger before db
initialize_logger()
# Database and Routes
initialize_db(app)
initialize_routes(api)
# Admin and Development
initialize_admin(app)
initialize_views()
initialize_debugtoolbar(app)
# /run.py
from app import app
app.run(debug=True)
then in any module...
from admin.logger import log
from db.models import User
# inside some class/view/queryset or however your objects are written...
log.info('Saving an item through MongoEngine...')
User(name='Foo').save()
What I'm trying to figure out now is how to integrate Flask DebuggerToolbar's Logging panel with the monitoring messages from these listeners...
Related
I use flask app factory:
app.py
from flask import Flask, g
from . import commands
from .database import DatabaseManager
from .extensions import db
from .settings import ProdConfig
def create_app(config_object=ProdConfig):
'''Flask app factory'''
app = Flask(__name__)
app.url_map.strict_slashes = False
app.config.from_object(config_object)
register_extensions(app)
register_blueprints(app)
register_commands(app)
#app.before_request
def create_context():
g.db = db.database
g.db_manager = DatabaseManager()
return app
# Another functions
# ...
def register_commands(app):
app.cli.add_command(commands.root_group)
Then i wrote command in:
commands.py
from flask import g
from flask.cli import AppGroup
root_group = AppGroup('myapp')
#root_group.command('test')
def test():
print(g.db)
Then when i run command flask myapp test i get AttributeError on g.db. So somehow flask.g dont receive attrs g.db and g.db_manager initialized in before_request. As i understand it have to receive, because app_context must be pushed when cli command based on AppGroup runs. How to fix this?
From testing i understand that when CLI command executes, Flask not push Request context, so #before_request hook not executes. I dont find good way to push Request context in my cli command and make this hack:
#root_group.command()
def test():
exec_before_request_hooks()
print(g.db_manager)
# HACK: run #app.before_request for CLI command
def exec_before_request_hooks():
current_app.test_client().get()
If anybody know better solution, i will be very appreciate to hear it)
I had tried the code below for getting the exception in https://learn.microsoft.com/en-us/azure/azure-monitor/app/opencensus-python
from opencensus.ext.azure.log_exporter import AzureLogHandler
logger = logging.getLogger(__name__)
# TODO: replace the all-zero GUID with your instrumentation key.
logger.addHandler(AzureLogHandler(
connection_string='InstrumentationKey=00000000-0000-0000-0000-000000000000')
)
properties = {'custom_dimensions': {'key_1': 'value_1', 'key_2': 'value_2'}}
# Use properties in exception logs
try:
result = 1 / 0 # generate a ZeroDivisionError
except Exception:
logger.exception('Captured an exception.', extra=properties)
It is working. I can catch the exception.
However, I want to ask if there is an easy way to catch the exception automatically in the python flask?
Since I try the below code, it just gives me a request record, not the exception.
app = Flask(__name__)
app.logger.addHandler(file_handler)
handler = AzureEventHandler(
connection_string="InstrumentationKey={}".format(app.config['APPINSIGHTS_INSTRUMENTATIONKEY']))
handler.setFormatter(logging.Formatter(
'%(asctime)s %(levelname)s: %(message)s '
'[in %(pathname)s:%(lineno)d]'))
handler.setLevel(logging.ERROR)
app.logger.addHandler(handler)
Thank you for helping
Your code should like below. For more details, you can check offical sample code.
Flask "To-Do" Sample Application
import logging
import sys
from flask import Flask
sys.path.append('..')
from config import Config
from flask_sqlalchemy import SQLAlchemy
from opencensus.ext.azure import metrics_exporter
from opencensus.ext.azure.log_exporter import AzureLogHandler
from opencensus.ext.flask.flask_middleware import FlaskMiddleware
from opencensus.trace import config_integration
logger = logging.getLogger(__name__)
app = Flask(__name__)
app.config.from_object(Config)
db = SQLAlchemy(app)
# Import here to avoid circular imports
from app import routes # noqa isort:skip
# Trace integrations for sqlalchemy library
config_integration.trace_integrations(['sqlalchemy'])
# Trace integrations for requests library
config_integration.trace_integrations(['requests'])
# FlaskMiddleware will track requests for the Flask application and send
# request/dependency telemetry to Azure Monitor
middleware = FlaskMiddleware(app)
# Processor function for changing the role name of the app
def callback_function(envelope):
envelope.tags['ai.cloud.role'] = "To-Do App"
return True
# Adds the telemetry processor to the trace exporter
middleware.exporter.add_telemetry_processor(callback_function)
# Exporter for metrics, will send metrics data
exporter = metrics_exporter.new_metrics_exporter(
enable_standard_metrics=False,
connection_string='InstrumentationKey=' + Config.INSTRUMENTATION_KEY)
# Exporter for logs, will send logging data
logger.addHandler(
AzureLogHandler(
connection_string='InstrumentationKey=' + Config.INSTRUMENTATION_KEY
)
)
if __name__ == '__main__':
app.run(host='localhost', port=5000, threaded=True, debug=True)
I am newbiee on pyhton development and trying to make the following example work.
The following code is for service authentication. Whenever I call localhost:3000/callback through browser, I am receiving code error because it is null.
When I create a webpage on auth0, then it puts all required information and then make the source code available to use. However, code is not included. I wonder what needs to inserted.
token = get_token.authorization_code(AUTH0_CLIENT_ID,
AUTH0_CLIENT_SECRET, code,
AUTH0_CALLBACK_URL)
auth0.v3.exceptions.Auth0Error: invalid_request: Missing required
parameter: code
.env
AUTH0_CLIENT_ID=xxxxxxxxxxx
AUTH0_DOMAIN=xxxxxx.auth0.com
AUTH0_CLIENT_SECRET=v-xxxxx2yVHntpn01RoEMMxhj6RLxxxxxxxxxx
AUTH0_CALLBACK_URL=http://localhost:3000/callback
API_IDENTIFIER={API_AUDIENCE}
server.py
"""Python Flask WebApp Auth0 integration example"""
from functools import wraps
from urllib.parse import urlparse
from os import environ as env, path
import json
from auth0.v3.authentication import GetToken
from auth0.v3.authentication import Users
from dotenv import load_dotenv
from flask import Flask
from flask import redirect
from flask import render_template
from flask import request
from flask import send_from_directory
from flask import session
import constants
load_dotenv(path.join(path.dirname(__file__), ".env"))
AUTH0_CALLBACK_URL = env[constants.AUTH0_CALLBACK_URL]
AUTH0_CLIENT_ID = env[constants.AUTH0_CLIENT_ID]
AUTH0_CLIENT_SECRET = env[constants.AUTH0_CLIENT_SECRET]
AUTH0_DOMAIN = env[constants.AUTH0_DOMAIN]
#APP.route('/callback')
def callback_handling():
code = request.args.get(constants.CODE_KEY)
get_token = GetToken(AUTH0_DOMAIN)
auth0_users = Users(AUTH0_DOMAIN)
#Receive exception
token = get_token.authorization_code(AUTH0_CLIENT_ID,
AUTH0_CLIENT_SECRET, code, AUTH0_CALLBACK_URL)
user_info = auth0_users.userinfo(token['access_token'])
session[constants.PROFILE_KEY] = json.loads(user_info)
return redirect('/dashboard')
if __name__ == "__main__":
APP.run(host='0.0.0.0', port=env.get('PORT', 3000))
I wanna be able to run multiple twisted proxy servers on different directories on the same port simultaneously, and I figured I might use flask.
so here's my code:
from flask import Flask
from twisted.internet import reactor
from twisted.web import proxy, server
app = Flask(__name__)
#app.route('/example')
def index():
site = server.Site(proxy.ReverseProxyResource('www.example.com', 80, ''.encode("utf-8")))
reactor.listenTCP(80, site)
reactor.run()
app.run(port=80, host='My_IP')
But whenever I run this script, I get an Internal Server Error, I'm assuming because when app.run is called on port 80, the reactor.run can't be listening on port 80 as well. I wondering if there is some kind of work around to this, or what it is I'm doing wrong. Any help is greatly appreciated, Thanks!!
You can use the WSGIResource from Twisted istead of a ReverseProxy.
UPDATE: Added a more complex example that sets up a WSGIResource at /my_flask and a ReverseProxy at /example
from flask import Flask
from twisted.internet import reactor
from twisted.web.proxy import ReverseProxyResource
from twisted.web.resource import Resource
from twisted.web.server import Site
from twisted.web.wsgi import WSGIResource
app = Flask(__name__)
#app.route('/example')
def index():
return 'My Twisted Flask'
flask_site = WSGIResource(reactor, reactor.getThreadPool(), app)
root = Resource()
root.putChild('my_flask', flask_site)
site_example = ReverseProxyResource('www.example.com', 80, '/')
root.putChild('example', site_example)
reactor.listenTCP(8081, Site(root))
reactor.run()
Try running the above in your localhost and then visiting localhost:8081/my_flask/example or localhost:8081/example
You should give klein a try. It's made and used by most of the twisted core devs. The syntax is very much like flask so you won't have to rewrite much if you already have a working flask app. So something like the following should work:
from twisted.internet import reactor
from twisted.web import proxy, server
from klein import Klein
app = Klein()
#app.route('/example')
def home(request):
site = server.Site(proxy.ReverseProxyResource('www.example.com', 80, ''.encode("utf-8")))
reactor.listenTCP(80, site)
app.run('localhost', 8000) # start the klein app on port 8000 and reactor event loop
Links
Klein Docs
Klein Github
The accepted answer does not cover how to run twisted with Flask, and points to a different framework. The answer with an example no longer works either.
Here are two different examples. The first one is the same as the first answer, but fixed to work on Python 3
from flask import Flask
from twisted.internet import reactor
from twisted.web.proxy import ReverseProxyResource
from twisted.web.resource import Resource
from twisted.web.server import Site
from twisted.web.wsgi import WSGIResource
app = Flask(__name__)
#app.route('/example')
def index():
return 'My Twisted Flask'
flask_site = WSGIResource(reactor, reactor.getThreadPool(), app)
root = Resource()
root.putChild(b'my_flask', flask_site)
site_example = ReverseProxyResource('www.example.com', 80, b'/')
root.putChild(b'example', site_example)
reactor.listenTCP(8081, Site(root))
reactor.run()
For this example, run it and open any of these:
localhost:8081/my_flask/example
localhost:8081/example
This other example is recommended, since it sets up two services and provides them through a .tac file to twistd.
Take the base code from here: https://github.com/pika/pika/blob/master/examples/twisted_service.py
"""Modify the bottom of the file to pick the new MultiService"""
# ... all the code from twisted_service.py goes here.
# Scroll to the bottom of the file and
# remove everything below application = service.Application("pikaapplication")
# You should keep the PikaService, PikaProtocol and PikaFactory
# classes, since we need them for this code:
from pika.connection import ConnectionParameters
from pika.credentials import PlainCredentials
from twisted.application import service, strports
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.wsgi import WSGIResource
from flask import Flask
# This IServiceCollection will hold Pika and Flask
flask_pika_multiservice = service.MultiService()
# FLASK SERVICE SETUP
app = Flask("demoapp")
#app.route('/')
def hello_world():
return 'Hello, World!'
flask_site = Site(WSGIResource(reactor, reactor.getThreadPool(), app))
# New resources can be added, such as WSGI, or proxies by creating
# a root resource in the place of the flask_site, and adding the
# new resources to the root.
# root = Resource()
# root.putChild(b'my_flask_site', flask_site)
# from twisted.web.proxy import ReverseProxyResource
# site_example = ReverseProxyResource('www.example.com', 80, b'/')
# root.putChild(b'example', site_example)
i = strports.service(f"tcp:8088", flask_site)
i.setServiceParent(flask_pika_multiservice)
# PIKA SERVICE SETUP
ps = PikaService(
ConnectionParameters(
host="localhost",
virtual_host="/",
credentials=PlainCredentials("guest", "guest")))
ps.setServiceParent(flask_pika_multiservice)
# Application setup
application = service.Application('flask_pika')
flask_pika_multiservice.setServiceParent(application)
Now you can run it with:
PYTHONPATH=. twistd -ny twisted_service.py
you can skip the python path if you don't want to import anything from the same path. twisted expects projects to actually be installed, and does not support running them directly from the source folder unless you use that workaround.
This second example establishes two services, on different ports. It's for pika and flask running simultaneously on the same twisted server. The best part is that it shows how to set up flask as a service, that can be part of an IServiceCollection
I'm having trouble with the MongoHQ Heroku addon. Locally my app works and the os variable is present and well-formed on Heroku. However, when I attempt to access the db it throws an error: OperationFailure: database error: unauthorized db:my_database ns:my_database.cars lock type:0 client:128.62.187.133. If I try to hard-code the connection string from MongoHQ and run locally, I get the same error.
My app is below:
import os
import datetime
from flask import Flask
from flask import g
from flask import jsonify
from flask import json
from flask import request
from flask import url_for
from flask import redirect
from flask import render_template
from flask import make_response
import pymongo
from pymongo import Connection
from bson import BSON
from bson import json_util
app = Flask(__name__)
def mongo_conn():
# Format: MONGOHQ_URL: mongodb://<user>:<pass>#<base_url>:<port>/<url_path>
if os.environ.get('MONGOHQ_URL'):
return Connection(os.environ['MONGOHQ_URL'])
else:
return Connection()
#app.route('/', methods=['GET', 'POST'])
def hello():
# Get your DB
connection = mongo_conn()
db = connection.my_database
# Create an object
car = {"brand": "Ford",
"model": "Mustang",
"date": datetime.datetime.utcnow()}
# Get your collection
cars = db.cars # crashes
# Insert it
cars.insert(car)
...
Edit: MongoHQ support helped me. Problem was that I was calling my database my_database instead of the actual DB name given to me by the MongoHQ addon. E.g., db = connection.app52314314. That change fixed it.
You likely need to run the authenticate command against the DB directly after you connect.
Try something like this:
db.authenticate([USER], [PASSWORD])
If that doesn't work, feel free to email support#mongohq.com and we can help you out with your specific DB.
You don't need to do all that. You can simply:
from pymongo import MongoClient
client = MongoClient(os.environ['MONGOHQ_URL'])
mongo_db = client.get_default_database()
It will automatically authenticate you, and connect to the provisioned database, the <url_path> part of your connection url.