We Keep Coding

How to use pymongo.monitoring in a Flask project with mongoengine

I am trying to add some monitoring to a simple REST web service with flask and mongoengine and have come across what I think is a lack of understanding on my part of how imports and mongoengine is working in flask applications.
I'm following pymongo's documentation on monitoring : https://pymongo.readthedocs.io/en/3.7.2/api/pymongo/monitoring.html
I defined the following CommandListener in a separate file:
import logging
from pymongo import monitoring
log = logging.getLogger('my_logger')
class CommandLogger(monitoring.CommandListener):
def started(self, event):
log.debug("Command {0.command_name} with request id "
"{0.request_id} started on server "
"{0.connection_id}".format(event))
monitoring.register(CommandLogger())
I made an application_builder.py file to create my flask App, code looks something like this:
from flask_restful import Api
from flask import Flask
from command_logger import CommandLogger # <----
from db import initialize_db
from routes import initialize_routes
def create_app():
app = Flask(__name__)
api = Api(app)
initialize_db(app)
initialize_routes(api)
return app
The monitoring only seems to works if I import : CommandLogger in application_builder.py. I'd like to understand what is going on here, how does the import affect the monitoring registration?
Also I'd like to extract monitoring.register(CommandLogger()) as a function and call it at a latter stage in my code something like def register(): monitoring.register(CommandLogger())
But this doesn't seem to work, "registration' only works when it is in the same file as the CommandLogger class...

From the MongoEngine's doc, it seems important that the listener gets registered before connecting mongoengine
To use pymongo.monitoring with MongoEngine, you need to make sure that
you are registering the listeners before establishing the database
connection (i.e calling connect)

This worked for me. I'm just initializing/registering it the same way as I did other modules to avoid circular imports.
# admin/logger.py
import logging
from pymongo import monitoring
log = logging.getLogger()
log.setLevel(logging.DEBUG)
logging.basicConfig(level=logging.DEBUG)
class CommandLogger(monitoring.CommandListener):
# def methods...
class ServerLogger(monitoring.ServerListener):
# def methods
class HeartbeatLogger(monitoring.ServerHeartbeatListener):
# def methods
def initialize_logger():
monitoring.register(CommandLogger())
monitoring.register(ServerLogger())
monitoring.register(HeartbeatLogger())
monitoring.register(TopologyLogger())
# /app.py
from flask import Flask
from admin.toolbar import initialize_debugtoolbar
from admin.admin import initialize_admin
from admin.views import initialize_views
from admin.logger import initialize_logger
from database.db import initialize_db
from flask_restful import Api
from resources.errors import errors
app = Flask(__name__)
# imports requiring app
from resources.routes import initialize_routes
api = Api(app, errors=errors)
# Logger before db
initialize_logger()
# Database and Routes
initialize_db(app)
initialize_routes(api)
# Admin and Development
initialize_admin(app)
initialize_views()
initialize_debugtoolbar(app)
# /run.py
from app import app
app.run(debug=True)
then in any module...
from admin.logger import log
from db.models import User
# inside some class/view/queryset or however your objects are written...
log.info('Saving an item through MongoEngine...')
User(name='Foo').save()
What I'm trying to figure out now is how to integrate Flask DebuggerToolbar's Logging panel with the monitoring messages from these listeners...

Use zerorpc inside Flask app throws error "operation would block forever"

I have a RPC Server using zerorpc in Python, written this way
import zerorpc
from service import Service
print('RPC server - loading')
def main():
print('RPC server - main')
s = zerorpc.Server(Service())
s.bind("tcp://*:4242")
s.run()
if __name__ == "__main__" : main()
It works fine when I create a client
import zerorpc, sys
client_rpc = zerorpc.Client()
client_rpc.connect("tcp://127.0.0.1:4242")
name = sys.argv[1] if len(sys.argv) > 1 else "dude"
print(client_rpc.videos('138cd9e5-3c4c-488a-9b6f-49907b55a040.webm'))
and runs it. The print() outputs what this 'videos' function returns.
But when I try to use it this same code inside route from a Flask app, I receive the following error:
File "src/gevent/__greenlet_primitives.pxd", line 35, in
gevent.__greenlet_primitives._greenlet_switch
gevent.exceptions.LoopExit: This operation would block forever Hub:
The flask method/excerpt
import zerorpc, sys
client_rpc = zerorpc.Client()
client_rpc.connect("tcp://127.0.0.1:4242")
#app.route('/videos', methods=['POST'])
def videos():
global client_rpc
client_rpc.videos('138cd9e5-3c4c-488a-9b6f-49907b55a040.webm')
I can't find out what might be happening. I'm quite new to Python and I understand that this may have some relation with Flask and how it handles the thread, but I can't figure out how to solve it.

zerorpc depends on gevent, which provides async IO with cooperative coroutines. This means your flask application must use gevent for all IO operations.
In your specific case, you are likely starting your application with a standard blocking IO WSGI server.
Here is a snippet using the WSGI server from gevent:
import zerorpc
from gevent.pywsgi import WSGIServer
app = Flask(__name__)
client_rpc = zerorpc.Client()
client_rpc.connect("tcp://127.0.0.1:4242")
#app.route('/videos', methods=['POST'])
def videos():
global client_rpc
client_rpc.videos('138cd9e5-3c4c-488a-9b6f-49907b55a040.webm')
# ...
if __name__ == "__main__":
http = WSGIServer(('', 5000), app)
http.serve_forever()
Excerpt from https://sdiehl.github.io/gevent-tutorial/#chat-server

Trying to deploy a Flask app on CherryPy server

I was trying to deploy my Flask app on CherryPy server. I liked its simplistic and minimalistic nature.
So I PIP'ed CherryPy like below
pip install CherryPy-15.0.0-py2.py3-none-any.whl
and wrote script like below -very common suggested by many sources
from cherrypy import wsgiserver
from hello import app
d = wsgiserver.WSGIPathInfoDispatcher({'/': app})
server = wsgiserver.CherryPyWSGIServer(('0.0.0.0', 80), d)
if __name__ == '__main__':
try:
server.start()
except KeyboardInterrupt:
server.stop()
To my surprise, I had imports errors. After a few googling around, I learned that I had to change my import lines to cheroot to make it work.
from cheroot.wsgi import Server
from cheroot.wsgi import PathInfoDispatcher
Now, my code is working fine.
However, I am a bit confused if this is the right way of using CherryPy WSGI server or if I pip'ed a wrong version of CherryPy. I am confused because Cheroot seems to be more than year old (dates all the way back to 2014), yet all the information I found around Flask on CherryPy WSGI server is using from cherrypy import wsgiserver, not from cheroot.wsgi import Server, even the latest postings.
This makes me unsure if I am doing the right thing or not.
Can someone please shed light on this confusion?

Cheroot (src) is a low-level HTTP and WSGI server, which used to be a part of CherryPy (src) once, but has been factored out into a separate repo a while back. So former cherrypy.wsgiserver has moved to cheroot.wsgi module.
It's completely replaceable and designed to allow developers to depend on Cheroot directly if they only use WSGI server, not requiring other parts of CherryPy.
So here's how you can use it in a version-agnostic way:
try:
from cheroot.wsgi import Server as WSGIServer, PathInfoDispatcher
except ImportError:
from cherrypy.wsgiserver import CherryPyWSGIServer as WSGIServer, WSGIPathInfoDispatcher as PathInfoDispatcher
from hello import app
d = PathInfoDispatcher({'/': app})
server = WSGIServer(('0.0.0.0', 80), d)
if __name__ == '__main__':
try:
server.start()
except KeyboardInterrupt:
server.stop()

Tornado: How to securely store SSL certificate?

I currently have my crt and key files stored locally. This is the code for my tornado server:
import tornado.httpserver
import tornado.ioloop
import tornado.web
from flasky import app
from tornado.wsgi import WSGIContainer
from tornado.ioloop import IOLoop
from tornado.web import FallbackHandler
tr = WSGIContainer(app)
application = tornado.web.Application([
(r".*", FallbackHandler, dict(fallback=tr)),
])
if __name__ == '__main__':
http_server = tornado.httpserver.HTTPServer(application, ssl_options={
"certfile": "C:/Source/cert/certificate.crt",
"keyfile": "C:/Source/cert/keyfile-decrypted.key",
})
http_server.listen(8080)
IOLoop.instance().start()
Is there a way I can cache the keyfile? Or store it in some keyvault? If it can be done using keyvault, what commands do i need to run to store it and later access it on python? I'm not sure what are the best security practices concerning this?

How to run twisted with flask?

I wanna be able to run multiple twisted proxy servers on different directories on the same port simultaneously, and I figured I might use flask.
so here's my code:
from flask import Flask
from twisted.internet import reactor
from twisted.web import proxy, server
app = Flask(__name__)
#app.route('/example')
def index():
site = server.Site(proxy.ReverseProxyResource('www.example.com', 80, ''.encode("utf-8")))
reactor.listenTCP(80, site)
reactor.run()
app.run(port=80, host='My_IP')
But whenever I run this script, I get an Internal Server Error, I'm assuming because when app.run is called on port 80, the reactor.run can't be listening on port 80 as well. I wondering if there is some kind of work around to this, or what it is I'm doing wrong. Any help is greatly appreciated, Thanks!!

You can use the WSGIResource from Twisted istead of a ReverseProxy.
UPDATE: Added a more complex example that sets up a WSGIResource at /my_flask and a ReverseProxy at /example
from flask import Flask
from twisted.internet import reactor
from twisted.web.proxy import ReverseProxyResource
from twisted.web.resource import Resource
from twisted.web.server import Site
from twisted.web.wsgi import WSGIResource
app = Flask(__name__)
#app.route('/example')
def index():
return 'My Twisted Flask'
flask_site = WSGIResource(reactor, reactor.getThreadPool(), app)
root = Resource()
root.putChild('my_flask', flask_site)
site_example = ReverseProxyResource('www.example.com', 80, '/')
root.putChild('example', site_example)
reactor.listenTCP(8081, Site(root))
reactor.run()
Try running the above in your localhost and then visiting localhost:8081/my_flask/example or localhost:8081/example

You should give klein a try. It's made and used by most of the twisted core devs. The syntax is very much like flask so you won't have to rewrite much if you already have a working flask app. So something like the following should work:
from twisted.internet import reactor
from twisted.web import proxy, server
from klein import Klein
app = Klein()
#app.route('/example')
def home(request):
site = server.Site(proxy.ReverseProxyResource('www.example.com', 80, ''.encode("utf-8")))
reactor.listenTCP(80, site)
app.run('localhost', 8000) # start the klein app on port 8000 and reactor event loop
Links
Klein Docs
Klein Github

The accepted answer does not cover how to run twisted with Flask, and points to a different framework. The answer with an example no longer works either.
Here are two different examples. The first one is the same as the first answer, but fixed to work on Python 3
from flask import Flask
from twisted.internet import reactor
from twisted.web.proxy import ReverseProxyResource
from twisted.web.resource import Resource
from twisted.web.server import Site
from twisted.web.wsgi import WSGIResource
app = Flask(__name__)
#app.route('/example')
def index():
return 'My Twisted Flask'
flask_site = WSGIResource(reactor, reactor.getThreadPool(), app)
root = Resource()
root.putChild(b'my_flask', flask_site)
site_example = ReverseProxyResource('www.example.com', 80, b'/')
root.putChild(b'example', site_example)
reactor.listenTCP(8081, Site(root))
reactor.run()
For this example, run it and open any of these:
localhost:8081/my_flask/example
localhost:8081/example
This other example is recommended, since it sets up two services and provides them through a .tac file to twistd.
Take the base code from here: https://github.com/pika/pika/blob/master/examples/twisted_service.py
"""Modify the bottom of the file to pick the new MultiService"""
# ... all the code from twisted_service.py goes here.
# Scroll to the bottom of the file and
# remove everything below application = service.Application("pikaapplication")
# You should keep the PikaService, PikaProtocol and PikaFactory
# classes, since we need them for this code:
from pika.connection import ConnectionParameters
from pika.credentials import PlainCredentials
from twisted.application import service, strports
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.wsgi import WSGIResource
from flask import Flask
# This IServiceCollection will hold Pika and Flask
flask_pika_multiservice = service.MultiService()
# FLASK SERVICE SETUP
app = Flask("demoapp")
#app.route('/')
def hello_world():
return 'Hello, World!'
flask_site = Site(WSGIResource(reactor, reactor.getThreadPool(), app))
# New resources can be added, such as WSGI, or proxies by creating
# a root resource in the place of the flask_site, and adding the
# new resources to the root.
# root = Resource()
# root.putChild(b'my_flask_site', flask_site)
# from twisted.web.proxy import ReverseProxyResource
# site_example = ReverseProxyResource('www.example.com', 80, b'/')
# root.putChild(b'example', site_example)
i = strports.service(f"tcp:8088", flask_site)
i.setServiceParent(flask_pika_multiservice)
# PIKA SERVICE SETUP
ps = PikaService(
ConnectionParameters(
host="localhost",
virtual_host="/",
credentials=PlainCredentials("guest", "guest")))
ps.setServiceParent(flask_pika_multiservice)
# Application setup
application = service.Application('flask_pika')
flask_pika_multiservice.setServiceParent(application)
Now you can run it with:
PYTHONPATH=. twistd -ny twisted_service.py
you can skip the python path if you don't want to import anything from the same path. twisted expects projects to actually be installed, and does not support running them directly from the source folder unless you use that workaround.
This second example establishes two services, on different ports. It's for pika and flask running simultaneously on the same twisted server. The best part is that it shows how to set up flask as a service, that can be part of an IServiceCollection