We Keep Coding

How to share/ensure connections are closed in Flask + mongo (pymongo) in Vercel serverless functions?

I am currently using vercel as my api with python serverless functions. I have serveral functions defined.
An example file could look something like...
import pymongo
from flask import Flask, jsonify
MONGO_URL = os.environ["MONGO_URL"]
CLIENT = pymongo.MongoClient(MONGO_URL)
DB = pymongo.database.Database(CLIENT, os.environ.get("DB") or "test")
app = Flask(__name__)
#app.route("/api/sample_route", methods=["GET"])
def sample_route(path=""):
COLLECTION = DB.sample_collection
if request.method == "GET":
query = {}
data = list(COLLECTION.find(query, {"_id": 0}))
return jsonify(data)
However I notice that I quickly hit the 500 connection pool working on the site on my local dev without any users. I believe there is no connection pooling going on and that they are not adequately getting closed. Any idea why?

Flask-pymongo RuntimeError: Working outside of application context

I'm writing a program to read mongodb document based on id field using flask-pymongo. But I'm getting error, can anybody tell me where am I going wrong?
code:
from flask import Flask, make_response, jsonify
from flask_pymongo import PyMongo
from collections import OrderedDict
from bson import json_util
import json
app = Flask('__name__')
app.config['MONGO_DBNAME'] = 'db_name'
app.config['MONGO_URI'] = 'mongodb://192.168.55.24:27017/db_name'
mongo_connection = PyMongo(app)
#app.route('/')
def index(inp_id):
collection = mongo_connection.db.table_name
one_record = collection.find_one({'id': inp_id})
obj_str = json_util.dumps(one_record)
obj_dict = json.loads(obj_str, object_hook=OrderedDict)
return make_response(jsonify(obj_dict), 200)
if __name__ == '__main__':
index('5cd00a468b36db516b6d2f16') # I think this is where I'm going wrong
giving me the below error:
RuntimeError: Working outside of application context.
If I pass id value directly in the place of inp_id I get the result but I'm trying to write a generic one.

Flask has an application context, You might need to use app.app_context() to make it work.
The application context keeps track of the application-level data
during a request, CLI command, or other activity. Rather than passing
the application around to each function, the current_app and g proxies
are accessed instead.
Try this :
def index(inp_id):
with app.app_context():
collection = mongo_connection.db.table_name
one_record = collection.find_one({'id': inp_id})
obj_str = json_util.dumps(one_record)
obj_dict = json.loads(obj_str, object_hook=OrderedDict)
return make_response(jsonify(obj_dict), 200)
For more information, read Flask Application context

How do I mock PyMongo for testing with a Flask app?

I've found similar questions, but they seem to only cover mocking MongoDB and don't mention Flask.
I have a Flask app and I'm trying to unit test it with PyTest (including PyTest-Mongo and PyTest-Flask). However, before I can even get to the point of writing any tests, my test script crashes. The crash happens when importing the script with my Flash app: It's trying to create the PyMongo object without a url.
My question is: How can I ensure that PyMongo is mocked correctly at this point? According to the PyTest-Mongo documentation, the MongoDB test fixture should be passed to each of the test functions, but that doesn't help me if it's crashing on import.
test_app.py:
import pytest
import pytest_mongodb
from app import app
#pytest.fixture
def client():
app.config['TESTING'] = True
return client
app.py:
import ...
app = Flask(__name__)
app.config["MONGO_DBNAME"] = os.environ.get('DB_NAME')
app.config["MONGO_URI"] = os.environ.get('MONGO_URI')
app.secret_key = os.environ.get('SECRET')
mongo = PyMongo(app)
...
if __name__ == '__main__':
app.run(host=os.environ.get('IP'),
port=int(os.environ.get('PORT')),
debug=False)

we can wrap app and mongo in a function
This works because mongo is used as a local variable.
app.py
from flask import Flask
from flask_pymongo import PyMongo
def get_app_with_config(config):
app = Flask(__name__)
app.config.from_object(config)
mongo = PyMongo(app)
#app.route("/")
def index():
pass
.
.
return app, mongo
then we can create a test file and an application execution file with different databases:
test_app.py
from app import get_app_with_config
from config import TestConfig
app, mongo = get_app_with_config(TestConfig)
run.py
from app import get_app_with_config
from config import RunConfig
app, mongo = get_app_with_config(RunConfig)
if __name__ == '__main__':
app.run(port=8000)
Sample of config.py file:
class RunConfig:
MONGO_HOST = '192.168.1.37'
MONGO_PORT = 27017
MONGO_DBNAME = 'my_database'
MONGO_URI = f"mongodb://{MONGO_HOST}:{MONGO_PORT}/{MONGO_DBNAME}"
class TestConfig:
MONGO_HOST = '192.168.1.37'
MONGO_PORT = 27017
MONGO_DBNAME = 'my_database_test'
MONGO_URI = f"mongodb://{MONGO_HOST}:{MONGO_PORT}/{MONGO_DBNAME}"
TESTING = True

Needed a quick fix so I edited app.py so that it only hard-fails if PyMongo doesn't initialise when the file is executed (i.e. it ignores PyMongo's failed initialisation when running unit-tests.)
app = Flask(__name__)
app.config["MONGO_DBNAME"] = os.environ.get('DB_NAME')
app.config["MONGO_URI"] = os.environ.get('MONGO_URI')
app.secret_key = os.environ.get('SECRET')
try:
mongodb = PyMongo(app).db
except ValueError:
"""We don't provide a URI when running unit tests, so PyMongo will fail to initialize.
This is okay because we replace it with a version for testing anyway. """
print('PyMongo not initialized!')
mongodb = None
.
.
.
if __name__ == '__main__':
if not mongodb:
print('Cannot run. PyMongo failed to initialize. Double check environment variables.')
exit(1)
app.run(host=os.environ.get('IP'),
port=int(os.environ.get('PORT')),
debug=False)
In my tests file, I just assign the mocked mongoDB client to the app in the tests that need it. Definitely not the ideal solution.
def test_redacted(client, mongodb):
app.mongodb = mongodb
...

Difficulty implementing server side session storage using redis and flask

I have a setup where a node.js app is making ajax requests to a flask based python server. Since ajax requests lack cookie data, I can't use the simple flask session object to persist data across requests. To remedy this, I'd like to implement a redis based server side implementation of a session storage system, but the solutions I've found so far do not work.
One solution I've tried is the following this snippet.
But this doesn't work. Is there more setup I need to do to configure redis beyond what is mentioned in the quickstart guide? Here is my attempt:
...
from flask import session
# Snippet code is copy pasted here verbatum
import session_interface
...
app = Flask(__name__)
app.session_interface = session_interface.RedisSessionInterface()
...
# Can't access this as session['key'] across requests
session['key'] = value
...
if __name__ == '__main__':
app.secret_key = '123456789012345678901234'
app.run(debug=True)
Another solution I've tried is importing the Flask-Session extention.
However, I can't get this to work either. The section I'm confused about is the following:
"We are not supplying something like SESSION_REDIS_HOST and SESSION_REDIS_PORT, if you want to use the RedisSessionInterface, you should configure SESSION_REDIS to your own redis.Redis instance. This gives you more flexibility, like maybe you want to use the same redis.Redis instance for cache purpose too, then you do not need to keep two redis.Redis instance in the same process."
What is meant by this section and how would I have figured this out? Here is my attempt to make this extension work:
...
from flask import session
from flask_session import Session
import redis
...
app = Flask(__name__)
SESSION_TYPE = 'redis'
app.config.from_object(__name__)
Session(app)
...
# Can't access this as session['key'] across requests
session['key'] = value
...
if __name__ == '__main__':
app.secret_key = '123456789012345678901234'
app.run(debug=True)
Has anyone successfully implemented manual session storage on a server running flask? Are there other options for getting this functionality?
Thanks for your input.

I think that's because you missed the URL configuration for your storage Redis, to check that, you can use Redis-CLI to see if there is anything being inserted into Redis.
I use this code and it worked:
from flask import Flask
from flask_session import Session
import redis
……
app = Flask(__name__)
app.config['SESSION_TYPE'] = 'redis'
app.config['SESSION_REDIS'] = redis.from_url('127.0.0.1:6379')
sess = Session()
sess.init_app(app)
def getSession():
return session.get('key', 'not set')
def setSession():
session.set('key')=123
return 'ok'
……

The following works for me:
...
from flask_session import Session
import redis
...
app = Flask(__name__)
SECRET_KEY = '123456789012345678901234'
SESSION_TYPE = 'redis'
SESSION_REDIS = redis.from_url('localhost:6379')
app.config.from_object(__name__)
sess = Session()
sess.init_app(app)
...
# Should be available accross requests now
session['key'] = value
...
if __name__ == '__main__':
app.run(debug=True)
Using sess.init_app(app) instead of Session(app) did the trick.

Heroku MongoHQ add-on and PyMongo -- OperationFailure: database error: unauthorized

I'm having trouble with the MongoHQ Heroku addon. Locally my app works and the os variable is present and well-formed on Heroku. However, when I attempt to access the db it throws an error: OperationFailure: database error: unauthorized db:my_database ns:my_database.cars lock type:0 client:128.62.187.133. If I try to hard-code the connection string from MongoHQ and run locally, I get the same error.
My app is below:
import os
import datetime
from flask import Flask
from flask import g
from flask import jsonify
from flask import json
from flask import request
from flask import url_for
from flask import redirect
from flask import render_template
from flask import make_response
import pymongo
from pymongo import Connection
from bson import BSON
from bson import json_util
app = Flask(__name__)
def mongo_conn():
# Format: MONGOHQ_URL: mongodb://<user>:<pass>#<base_url>:<port>/<url_path>
if os.environ.get('MONGOHQ_URL'):
return Connection(os.environ['MONGOHQ_URL'])
else:
return Connection()
#app.route('/', methods=['GET', 'POST'])
def hello():
# Get your DB
connection = mongo_conn()
db = connection.my_database
# Create an object
car = {"brand": "Ford",
"model": "Mustang",
"date": datetime.datetime.utcnow()}
# Get your collection
cars = db.cars # crashes
# Insert it
cars.insert(car)
...
Edit: MongoHQ support helped me. Problem was that I was calling my database my_database instead of the actual DB name given to me by the MongoHQ addon. E.g., db = connection.app52314314. That change fixed it.

You likely need to run the authenticate command against the DB directly after you connect.
Try something like this:
db.authenticate([USER], [PASSWORD])
If that doesn't work, feel free to email support#mongohq.com and we can help you out with your specific DB.

You don't need to do all that. You can simply:
from pymongo import MongoClient
client = MongoClient(os.environ['MONGOHQ_URL'])
mongo_db = client.get_default_database()
It will automatically authenticate you, and connect to the provisioned database, the <url_path> part of your connection url.