I'm developing a basic flask application to receive an input request from the user and insert that into MongoDB Atlas Cluster.
I have a route /save which is of type POST. This endpoint receives the request, makes a new mongo connection, inserts to mongo and finally closes the connection. This approach is slow, with average response latency of 700-800 ms even though I am only trying to insert one document.
Note-: My use case does not make sense to use bulk-insert.
Sample Code
app = Flask(__name__)
app.logger.setLevel(logging.INFO)
DBNAME = 'DBNAME as String'
CONNSTRING = 'CONNECTION as String'
class mongoDB:
def __init__(self):
try:
self.client = MongoClient(CONNSTRING, maxPoolSize=None)
self.database = self.client[DBNAME]
app.logger.info('Mongo Connection Established')
except Exception as e:
app.logger.warning('Mongo Connection could not be established')
app.logger.warning('Error Message: ' + str(e))
def close_connection(self):
try:
self.client.close()
except Exception as e:
app.logger.warning('connection failed to close')
app.logger.warning('Error Message: ' + str(e))
#app.route('/save', methods=['POST'])
def save():
data_info = flask.request.get_json()
try:
db = mongoDB()
image_collection = db.database['DUMMY_COLLECTION']
image_collection.insert_one({'VALUE_ID' : data_info['value1'], 'VALUE_STRING' : data_info['value2']})
app.logger.info('Inserted Successfully')
return {'message': 'Success'}, 200, {'Content-Type': 'application/json'}
except Exception as e:
app.logger.error('Error Adding data to Mongo: ' + str(e))
return {'message': 'Error'}, 500, {'Content-Type': 'application/json'}
finally:
db.close_connection()
app.logger.info('connection closed')
if __name__ == '__main__':
app.run()
However if I establish the Mongo connection at time of application initialization and keep it open and never close the connection the latency drops to 70-80ms.
Could someone please help understand the consequences of keeping an open connection, instead of establishing a new connection with each request? Or is there any method to reduce latency with multiple connections open?
Note-: Keeping multiple connections approach, I tried using writeConcern=0, maxPoolSize=None, and journal=False but all these did not improve the latency much at all.
Any help would be appreciated. Thanks
The MongoClient(CONNSTRING, maxPoolSize=None) is not just a single connection but a connection pool. Meaning that already with that object you can have multiple concurrent requests to MongoDB. By setting maxPoolSize=None you make them limitless (which can have some implications under heavy load).
It is an antipattern to create a connection pool per request (as you realized by the high latency) the reason for that is that you need to pay each time the cost to create the connection pool and the handshake to the database.
The best way is to initiate one on startup and then maintain it. Meaning that you should handle all the exceptions that might arise in case of DB or network failures. However, I assume that most things are already handled by MongoClient already.
Related
I am developing a flutter app using flask as back end framework and mariabd as database
Trying to reduce web service time response of ws:
1- open the connexion at the begining of ws
2- Execute queries
3-close connexion to database before returnning the response
Here is an exemple of my code archi:
#app.route('/ws_name', methods=['GET'])
def ws_name():
cnx=db_connexion()
try:
id_lanparamguage = request.args.get('param')
result = function_execute_many_query(cnx,param)
except:
cnx.close()
return jsonify(result), 200
response = {}
cnx.close()
return jsonify(result), 200
db_connexion is my function that handle connecting to database
The probleme is when only one user is connecting to the app (use ws) the time response is perfect
but if 3 users (as exemple) are connected th time response is up from millisecond to 10 seconds
I suspect you have a problem with many requests sharing the same thread. Read https://werkzeug.palletsprojects.com/en/1.0.x/local/ for how the local context works and why you need werkzeug to manage your local context in an WSGI application.
You would want to do something like:
from werkzeug.local import LocalProxy
cnx=LocalProxy(db_connexion)
I also recommend closing your connextion in a function decorated by #app.teardown_request
See https://flask.palletsprojects.com/en/1.1.x/api/#flask.Flask.teardown_request
My application will be sending hundreds, if not thousands, of messages over redis every second, so we are going to open the connection when the app launches and use that connection for every transaction. I am having trouble keeping that connection open.
Here is some of my redis handler class:
class RedisSingleDatabase(AsyncGetSetDatabase):
async def redis(self):
if not self._redis:
self._redis = await aioredis.create_redis(('redis', REDIS_PORT))
return self._redis
def __init__(self):
self._redis = None
async def get(self, key):
r = await self.redis()
# r = await aioredis.create_redis(('redis', REDIS_PORT))
print(f'CONNECTION IS {"CLOSED" if r.closed else "OPEN"}!')
data = await r.get(key, encoding='utf-8')
if data is not None:
return json.loads(data)
This does not work. By the time we call r.get, the connection is closed (ConnectionClosedError).
If I uncomment the second line of the get method, and connect to the database locally in the method, it works. But then we're no longer using the same connection.
I've considered trying all this using dependency injection as well. This is a flask app, and in my create_app method I would connect to the database and construct my RedisSingleDatabase with that connection. Besides the question of whether my current problem would be the same, I can't seem to get this approach to work since we need to await the connection which I can't do in my root-level create_app which can't be async! (unless I'm missing something).
I've tried asyncio.run all over the place and it either doesn't fix the problem or raises its own error that it can't be called from a running event loop.
Please help!!!
Working with a MySQL database and flask-sqlalchemy I am encountering a lost connection error ('Lost connection to MySQL server during query'). I already have adapted the app.config['SQLALCHEMY_POOL_RECYCLE'] to be smaller than the engine timeout. I also added a pool_pre_ping, to ensure the database is not going away during two requests. Now I have no idea left, how this can still be an issue, since it is my understanding that flask-sqlalchemy should be taking care of opening and closing sessions correctly.
As a workaround, I thought about a way to tell flask-sqlalchemy to catch lost connection responses and restart the connection on the fly. But I have no idea how to do this. So, my questions are:
Do you know what could possibly cause my connection loss?
Do you think, my recent approach of catching is a good idea or do you have a better suggestion?
If this is a good idea, how can I do this most conveniently? I don't want to wrap all requests in try-catch-statements, since I have a lot of code.
I do not know the answer to your 1st and 2nd questions, but for the 3rd question, I used a decorator to wrap all my functions instead of using try / except directly inside the functions. The explicit pre_ping and session rollback / close somehow also solved the problem of Lost Connection for me (mariadb was the backend I was using)!
def manage_session(f):
def inner(*args, **kwargs):
# MANUAL PRE PING
try:
db.session.execute("SELECT 1;")
db.session.commit()
except:
db.session.rollback()
finally:
db.session.close()
# SESSION COMMIT, ROLLBACK, CLOSE
try:
res = f(*args, **kwargs)
db.session.commit()
return res
except Exception as e:
db.session.rollback()
raise e
# OR return traceback.format_exc()
finally:
db.session.close()
return inner
and then wrapping my functions with the decorator:
#manage_session
my_funtion(*args, **kwargs):
return "result"
I have a web application, and there are thousands of requests every minutes.
the following is my python code for mongoDB connection:
Tool.py:
globalconnection = None
def getCollection(name,safe=False,readpref=ReadPreference.PRIMARY):
global globalconnection
while globalconnection is None:
try:
if not globalconnection is None:
globalconnection.close()
globalconnection = Connection('mongodb://host:port',replicaSet='mysetname',safe=False,read_preference=ReadPreference.PRIMARY,network_timeout=30,max_pool_size=1024)
except Exception as e:
globalconnection = None
request_context.connection = globalconnection
return request_context.connection["mydb"]["mycoll"]
web.py
#app.route("/test")
def test():
request_collection = getCollection("user")
results = request_collection.find()
for result in results:
#do something...
request_collection.save(result)
request_collection.end_request()
One http request gets the connection through this function,
and the http request calls end_request before the end of the request.
But I found that there are many AutoReconnect errors and over 20000 connections in mongoDB while increasing requests.
Do you have any suggestion?
For auto-reconnection you simply catch the exception, and try to get the connection again:
http://api.mongodb.org/python/current/api/pymongo/errors.html
30 secs timeout sounds too long for, try shorter timeout instead?
Increase max number of connection from mongodb (default:20000)
http://www.mongodb.org/display/DOCS/Connections
I'm developing a Python Service(Class) for accessing Redis Server. I want to know how to check if Redis Server is running or not. And also if somehow I'm not able to connect to it.
Here is a part of my code
import redis
rs = redis.Redis("localhost")
print rs
It prints the following
<redis.client.Redis object at 0x120ba50>
even if my Redis Server is not running.
As I found that my Python Code connects to the Server only when I do a set() or get() with my redis instance.
So I dont want other services using my class to get an Exception saying
redis.exceptions.ConnectionError: Error 111 connecting localhost:6379. Connection refused.
I want to return proper message/Error code. How can I do that??
If you want to test redis connection once at startup, use the ping() command.
from redis import Redis
redis_host = '127.0.0.1'
r = Redis(redis_host, socket_connect_timeout=1) # short timeout for the test
r.ping()
print('connected to redis "{}"'.format(redis_host))
The command ping() checks the connection and if invalid will raise an exception.
Note - the connection may still fail after you perform the test so this is not going to cover up later timeout exceptions.
The official way to check if redis server availability is ping ( http://redis.io/topics/quickstart ).
One solution is to subclass redis and do 2 things:
check for a connection at instantiation
write an exception handler in the case of no connectivity when making requests
As you said, the connection to the Redis Server is only established when you try to execute a command on the server. If you do not want to go head forward without checking that the server is available, you can just send a random query to the server and check the response. Something like :
try:
response = rs.client_list()
except redis.ConnectionError:
#your error handlig code here
There are already good solutions here, but here's my quick and dirty for django_redis which doesn't seem to include a ping function (though I'm using an older version of django and can't use the newest django_redis).
# assuming rs is your redis connection
def is_redis_available():
# ... get redis connection here, or pass it in. up to you.
try:
rs.get(None) # getting None returns None or throws an exception
except (redis.exceptions.ConnectionError,
redis.exceptions.BusyLoadingError):
return False
return True
This seems to work just fine. Note that if redis is restarting and still loading the .rdb file that holds the cache entries on disk, then it will throw the BusyLoadingError, though it's base class is ConnectionError so it's fine to just catch that.
You can also simply except on redis.exceptions.RedisError which is the base class of all redis exceptions.
Another option, depending on your needs, is to create get and set functions that catch the ConnectionError exceptions when setting/getting values. Then you can continue or wait or whatever you need to do (raise a new exception or just throw out a more useful error message).
This might not work well if you absolutely depend on setting/getting the cache values (for my purposes, if cache is offline for whatever we generally have to "keep going") in which case it might make sense to have the exceptions and let the program/script die and get the redis server/service back to a reachable state.
I have also come across a ConnectionRefusedError from the sockets library, when redis was not running, therefore I had to add that to the availability check.
r = redis.Redis(host='localhost',port=6379,db=0)
def is_redis_available(r):
try:
r.ping()
print("Successfully connected to redis")
except (redis.exceptions.ConnectionError, ConnectionRefusedError):
print("Redis connection error!")
return False
return True
if is_redis_available(r):
print("Yay!")
Redis server connection can be checked by executing ping command to the server.
>>> import redis
>>> r = redis.Redis(host="127.0.0.1", port="6379")
>>> r.ping()
True
using the ping method, we can handle reconnection etc. For knowing the reason for error in connecting, exception handling can be used as suggested in other answers.
try:
is_connected = r.ping()
except redis.ConnectionError:
# handle error
Use ping()
from redis import Redis
conn_pool = Redis(redis_host)
# Connection=Redis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>
try:
conn_pool.ping()
print('Successfully connected to redis')
except redis.exceptions.ConnectionError as r_con_error:
print('Redis connection error')
# handle exception