I have a web application, and there are thousands of requests every minutes.
the following is my python code for mongoDB connection:
Tool.py:
globalconnection = None
def getCollection(name,safe=False,readpref=ReadPreference.PRIMARY):
global globalconnection
while globalconnection is None:
try:
if not globalconnection is None:
globalconnection.close()
globalconnection = Connection('mongodb://host:port',replicaSet='mysetname',safe=False,read_preference=ReadPreference.PRIMARY,network_timeout=30,max_pool_size=1024)
except Exception as e:
globalconnection = None
request_context.connection = globalconnection
return request_context.connection["mydb"]["mycoll"]
web.py
#app.route("/test")
def test():
request_collection = getCollection("user")
results = request_collection.find()
for result in results:
#do something...
request_collection.save(result)
request_collection.end_request()
One http request gets the connection through this function,
and the http request calls end_request before the end of the request.
But I found that there are many AutoReconnect errors and over 20000 connections in mongoDB while increasing requests.
Do you have any suggestion?
For auto-reconnection you simply catch the exception, and try to get the connection again:
http://api.mongodb.org/python/current/api/pymongo/errors.html
30 secs timeout sounds too long for, try shorter timeout instead?
Increase max number of connection from mongodb (default:20000)
http://www.mongodb.org/display/DOCS/Connections
Related
I'm developing a basic flask application to receive an input request from the user and insert that into MongoDB Atlas Cluster.
I have a route /save which is of type POST. This endpoint receives the request, makes a new mongo connection, inserts to mongo and finally closes the connection. This approach is slow, with average response latency of 700-800 ms even though I am only trying to insert one document.
Note-: My use case does not make sense to use bulk-insert.
Sample Code
app = Flask(__name__)
app.logger.setLevel(logging.INFO)
DBNAME = 'DBNAME as String'
CONNSTRING = 'CONNECTION as String'
class mongoDB:
def __init__(self):
try:
self.client = MongoClient(CONNSTRING, maxPoolSize=None)
self.database = self.client[DBNAME]
app.logger.info('Mongo Connection Established')
except Exception as e:
app.logger.warning('Mongo Connection could not be established')
app.logger.warning('Error Message: ' + str(e))
def close_connection(self):
try:
self.client.close()
except Exception as e:
app.logger.warning('connection failed to close')
app.logger.warning('Error Message: ' + str(e))
#app.route('/save', methods=['POST'])
def save():
data_info = flask.request.get_json()
try:
db = mongoDB()
image_collection = db.database['DUMMY_COLLECTION']
image_collection.insert_one({'VALUE_ID' : data_info['value1'], 'VALUE_STRING' : data_info['value2']})
app.logger.info('Inserted Successfully')
return {'message': 'Success'}, 200, {'Content-Type': 'application/json'}
except Exception as e:
app.logger.error('Error Adding data to Mongo: ' + str(e))
return {'message': 'Error'}, 500, {'Content-Type': 'application/json'}
finally:
db.close_connection()
app.logger.info('connection closed')
if __name__ == '__main__':
app.run()
However if I establish the Mongo connection at time of application initialization and keep it open and never close the connection the latency drops to 70-80ms.
Could someone please help understand the consequences of keeping an open connection, instead of establishing a new connection with each request? Or is there any method to reduce latency with multiple connections open?
Note-: Keeping multiple connections approach, I tried using writeConcern=0, maxPoolSize=None, and journal=False but all these did not improve the latency much at all.
Any help would be appreciated. Thanks
The MongoClient(CONNSTRING, maxPoolSize=None) is not just a single connection but a connection pool. Meaning that already with that object you can have multiple concurrent requests to MongoDB. By setting maxPoolSize=None you make them limitless (which can have some implications under heavy load).
It is an antipattern to create a connection pool per request (as you realized by the high latency) the reason for that is that you need to pay each time the cost to create the connection pool and the handshake to the database.
The best way is to initiate one on startup and then maintain it. Meaning that you should handle all the exceptions that might arise in case of DB or network failures. However, I assume that most things are already handled by MongoClient already.
My Stack is uwsgi with gevents. I am trying to wrap my api end points with a decorator to push all request data (url, method, body and response) to a kafka topic, But its not working. My theory is because I am using gevents, and I am trying to run these in async mode, the async thread which actually pushes to kafka, is not able to run with gevents. And If I try to make the method sync, Then also it does not work, it dies in the produce worker, i.e. after produce the call never returns. Although both the methods work good on python shell and if I run uwsgi on threads.
Follows the sample code:
1. with kafka-python (async)
try:
kafka_producer = KafkaProducer(bootstrap_servers=KAFKAHOST.split(','))
except NoBrokersAvailable:
logger.info(u'Kafka Host not available: {}'.format(KAFKAHOST))
kafka_producer = None
def send_message_to_kafka(topic, key, message):
"""
:param topic: topic name
:param key: key to decide partition
:param message: json serializable object to send
:return:
"""
if not kafka_producer:
logger.info(u'Kafka Host not available: {}'.format(KAFKAHOST))
return
data = json.dumps(message)
try:
start = time.time()
kafka_producer.send(topic, key=str(key), value=data)
logger.info(u'Time take to push to Kafka: {}'.format(time.time() - start))
except KafkaTimeoutError as e:
logger.info(u'Message not sent: {}'.format(KAFKAHOST))
logger.info(e)
pass
except Exception as e:
logger.info(u'Message not sent: {}'.format(KAFKAHOST))
logger.exception(e)
pass
with py-kafka (sync):
try:
client = KafkaClient(hosts=KAFKAHOST)
except Exception as e:
logger.info(u'Kafka Host Not Found: {}'.format(KAFKAHOST))
client = None
def send_message_to_kafka(topic, key, message):
"""
:param topic: topic name
:param key: key to decide partition
:param message: json serializable object to send
:return:
"""
if not client:
logger.info(u'Kafka Host is None')
return
data = json.dumps(message)
try:
start = time.time()
topic = client.topics[topic]
with topic.get_sync_producer() as producer:
producer.produce(data, partition_key='{}'.format(key))
logger.info(u'Time take to push to Kafka: {}'.format(time.time() - start))
except Exception as e:
logger.exception(e)
pass
I have more experience with pykafka so I can answer that section. pykafka uses a pluggable thread handler and gevent support is built in. You need to instantiate the KafkaClient with use_greenlets=True. Docs here
Other thoughts on your approach. Creating a new topic object and producer for every message is extremely expensive. Its better to create once and reuse.
# setup once
client = KafkaClient(hosts=KAFKAHOST, use_greenlets=True)
topic = client.topics[topic]
producer = topic.get_sync_producer()
def send_message_to_kafka(producer, key, message):
"""
:param producer: pykafka producer
:param key: key to decide partition
:param message: json serializable object to send
:return:
"""
data = json.dumps(message)
try:
start = time.time()
producer.produce(data, partition_key='{}'.format(key))
logger.info(u'Time take to push to Kafka: {}'.format(time.time() - start))
except Exception as e:
logger.exception(e)
pass # for at least once delivery you will need to catch network errors and retry.
Finally, kafka gets all of it's speed from batching and compression. Using the sync producer keeps the client from exploiting these features. It will work, but is slower and uses more space. Some applications require sync but it might make sense to rethink your application to batch messages if you are hitting performance bottlenecks.
I have a typical Pyramid+SQLAlchemy+Postgres app. In stress testing or during moments of exceptional load and with low max_connections setting in PG it might happen that OperationalException is raised:
OperationalError: (psycopg2.OperationalError) FATAL: sorry, too many clients already
Now, obviously I do not want to do this everywhere:
try:
DBSession.query(Item)...
except OperationalError as e:
log.error(...)
Is there some way of catching this exception "globally" to be properly handled?
My app uses ZopeTransactionExtension in typical Pyramid manner:
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
I managed to develop a tween that can do this (example):
def catch_pg_exc_tween_factory(handler, registry):
def catch_pg_exc_tween_clos(request):
response = None
try:
response = handler(request)
except Exception as e:
log.error('\n\n\n +++ problem: %s', e)
return response
return catch_pg_exc_tween_clos
The strange thing is that nothing but explicit tween ordering in development.ini works (no amount of over= or under= tuning of config.add_tween call seems to have worked):
pyramid.tweens = pyramid_debugtoolbar.toolbar_tween_factory
pyramid.tweens.excview_tween_factory
pyramid_tm.tm_tween_factory
mypkg.util.tweens.catch_pg_exc_tween_factory
I'm developing a Python Service(Class) for accessing Redis Server. I want to know how to check if Redis Server is running or not. And also if somehow I'm not able to connect to it.
Here is a part of my code
import redis
rs = redis.Redis("localhost")
print rs
It prints the following
<redis.client.Redis object at 0x120ba50>
even if my Redis Server is not running.
As I found that my Python Code connects to the Server only when I do a set() or get() with my redis instance.
So I dont want other services using my class to get an Exception saying
redis.exceptions.ConnectionError: Error 111 connecting localhost:6379. Connection refused.
I want to return proper message/Error code. How can I do that??
If you want to test redis connection once at startup, use the ping() command.
from redis import Redis
redis_host = '127.0.0.1'
r = Redis(redis_host, socket_connect_timeout=1) # short timeout for the test
r.ping()
print('connected to redis "{}"'.format(redis_host))
The command ping() checks the connection and if invalid will raise an exception.
Note - the connection may still fail after you perform the test so this is not going to cover up later timeout exceptions.
The official way to check if redis server availability is ping ( http://redis.io/topics/quickstart ).
One solution is to subclass redis and do 2 things:
check for a connection at instantiation
write an exception handler in the case of no connectivity when making requests
As you said, the connection to the Redis Server is only established when you try to execute a command on the server. If you do not want to go head forward without checking that the server is available, you can just send a random query to the server and check the response. Something like :
try:
response = rs.client_list()
except redis.ConnectionError:
#your error handlig code here
There are already good solutions here, but here's my quick and dirty for django_redis which doesn't seem to include a ping function (though I'm using an older version of django and can't use the newest django_redis).
# assuming rs is your redis connection
def is_redis_available():
# ... get redis connection here, or pass it in. up to you.
try:
rs.get(None) # getting None returns None or throws an exception
except (redis.exceptions.ConnectionError,
redis.exceptions.BusyLoadingError):
return False
return True
This seems to work just fine. Note that if redis is restarting and still loading the .rdb file that holds the cache entries on disk, then it will throw the BusyLoadingError, though it's base class is ConnectionError so it's fine to just catch that.
You can also simply except on redis.exceptions.RedisError which is the base class of all redis exceptions.
Another option, depending on your needs, is to create get and set functions that catch the ConnectionError exceptions when setting/getting values. Then you can continue or wait or whatever you need to do (raise a new exception or just throw out a more useful error message).
This might not work well if you absolutely depend on setting/getting the cache values (for my purposes, if cache is offline for whatever we generally have to "keep going") in which case it might make sense to have the exceptions and let the program/script die and get the redis server/service back to a reachable state.
I have also come across a ConnectionRefusedError from the sockets library, when redis was not running, therefore I had to add that to the availability check.
r = redis.Redis(host='localhost',port=6379,db=0)
def is_redis_available(r):
try:
r.ping()
print("Successfully connected to redis")
except (redis.exceptions.ConnectionError, ConnectionRefusedError):
print("Redis connection error!")
return False
return True
if is_redis_available(r):
print("Yay!")
Redis server connection can be checked by executing ping command to the server.
>>> import redis
>>> r = redis.Redis(host="127.0.0.1", port="6379")
>>> r.ping()
True
using the ping method, we can handle reconnection etc. For knowing the reason for error in connecting, exception handling can be used as suggested in other answers.
try:
is_connected = r.ping()
except redis.ConnectionError:
# handle error
Use ping()
from redis import Redis
conn_pool = Redis(redis_host)
# Connection=Redis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>
try:
conn_pool.ping()
print('Successfully connected to redis')
except redis.exceptions.ConnectionError as r_con_error:
print('Redis connection error')
# handle exception
We are using celery to get flights data from different travel
agencies, every request takes ~20-30 seconds(most agencies require
request sequence - authorize, send request, poll for results).
Normal
celery task looks like this:
from eventlet.green import urllib2, time
def get_results(attr, **kwargs):
search, provider, minprice = attr
data = XXX # prepared data
host = urljoin(MAIN_URL, "RPCService/Flights_SearchStart")
req = urllib2.Request(host, data, {'Content-Type': 'text/xml'})
try:
response_stream = urllib2.urlopen(req)
except urllib2.URLError as e:
return [search, None]
response = response_stream.read()
rsp_host = urljoin(MAIN_URL, "RPCService/FlightSearchResults_Get")
rsp_req = urllib2.Request(rsp_host, response, {'Content-Type':
'text/xml'})
ready = False
sleeptime = 1
rsp_response = ''
while not ready:
time.sleep(sleeptime)
try:
rsp_response_stream = urllib2.urlopen(rsp_req)
except urllib2.URLError as e:
log.error('go2see: results fetch failed for %s IOError %s'%
(search.id, str(e)))
else:
rsp_response = rsp_response_stream.read()
try:
rsp = parseString(rsp_response)
except ExpatError as e:
return [search, None]
else:
ready = rsp.getElementsByTagName('SearchResultEx')
[0].getElementsByTagName('IsReady')[0].firstChild.data
ready = (ready == 'true')
sleeptime += 1
if sleeptime > 10:
return [search, None]
hash = "%032x" % random.getrandbits(128)
open(RESULT_TMP_FOLDER+hash, 'w+').write(rsp_response)
# call to parser
parse_agent_results.apply_async(queue='parsers', args=[__name__,
search, provider, hash])
This tasks are run in eventlet pool with concurency 300,
prefetch_multiplier = 1, broker_limit = 300
When ~100-200 task are fetched from queue - CPU usage raises up to 100%
( whole CPU core is used) and task fetching from queue is performed
with delays.
Could you please point on possible issues - blocking
operations( eventlet ALARM DETECTOR gives no exceptions ), wrong
architecture or whatever.
A problem occurs if you fire 200 requests to a server, responses could be delayed and therefore urllib.urlopen will hang.
Another thing i noticed: If an URLError is raised, the program stays in the while loop until sleeptime is greater than 10. So an URLError error will let this script sleep for 55 sec (1+2+3.. etc)
Sorry for late response.
Thing i would try first in such situation is to turn off Eventlet completely in both Celery and your code, use process or OS thread model. 300 threads or even processes is not that much load for OS scheduler (although you may lack memory to run many processes). So i would try it and see if CPU load drops dramatically. If it does not, then problem is in your code and Eventlet can't magically fix it. If it does drop, however, we would need to investigate the issue closer.
If bug still persists, please, report it via any of these ways:
https://bitbucket.org/which_linden/eventlet/issues/new
https://github.com/eventlet/eventlet/issues/new
email to eventletdev#lists.secondlife.com