We Keep Coding

Why am I getting pymongo.errors.AutoReconnect: connection pool paused

I'm getting this error of autoreconnect, and there are around 100 connections in the logs during this call to .objects. Here is the document:
class NotificationDoc(Document):
patient_id = StringField(max_length=32)
type = StringField(max_length=32)
sms_sent = BooleanField(default=False)
email_sent = BooleanField(default=False)
sending_time = DateTimeField(default=datetime.utcnow)
def __unicode__(self):
return "{}_{}_{}".format(self.patient_id, self.type, self.sending_time.strftime('%Y-%m-%d'))
and this is the query call that causes the issue:
docs = NotificationDoc.objects(sending_time__lte=from_date)
This is the complete stacktrace. How can I understand what is going on?
pymongo.errors.AutoReconnect: connection pool paused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/app/src/reminder/configurations/app/worker.py", line 106, in schedule_recall
schedule_recall_use_case.execute()
File "/app/src/reminder/domain/notification/recall/use_cases/schedule_recall.py", line 24, in execute
notifications_sent = self.notifications_provider.find_recalled_notifications_for_date(no_recalls_before_date)
File "/app/src/reminder/data_providers/database/odm/repositories.py", line 42, in find_recalled_notifications_for_date
if NotificationDoc.objects(sending_time__lte=from_date).count() > 0:
File "/usr/local/lib/python3.8/site-packages/mongoengine/queryset/queryset.py", line 144, in count
return super().count(with_limit_and_skip)
File "/usr/local/lib/python3.8/site-packages/mongoengine/queryset/base.py", line 423, in count
count = count_documents(
File "/usr/local/lib/python3.8/site-packages/mongoengine/pymongo_support.py", line 38, in count_documents
return collection.count_documents(filter=filter, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pymongo/collection.py", line 1502, in count_documents
return self.__database.client._retryable_read(
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1307, in _retryable_read
with self._secondaryok_for_server(read_pref, server, session) as (
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1162, in _secondaryok_for_server
with self._get_socket(server, session) as sock_info:
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1099, in _get_socket
with server.get_socket(
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1371, in get_socket
sock_info = self._get_socket(all_credentials)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1436, in _get_socket
self._raise_if_not_ready(emit_event=True)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1407, in _raise_if_not_ready
_raise_connection_failure(
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 250, in _raise_connection_failure
raise AutoReconnect(msg) from error
pymongo.errors.AutoReconnect: mongo:27017: connection pool paused

turned out to be celery is leaving connections open, so here is the solution
from mongoengine import disconnect
from celery.signals import task_prerun
#task_prerun.connect
def on_task_init(*args, **kwargs):
disconnect(alias='default')
connect(db, host=host, port=port, maxPoolSize=400, minPoolSize=200, alias='default')

This could be related to the fact that PyMongo is not fork-safe. If you're using a process pool in any way, which includes server software like uWSGI and even some configurations of popular ASGI servers.
Every time you run a query in PyMongo, that MongoClient object becomes fork-unsafe. A MongoClient object that has never run any queries is fork-safe. Created Database and Collection objects will reference back to their parent MongoClient without checking for existence.
I do see you are using MongoEngine, which uses PyMongo underneath. I don't know the semantics of that particular library but I would assume they are the same.
In short: you must re-create your connections as part of your forking process.

pip install -U pymongo; is already fix; see this https://github.com/mongodb/mongo-python-driver/pull/944

How to force reading from replicaset secondary member with PyMongo?

I am trying to connect to a MongoDB replicaset with PyMongo and to manually balance the reading load with ReadPreference.
The problem is that whatever I try with MongoClient, it always reads from PRIMARY.
I am using Python 2.7.6 and PyMongo 2.6.3 with mongodb-10gen 2.4.14 (all for legacy reasons) on Linux Mint 17.2.
My connection sequence looks like this (without the print, and with massive request on some collection of some database from the connection):
>>> from pymongo import MongoClient, ReadPreference
>>> HOST = "10.0.0.51"
>>> PORT = 49029
>>> print MongoClient(host=HOST, port=PORT, replicaset="rs02", readPreference=ReadPreference.SECONDARY)
MongoClient([u'XXXX-MNGO03664:49029', u'XXXX-MNGO03663:49029'])
This one would be the right way to go with PyMongo > 3, from what I have read. Unfortunately I am stuck with PyMongo 2.6.3, and I can only assume that this is why it doesn't read from secondary.
After a bit of digging, I found about ReplicaSetConnection (deprecated since PyMongo 2.4) and MongoReplicaSetClient (see e.g. pymongo replication secondary readreference not work and pymongo: Advantage of using MongoReplicaSetClient?), but it also doesn't seem to work for me, for different reasons though.
>>> print MongoReplicaSetClient(host=HOST, port=PORT, replicaset="rs02", readPreference=ReadPreference.SECONDARY)
MongoReplicaSetClient([])
The client doesn't seem to be able to see the members of the replicaset…
And of course when I start reading from this connection, it doesn't work.
>>> myConn = MongoReplicaSetClient(host=HOST, port=PORT, replicaset="rs02", readPreference=ReadPreference.SECONDARY)
>>> print myConn.someDB.someCollection.count()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/pymongo/collection.py", line 759, in count
return self.find().count()
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 640, in count
**command)
File "/usr/local/lib/python2.7/dist-packages/pymongo/database.py", line 391, in command
result = self["$cmd"].find_one(command, **extra_opts)
File "/usr/local/lib/python2.7/dist-packages/pymongo/collection.py", line 604, in find_one
for result in self.find(spec_or_id, *args, **kwargs).limit(-1):
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 904, in next
if len(self.__data) or self._refresh():
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 848, in _refresh
self.__uuid_subtype))
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 782, in __send_message
res = client._send_message_with_response(message, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_replica_set_client.py", line 1631, in _send_message_with_response
raise AutoReconnect(msg, errors)
pymongo.errors.AutoReconnect: No replica set secondary available for query with ReadPreference SECONDARY
Note that the same error appears when I try to read with ReadPreference.PRIMARY.
The weird thing here is that if I change the name of the replicaset to connect to, the client spots that it doesn't exist :
>>> print MongoReplicaSetClient(host=HOST, port=PORT, replicaset="rs42", readPreference=ReadPreference.SECONDARY)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_replica_set_client.py", line 742, in __init__
self.refresh()
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_replica_set_client.py", line 1135, in refresh
% (host, port, self.__name))
pymongo.errors.ConfigurationError: 10.0.0.51:49029 is not a member of replica set rs42
So I assume that in normal cases, it has a way to see that there is a replicaset here, and who are its members.

Flask_MongoEngine doesn't work on Webfaction?

I'm currently trying to deploy a website built with the Python Flask framework to Webfaction. I've deployed a MongoDB to which I can succesfully connect from the MongoDB interactive interpreter:
[celli#web498 ~]$ mongo localhost:18209/admin
MongoDB shell version: 3.0.2
connecting to: localhost:18209/admin
> db.auth("awesomeusername", "anawesomepassword")
1
When starting the website, it can successfully connect using these credentials in my flask config.py file:
MONGODB_SETTINGS = {
'db': 'tc',
'port': 18209,
'username': 'awesomeusername',
'password': 'anawesomepassword',
}
But when I visit a page which does a query I get the error below. This code works perfectly well on another server, so the mistake lies somewhere in either the setup of my MongoDB instance, or in the connection to it. I also find it strange that setting up the connection works fine, but querying it creates this Authentication failed error.
Any idea how I can fix this? All tips are welcome!
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
return func(*args, **kwargs)
File "/home/celli/webapps/tc/tc/app/views/webviews.py", line 220, in myTickets
userDocs = UserDocument.objects(id__in=[ticket.doc_id for ticket in userTickets])
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/mongoengine/queryset/manager.py", line 37, in __get__
queryset = queryset_class(owner, owner._get_collection())
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/mongoengine/document.py", line 168, in _get_collection
db = cls._get_db()
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/mongoengine/document.py", line 162, in _get_db
return get_db(cls._meta.get("db_alias", DEFAULT_CONNECTION_NAME))
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/mongoengine/connection.py", line 143, in get_db
source=conn_settings['authentication_source'])
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/pymongo/database.py", line 978, in authenticate
self.connection._cache_credentials(self.name, credentials)
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/pymongo/mongo_client.py", line 467, in _cache_credentials
auth.authenticate(credentials, sock_info, self.__simple_command)
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/pymongo/auth.py", line 475, in authenticate
auth_func(credentials[1:], sock_info, cmd_func)
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/pymongo/auth.py", line 450, in _authenticate_default
return _authenticate_scram_sha1(credentials, sock_info, cmd_func)
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/pymongo/auth.py", line 238, in _authenticate_scram_sha1
sasl_start, sasl_continue)
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/pymongo/auth.py", line 185, in _scram_sha1_conversation
res, _ = cmd_func(sock_info, source, sasl_start)
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/pymongo/mongo_client.py", line 703, in __simple_command
helpers._check_command_response(response, None, msg)
File "/home/celli/webapps/tc/venv/lib/python2.7/site-packages/pymongo/helpers.py", line 182, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
OperationFailure: command SON([('saslStart', 1), ('mechanism', 'SCRAM-SHA-1'), ('autoAuthorize', 1), ('payload', Binary('n,,n=awesomeusername,r=NjAxNTEwMjc5MzAy', 0))]) on namespace tc.$cmd failed: Authentication failed.

MongoDB's users are per-database. In your example you show that you connect to the admin database, but the error message is simply telling you that your user is trying to authenticate with the tc database. Probably your user doesn't have a role on tc.
(The other possibility is that tc is not the database you want to connect to; you need to specify the correct database name in your connection params/string).

Python xmlrpclib raise IncompleteRead Error

I have a simple xmlrpc server which is written along the line of
server = SimpleXMLRPCServer(('127.0.0.1', 8000),allow_none=True)
server.register_function(self.fetch_buyer_data,'"fetch_buyer_data")
...
...
server.serve_forever()
This is not the complete code, but you get the idea (hopefully) !
In the same server script, I have a function that reads off the contents of an sqlite3 database and return all the data. Something like this:
def fetch_buyer_data(self, projectname):
conn = sqlite3.connect(...)
# read data from sqlite3 database and save it into list
conn.close()
return datalist
And I use this following code from client to access the above function.
proxy = xmlrpclib.ServerProxy("http://%s:%s/" %(hostip,hostport),allow_none=True)
data = proxy.fetch_buyer_data(SELECTED_PROJECT)
It's all good until the data in sqlite3 database get larger (not very large but something like a few Megabytes!), I keep getting the following error message!
Traceback (most recent call last):
File "C:\Custom\src\Client\client.py", line 178, in show_user_page
userpage = UserPage()
File "C:\Custom\src\Client\client.py", line 2371, in __init__
self.update_buyer_table()
File "C:\Custom\src\Client\client.py", line 6106, in update_buyer_table
data = proxy.fetch_buyer_data(SELECTED_PROJECT)
File "C:\Python27\Lib\xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
File "C:\Python27\Lib\xmlrpclib.py", line 1578, in __request
verbose=self.__verbose
File "C:\Python27\Lib\xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "C:\Python27\Lib\xmlrpclib.py", line 1297, in single_request
return self.parse_response(response)
File "C:\Python27\Lib\xmlrpclib.py", line 1453, in parse_response
stream = GzipDecodedResponse(response)
File "C:\Python27\Lib\xmlrpclib.py", line 1204, in __init__
self.stringio = StringIO.StringIO(response.read())
File "C:\Python27\Lib\httplib.py", line 548, in read
s = self._safe_read(self.length)
File "C:\Python27\Lib\httplib.py", line 649, in _safe_read
raise IncompleteRead(''.join(s), amt)
httplib.IncompleteRead: IncompleteRead(8031 bytes read, 1732 more expected)
NOTE: I have checked that the rest of the registered functions on server are working. So,I can rule out the connection problems(ip,port etc).
What is casuing this error message? How do I overcome the problem?
I am using python 2.7 on windows xp sp3.
UPDATE 1:
I found out that this doesn't entirely depends on the size of the database. Sometimes it gives me error message, sometimes it doesn't. Could anyone tell me what is casuing this IncompleteRead Problem?

Using SQLAlchemy on App Engine development server

I have seen some questions about using SQLAlchemy on App Engine to connect to Google Cloud SQL. But I'm not sure if it is possible to develop using a local MySQL database and the existing SQLAlchemy dialect. On my first attempt, I added SQLAlchemy 0.8.0 to the app and defined a schema:
from sqlalchemy import create_engine, Column, Integer, Table
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
foo_table = Table('foo', Base.metadata,
Column('id', Integer, primary_key=True, autoincrement=True),
)
And when I tried to create the tables on the development server using:
url = 'mysql+gaerdbms:///%s?instance=%s' % ('database_name', 'instance_name')
engine = create_engine(url)
Base.metadata.create_all(engine)
...I got an error DBAPIError: (ImportError) No module named pwd None None, which means that SQLAlchemy is importing a module that is blacklisted by the development server.
Am I doing something wrong? Or, if not, what should I do to use SQLAlchemy on the development server? Or maybe the first question is: Can I use the SQLAlchemy's gaerdbms dialect to develop in a local MySql database using the dev server?
Edit: this error doesn't happen only when trying to create tables. I created the tables manually and tried to query them, and the same error occurs.
The full traceback is:
Traceback (most recent call last):
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "[...]/webapp/admin.py", line 12, in get
db.Base.metadata.create_all(engine)
File "[...]/webapp/sqlalchemy/schema.py", line 2784, in create_all
tables=tables)
File "[...]/webapp/sqlalchemy/engine/base.py", line 1486, in _run_visitor
with self._optional_conn_ctx_manager(connection) as conn:
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "[...]/webapp/sqlalchemy/engine/base.py", line 1479, in _optional_conn_ctx_manager
with self.contextual_connect() as conn:
File "[...]/webapp/sqlalchemy/engine/base.py", line 1669, in contextual_connect
self.pool.connect(),
File "[...]/webapp/sqlalchemy/pool.py", line 272, in connect
return _ConnectionFairy(self).checkout()
File "[...]/webapp/sqlalchemy/pool.py", line 425, in __init__
rec = self._connection_record = pool._do_get()
File "[...]/webapp/sqlalchemy/pool.py", line 855, in _do_get
return self._create_connection()
File "[...]/webapp/sqlalchemy/pool.py", line 225, in _create_connection
return _ConnectionRecord(self)
File "[...]/webapp/sqlalchemy/pool.py", line 318, in __init__
self.connection = self.__connect()
File "[...]/webapp/sqlalchemy/pool.py", line 368, in __connect
connection = self.__pool._creator()
File "[...]/webapp/sqlalchemy/engine/strategies.py", line 80, in connect
return dialect.connect(*cargs, **cparams)
File "[...]/webapp/sqlalchemy/engine/default.py", line 279, in connect
return self.dbapi.connect(*cargs, **cparams)
File "[...]/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 183, in __init__
super(GoogleApiConnection, self).__init__(*args, **kwargs)
File "[...]/google_appengine/google/storage/speckle/python/api/rdbms.py", line 810, in __init__
self.OpenConnection()
File "[...]/google_appengine/google/storage/speckle/python/api/rdbms.py", line 832, in OpenConnection
self.SetupClient()
File "[...]/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 193, in SetupClient
self._client = RdbmsGoogleApiClient(**kwargs)
File "[...]/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 106, in __init__
rdbms.OAUTH_CREDENTIALS_PATH)
File "/usr/lib/python2.7/posixpath.py", line 259, in expanduser
import pwd
File "[...]/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 822, in load_module
raise ImportError('No module named %s' % fullname)
DBAPIError: (ImportError) No module named pwd None None

I found a workaround. As it is, SQLAlchemy's gaerdbms dialect can't connect to a local database. But with the dialect below it can. Folow the instructions from this answer but use this dialect instead:
# mysql/gaerdbms.py
# Copyright (C) 2005-2013 the SQLAlchemy authors and contributors <see AUTHORS file>
#
# This module is part of SQLAlchemy and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
"""
.. dialect:: mysql+gaerdbms
:name: Google Cloud SQL
:dbapi: rdbms
:connectstring: mysql+gaerdbms:///<dbname>?instance=<instancename>
:url: https://developers.google.com/appengine/docs/python/cloud-sql/developers-guide
This dialect is based primarily on the :mod:`.mysql.mysqldb` dialect with minimal
changes.
.. versionadded:: 0.7.8
Pooling
-------
Google App Engine connections appear to be randomly recycled,
so the dialect does not pool connections. The :class:`.NullPool`
implementation is installed within the :class:`.Engine` by
default.
"""
import os
import re
from sqlalchemy.dialects.mysql.mysqldb import MySQLDialect_mysqldb
from sqlalchemy.pool import NullPool
class MySQLDialect_gaerdbms(MySQLDialect_mysqldb):
#classmethod
def dbapi(cls):
# from django:
# http://code.google.com/p/googleappengine/source/
# browse/trunk/python/google/storage/speckle/
# python/django/backend/base.py#118
# see also [ticket:2649]
# see also https://stackoverflow.com/q/14224679/34549
if is_production():
# Production mode.
from google.storage.speckle.python.api import rdbms_apiproxy
return rdbms_apiproxy
elif is_remote_mode():
# Development mode with remote database.
from google.storage.speckle.python.api import rdbms_googleapi
return rdbms_googleapi
else:
# Development mode with local database.
from google.appengine.api import rdbms_mysqldb
return rdbms_mysqldb
#classmethod
def get_pool_class(cls, url):
# Cloud SQL connections die at any moment
return NullPool
def create_connect_args(self, url):
opts = url.translate_connect_args()
if is_production() or is_remote_mode():
# 'dsn' and 'instance' are because we are skipping
# the traditional google.api.rdbms wrapper.
# they are not needed in local mode; 'dns' even causes an error.
opts['dsn'] = ''
opts['instance'] = url.query['instance']
return [], opts
def _extract_error_code(self, exception):
match = re.compile(r"^(\d+):|^\((\d+),").match(str(exception))
# The rdbms api will wrap then re-raise some types of errors
# making this regex return no matches.
code = match.group(1) or match.group(2) if match else None
if code:
return int(code)
dialect = MySQLDialect_gaerdbms
def is_production():
return os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine')
def is_remote_mode():
return os.getenv('SETTINGS_MODE') == 'prod'
This dialect uses a local database by default when running on the development server. To use remote access to Google Cloud SQL during development, a variable must be set in the environment, following the pattern used by Django:
os.environ['SETTINGS_MODE'] = 'prod'