We Keep Coding

Why am I getting pymongo.errors.AutoReconnect: connection pool paused

I'm getting this error of autoreconnect, and there are around 100 connections in the logs during this call to .objects. Here is the document:
class NotificationDoc(Document):
patient_id = StringField(max_length=32)
type = StringField(max_length=32)
sms_sent = BooleanField(default=False)
email_sent = BooleanField(default=False)
sending_time = DateTimeField(default=datetime.utcnow)
def __unicode__(self):
return "{}_{}_{}".format(self.patient_id, self.type, self.sending_time.strftime('%Y-%m-%d'))
and this is the query call that causes the issue:
docs = NotificationDoc.objects(sending_time__lte=from_date)
This is the complete stacktrace. How can I understand what is going on?
pymongo.errors.AutoReconnect: connection pool paused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/app/src/reminder/configurations/app/worker.py", line 106, in schedule_recall
schedule_recall_use_case.execute()
File "/app/src/reminder/domain/notification/recall/use_cases/schedule_recall.py", line 24, in execute
notifications_sent = self.notifications_provider.find_recalled_notifications_for_date(no_recalls_before_date)
File "/app/src/reminder/data_providers/database/odm/repositories.py", line 42, in find_recalled_notifications_for_date
if NotificationDoc.objects(sending_time__lte=from_date).count() > 0:
File "/usr/local/lib/python3.8/site-packages/mongoengine/queryset/queryset.py", line 144, in count
return super().count(with_limit_and_skip)
File "/usr/local/lib/python3.8/site-packages/mongoengine/queryset/base.py", line 423, in count
count = count_documents(
File "/usr/local/lib/python3.8/site-packages/mongoengine/pymongo_support.py", line 38, in count_documents
return collection.count_documents(filter=filter, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pymongo/collection.py", line 1502, in count_documents
return self.__database.client._retryable_read(
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1307, in _retryable_read
with self._secondaryok_for_server(read_pref, server, session) as (
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1162, in _secondaryok_for_server
with self._get_socket(server, session) as sock_info:
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1099, in _get_socket
with server.get_socket(
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1371, in get_socket
sock_info = self._get_socket(all_credentials)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1436, in _get_socket
self._raise_if_not_ready(emit_event=True)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1407, in _raise_if_not_ready
_raise_connection_failure(
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 250, in _raise_connection_failure
raise AutoReconnect(msg) from error
pymongo.errors.AutoReconnect: mongo:27017: connection pool paused

turned out to be celery is leaving connections open, so here is the solution
from mongoengine import disconnect
from celery.signals import task_prerun
#task_prerun.connect
def on_task_init(*args, **kwargs):
disconnect(alias='default')
connect(db, host=host, port=port, maxPoolSize=400, minPoolSize=200, alias='default')

This could be related to the fact that PyMongo is not fork-safe. If you're using a process pool in any way, which includes server software like uWSGI and even some configurations of popular ASGI servers.
Every time you run a query in PyMongo, that MongoClient object becomes fork-unsafe. A MongoClient object that has never run any queries is fork-safe. Created Database and Collection objects will reference back to their parent MongoClient without checking for existence.
I do see you are using MongoEngine, which uses PyMongo underneath. I don't know the semantics of that particular library but I would assume they are the same.
In short: you must re-create your connections as part of your forking process.

pip install -U pymongo; is already fix; see this https://github.com/mongodb/mongo-python-driver/pull/944

How to fix LDAPSocketReceiveError: error receiving data: The read operation timed out while using LDAP_MATCHING_RULE_IN_CHAIN/1.2.840.113556.1.4.1941?

I am trying to fetch user's group's recursively.
For eg: User A is part of G1 and G1 is part of G2, I should get G1 and G2 as the output for A.
My code is as below.
query = "(&(objectClass=group)(member:1.2.840.113556.1.4.1941:=CN=nn\, rr,OU=tt,OU=uu,OU=mm,OU=ss,OU=bb,OU=ss,OU=ll,DC=aa,DC=ss,DC=com))"
tls = ldap3.Tls(validate=ssl.CERT_NONE, version=ssl.PROTOCOL_TLS)
server = ldap3.Server(<<domaincontroller>>, get_info=ldap3.ALL, mode=ldap3.IP_V4_PREFERRED, tls=tls, use_ssl=True)
with ldap3.Connection(server=server,authentication=ldap3.NTLM,auto_bind=True,password=domain.password,read_only=True,receive_timeout=self.config.ldap_timeout,user=domain.user) as ldap_connection:
search_parameters = {'search_base': domain.base_dn,'search_filter': ldap_query_find_all_groups_with_our_user_as_member,'attributes': ['*']}
ldap_connection.search(**search_parameters)
print(ldap_connection.entries)
It is working fine without the :1.2.840.113556.1.4.1941:, but with it, I am getting error as below.
Note:
There are chances of duplicacy also, where Parent has a group as its child and the Child has same group as its child again.
Also, although I don't know exactly there could be possibilities where 2 groups are a part of each other and cause a deadlock. I am not sure if LDAP_MATCHING_RULE_IN_CHAIN handles such situations.
Traceback (most recent call last):
File "/opt/myapp/venv/lib/python3.6/site-packages/ldap3/strategy/sync.py", line 82, in receiving
data = self.connection.socket.recv(self.socket_size)
File "/usr/local/lib/python3.6/ssl.py", line 994, in recv
return self.read(buflen)
File "/usr/local/lib/python3.6/ssl.py", line 871, in read
return self._sslobj.read(len, buffer)
File "/usr/local/lib/python3.6/ssl.py", line 633, in read
v = self._sslobj.read(len)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/myapp/venv/lib/python3.6/site-packages/app/core.py", line 283, in smita
ldap_connection.search(**search_parameters)
File "/opt/myapp/venv/lib/python3.6/site-packages/ldap3/core/connection.py", line 789, in search
response = self.post_send_search(self.send('searchRequest', request, controls))
File "/opt/myapp/venv/lib/python3.6/site-packages/ldap3/strategy/sync.py", line 139, in post_send_search
responses, result = self.get_response(message_id)
File "/opt/myapp/venv/lib/python3.6/site-packages/ldap3/strategy/base.py", line 324, in get_response
responses = self._get_response(message_id)
File "/opt/myapp/venv/lib/python3.6/site-packages/ldap3/strategy/sync.py", line 157, in _get_response
responses = self.receiving()
File "/opt/myapp/venv/lib/python3.6/site-packages/ldap3/strategy/sync.py", line 92, in receiving
raise communication_exception_factory(LDAPSocketReceiveError, type(e)(str(e)))(self.connection.last_error)
ldap3.core.exceptions.LDAPSocketReceiveError: error receiving data: The read operation timed out

A timeout, in general, means that the server did not respond in the expected amount of time, so the client gave up waiting. This can be a time-consuming query. Try increasing receive_timeout to allow more time for it to return the results.

Python xmlrpclib raise IncompleteRead Error

I have a simple xmlrpc server which is written along the line of
server = SimpleXMLRPCServer(('127.0.0.1', 8000),allow_none=True)
server.register_function(self.fetch_buyer_data,'"fetch_buyer_data")
...
...
server.serve_forever()
This is not the complete code, but you get the idea (hopefully) !
In the same server script, I have a function that reads off the contents of an sqlite3 database and return all the data. Something like this:
def fetch_buyer_data(self, projectname):
conn = sqlite3.connect(...)
# read data from sqlite3 database and save it into list
conn.close()
return datalist
And I use this following code from client to access the above function.
proxy = xmlrpclib.ServerProxy("http://%s:%s/" %(hostip,hostport),allow_none=True)
data = proxy.fetch_buyer_data(SELECTED_PROJECT)
It's all good until the data in sqlite3 database get larger (not very large but something like a few Megabytes!), I keep getting the following error message!
Traceback (most recent call last):
File "C:\Custom\src\Client\client.py", line 178, in show_user_page
userpage = UserPage()
File "C:\Custom\src\Client\client.py", line 2371, in __init__
self.update_buyer_table()
File "C:\Custom\src\Client\client.py", line 6106, in update_buyer_table
data = proxy.fetch_buyer_data(SELECTED_PROJECT)
File "C:\Python27\Lib\xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
File "C:\Python27\Lib\xmlrpclib.py", line 1578, in __request
verbose=self.__verbose
File "C:\Python27\Lib\xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "C:\Python27\Lib\xmlrpclib.py", line 1297, in single_request
return self.parse_response(response)
File "C:\Python27\Lib\xmlrpclib.py", line 1453, in parse_response
stream = GzipDecodedResponse(response)
File "C:\Python27\Lib\xmlrpclib.py", line 1204, in __init__
self.stringio = StringIO.StringIO(response.read())
File "C:\Python27\Lib\httplib.py", line 548, in read
s = self._safe_read(self.length)
File "C:\Python27\Lib\httplib.py", line 649, in _safe_read
raise IncompleteRead(''.join(s), amt)
httplib.IncompleteRead: IncompleteRead(8031 bytes read, 1732 more expected)
NOTE: I have checked that the rest of the registered functions on server are working. So,I can rule out the connection problems(ip,port etc).
What is casuing this error message? How do I overcome the problem?
I am using python 2.7 on windows xp sp3.
UPDATE 1:
I found out that this doesn't entirely depends on the size of the database. Sometimes it gives me error message, sometimes it doesn't. Could anyone tell me what is casuing this IncompleteRead Problem?

Occasional Flask server error: 'RuntimeError: maximum recursion depth exceeded while calling a Python object'

I'm trying to figure out what causes this error when I run my app using the basic Flask server during development. I start it with this:
from myapp import app
app.run(debug=True, port=5001)
All is well and I'll continue to code and refresh etc, but then after a while I get the recursion error and have to Ctrl-C the server and restart it. Not a big deal, just a little annoying to have to deal with every now and then.
Here's the full traceback, which I tried to use to determine the cause but can't see anything that stands out (possibly something to do with how werkzeug uses Cookie.py?):
Traceback (most recent call last):
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/app.py", line 1701, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/wsgi.py", line 411, in __call__
return self.app(environ, start_response)
(last bit repeated a bunch - trimmed to fit in posting size requirements)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/app.py", line 1685, in wsgi_app
with self.request_context(environ):
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/ctx.py", line 274, in __enter__
self.push()
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/ctx.py", line 238, in push
self.session = self.app.open_session(self.request)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/app.py", line 792, in open_session
return self.session_interface.open_session(self, request)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/sessions.py", line 191, in open_session
secret_key=key)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/contrib/securecookie.py", line 309, in load_cookie
data = request.cookies.get(key)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/utils.py", line 77, in __get__
value = self.func(obj)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/wrappers.py", line 418, in cookies
cls=self.dict_storage_class)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/http.py", line 741, in parse_cookie
cookie.load(header)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Cookie.py", line 632, in load
self.__ParseString(rawdata)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Cookie.py", line 665, in __ParseString
self.__set(K, rval, cval)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/_internal.py", line 290, in _BaseCookie__set
morsel = self.get(key, _ExtendedMorsel())
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/_internal.py", line 271, in __init__
Morsel.__init__(self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Cookie.py", line 438, in __init__
dict.__setitem__(self, K, "")
RuntimeError: maximum recursion depth exceeded while calling a Python object

Since it occurs during your developement process, you could increase recursion limit, before starting your server, using :
sys.setrecursionlimit(2000) # Choose the right figure for you here
# the value on my system is 1000 but this is platform-dependant
However, you should use it very carefully and probably not in production unless you have a good knowledge of it's impacts.
Ref : http://docs.python.org/2/library/sys.html#sys.setrecursionlimit

How to Integrate Pyramid 1.1 and Mongo DB - as few lines as possible

Goal: I try to integrate Mongo DB with Pyramid 1.1 basic application.
Background: Appliation is created by the book (https://docs.pylonsproject.org/projects/pyramid/1.1/narr/project.html#creating-the-project) using basic command "paste create -t pyramid_starter"
I followed this cookbook article: https://docs.pylonsproject.org/projects/pyramid_cookbook/dev/mongo.html
Problem: It seems that when ever I add MongoDB connection into request I got "Internal Server Error" with
I have tried several articles and it seems that I must start debug system more?
Has anybody found easy solution for this?
Exception if it helps some expert
Exception happened during processing of request from ('127.0.0.1', 53697)
Traceback (most recent call last):
File "virtualenv\lib\site-packages\paste-1.7.5.1-py2.7.egg\paste\httpserver.py", line 1068, in process_request_in_thread
self.finish_request(request, client_address)
File "C:\Python27\Lib\SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python27\Lib\SocketServer.py", line 639, in __init__
self.handle()
File "virtualenv\lib\site-packages\paste-1.7.5.1-py2.7.egg\paste\httpserver.py", line 442, in handle
BaseHTTPRequestHandler.handle(self)
File "C:\Python27\Lib\BaseHTTPServer.py", line 343, in handle
self.handle_one_request()
...
File "C:\Python27\lib\site-packages\pyramid_debugtoolbar-0.8-py2.7.egg\pyramid_debugtoolbar\panels\__init__.py", line 24, in render
return render(template_name, vars, request=request)
File "virtualenv\lib\site-packages\pyramid-1.2a1-py2.7.egg\pyramid\renderers.py", line 69, in render
return helper.render(value, None, request=request)
File "virtualenv\lib\site-packages\pyramid-1.2a1-py2.7.egg\pyramid\renderers.py", line 418, in render
result = renderer(value, system_values)
File "C:\Python27\lib\site-packages\pyramid_jinja2-1.1-py2.7.egg\pyramid_jinja2\__init__.py", line 277, in __call__
return self.template.render(system)
File "C:\Python27\lib\site-packages\jinja2-2.6-py2.7.egg\jinja2\environment.py", line 894, in render
return self.environment.handle_exception(exc_info, True)
File "C:\Python27\lib\site-packages\pyramid_debugtoolbar-0.8-py2.7.egg\pyramid_debugtoolbar\panels\templates\request_vars.jinja2", line 110, in top-level template code
<td>{{ value|escape }}</td>
File "virtualenv\lib\site-packages\markupsafe-0.15-py2.7.egg\markupsafe\_native.py", line 20, in escape
return s.__html__()
File "virtualenv\lib\site-packages\pymongo-2.0.1-py2.7-win-amd64.egg\pymongo\collection.py", line 1156, in __call__
self.__name)
TypeError: 'Collection' object is not callable. If you meant to call the '__html__' method on a 'Database' object it is failing because no such method exists.

Another possible solution is to use the 'debugtoolbar.panels' setting in your config file to disable the request_vars panel (which is what is causing the issue):
[app:main]
.. other stuff ...
debugtoolbar.panels =
pyramid_debugtoolbar.panels.versions.VersionDebugPanel
pyramid_debugtoolbar.panels.settings.SettingsDebugPanel
pyramid_debugtoolbar.panels.headers.HeaderDebugPanel
# pyramid_debugtoolbar.panels.request_vars.RequestVarsDebugPanel
pyramid_debugtoolbar.panels.renderings.RenderingsDebugPanel
pyramid_debugtoolbar.panels.logger.LoggingPanel
pyramid_debugtoolbar.panels.performance.PerformanceDebugPanel
pyramid_debugtoolbar.panels.routes.RoutesDebugPanel
pyramid_debugtoolbar.panels.sqla.SQLADebugPanel

Pymongo's Database and Collection objects respond to __getattr__ in order to provide a nicer interface and let you write code like:
db.foo.bar.find(...)
Any call to __getattr__ will succeed, but unfortunately this confuses some libraries which expect certain attributes to be callable (the Pymongo Collection object is not callable, except to raise that exception you're seeing above).
What I've done in Pyramid projects is only use the database from within the resources, to prevent references to the database or collections from becoming present in the module level in views or other code. As an added benefit, this ends up being a good way of enforcing separation of concerns so that resources handle database manipulation, and views only translate that for display in the templates.

That error means that your trying to call a method (html) that doesn't exist in the Database instance.
>>> conn = Connection()
>>> db = conn.mydb
>>> col = db.mycoll
>>> col = db.mycoll()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/virtualenvs/myenv/lib/python2.7/site-packages/pymongo-2.0-py2.7-macosx-10.6-x86_64.egg/pymongo/collection.py", line 1156, in __call__
self.__name)
TypeError: 'Collection' object is not callable. If you meant to call the 'mycoll' method on a 'Database' object it is failing because no such method exists.
If you haven't modified the code then it is possible it's an bug in markupsafe that tries to call html() in a Database instance
s.__html__()