We Keep Coding

Why am I getting pymongo.errors.AutoReconnect: connection pool paused

I'm getting this error of autoreconnect, and there are around 100 connections in the logs during this call to .objects. Here is the document:
class NotificationDoc(Document):
patient_id = StringField(max_length=32)
type = StringField(max_length=32)
sms_sent = BooleanField(default=False)
email_sent = BooleanField(default=False)
sending_time = DateTimeField(default=datetime.utcnow)
def __unicode__(self):
return "{}_{}_{}".format(self.patient_id, self.type, self.sending_time.strftime('%Y-%m-%d'))
and this is the query call that causes the issue:
docs = NotificationDoc.objects(sending_time__lte=from_date)
This is the complete stacktrace. How can I understand what is going on?
pymongo.errors.AutoReconnect: connection pool paused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/app/src/reminder/configurations/app/worker.py", line 106, in schedule_recall
schedule_recall_use_case.execute()
File "/app/src/reminder/domain/notification/recall/use_cases/schedule_recall.py", line 24, in execute
notifications_sent = self.notifications_provider.find_recalled_notifications_for_date(no_recalls_before_date)
File "/app/src/reminder/data_providers/database/odm/repositories.py", line 42, in find_recalled_notifications_for_date
if NotificationDoc.objects(sending_time__lte=from_date).count() > 0:
File "/usr/local/lib/python3.8/site-packages/mongoengine/queryset/queryset.py", line 144, in count
return super().count(with_limit_and_skip)
File "/usr/local/lib/python3.8/site-packages/mongoengine/queryset/base.py", line 423, in count
count = count_documents(
File "/usr/local/lib/python3.8/site-packages/mongoengine/pymongo_support.py", line 38, in count_documents
return collection.count_documents(filter=filter, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pymongo/collection.py", line 1502, in count_documents
return self.__database.client._retryable_read(
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1307, in _retryable_read
with self._secondaryok_for_server(read_pref, server, session) as (
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1162, in _secondaryok_for_server
with self._get_socket(server, session) as sock_info:
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1099, in _get_socket
with server.get_socket(
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1371, in get_socket
sock_info = self._get_socket(all_credentials)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1436, in _get_socket
self._raise_if_not_ready(emit_event=True)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1407, in _raise_if_not_ready
_raise_connection_failure(
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 250, in _raise_connection_failure
raise AutoReconnect(msg) from error
pymongo.errors.AutoReconnect: mongo:27017: connection pool paused

turned out to be celery is leaving connections open, so here is the solution
from mongoengine import disconnect
from celery.signals import task_prerun
#task_prerun.connect
def on_task_init(*args, **kwargs):
disconnect(alias='default')
connect(db, host=host, port=port, maxPoolSize=400, minPoolSize=200, alias='default')

This could be related to the fact that PyMongo is not fork-safe. If you're using a process pool in any way, which includes server software like uWSGI and even some configurations of popular ASGI servers.
Every time you run a query in PyMongo, that MongoClient object becomes fork-unsafe. A MongoClient object that has never run any queries is fork-safe. Created Database and Collection objects will reference back to their parent MongoClient without checking for existence.
I do see you are using MongoEngine, which uses PyMongo underneath. I don't know the semantics of that particular library but I would assume they are the same.
In short: you must re-create your connections as part of your forking process.

pip install -U pymongo; is already fix; see this https://github.com/mongodb/mongo-python-driver/pull/944

How to use multiprocessing in Python with Django to create thousands of model instances from xml file?

I have a script that parses data from an xml file into Django models. It works ok but recently we've been encountering files with dozen of thousands of models to create which is incredibly slowing down the process (up to 40 min to upload a single 20MB file).
I would like to try and use multiprocessing to speed it up as much as I can but I am new to this package. This is the current setup:
create_profiles_from_xml: is a function that loops through the data extracted from the xml, called xml_members, popping one element at a time (it has to be like that due to a relationship that can be created with the next element or not).
create_profile: is a just a function that handles the data to match with the model fields and then call the .create() method, that's why I omitted the details of this function here.
def create_profiles_from_xml(xml_members, device):
profiles = []
while len(xml_members) > 0:
parent_member = members.pop(0)
profile = create_profile(parent_member, parent=None)
if profile:
profiles.append(profile)
return profiles
And here is what I tried to do with the multipocessing
from multiprocessing import Pool
def create_profiles_from_xml(xml_members, device):
profiles = []
pool = Pool(processes=cpu_count())
profiles = pool.imap_unordered(create_profile, xml_members)
return profiles
And everytime I try to run it I get this error traceback:
Traceback (most recent call last):
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/base/base.py", line 235, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 223, in create_cursor
cursor = self.connection.cursor()
psycopg2.InterfaceError: connection already closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/axes/middleware.py", line 49, in __call__
failures_since_start = AxesProxyHandler.get_failures(request, credentials)
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/axes/handlers/proxy.py", line 94, in get_failures
return cls.get_implementation().get_failures(request, credentials)
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/axes/handlers/database.py", line 75, in get_failures
attempt_count = max(
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/axes/handlers/database.py", line 77, in <genexpr>
attempts.aggregate(Sum("failures_since_start"))[
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/models/query.py", line 379, in aggregate
return query.get_aggregation(self.db, kwargs)
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/models/sql/query.py", line 489, in get_aggregation
result = compiler.execute_sql(SINGLE)
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1140, in execute_sql
cursor = self.connection.cursor()
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/base/base.py", line 235, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/base/base.py", line 235, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 223, in create_cursor
cursor = self.connection.cursor()
django.db.utils.InterfaceError: connection already closed

_gdbm.error: Database needs recovery -- after running out of storage while fetching api data

I really don't know how to help myself, being unfamiliar with this kind of error, and not finding anything on the Google landscape really. My last hope is one of you guys since I don't know where else to go with this. I tried reinstalling all libraries and setting up a new venv. For more action I don't trust myself enough in these kinds of things.
The code triggering the error:
from wetterdienst import DWDObservationData
observations_daily = DWDObservationData(
station_ids=station_ids_d,
parameter=params_daily,
time_resolution=TimeResolution.DAILY,
start_date="2015-01-01",
end_date="2020-10-10",
tidy_data=True,
humanize_column_names=True,
)
for df in observations_hourly.collect_data():
name = str(df.STATION_ID.iloc[0]).strip(".0")
df.to_csv('./data/hourly/{}.csv'.format(name))
print('{} done'.format(name))
API is found here: https://github.com/earthobservations/wetterdienst
Error:
Traceback (most recent call last):
File "/Users/sashakaun/PycharmProjects/wetter2.0/main.py", line 83, in <module>
for df in observations_hourly.collect_data():
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/api.py", line 178, in collect_data
df_parameter = self._collect_parameter_from_station(
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/api.py", line 243, in _collect_parameter_from_station
df_period = collect_climate_observations_data(
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/access.py", line 82, in collect_climate_observations_data
filenames_and_files = download_climate_observations_data_parallel(remote_files)
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/access.py", line 106, in download_climate_observations_data_parallel
return list(zip(remote_files, files_in_bytes))
File "/usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
yield fs.pop().result()
File "/usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
raise self._exception
File "/usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/access.py", line 124, in _download_climate_observations_data
return BytesIO(__download_climate_observations_data(remote_file=remote_file))
File "<decorator-gen-2>", line 2, in __download_climate_observations_data
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/cache/region.py", line 1356, in get_or_create_for_user_func
return self.get_or_create(
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/cache/region.py", line 954, in get_or_create
with Lock(
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/lock.py", line 185, in __enter__
return self._enter()
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/lock.py", line 94, in _enter
generated = self._enter_create(value, createdtime)
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/lock.py", line 178, in _enter_create
return self.creator()
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/cache/region.py", line 920, in gen_value
self.backend.set(key, value)
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/cache/backends/file.py", line 239, in set
dbm[key] = pickle.dumps(value, pickle.HIGHEST_PROTOCOL)
_gdbm.error: Database needs recovery
Thanks a lot!!

A GDBM file has been corrupted. You need to use gdbmtool to recover the database. Install gdbmtool then run
gdbmtool FILENAME
Where FILENAME is the name of the GDBM database. A prompt will appear, then you can enter
gdbmtool> recover summary
If the database can be recovered it will display a summary of the recovery results, eg:
Recovery succeeded.
Keys recovered: 6870650, failed: 5, duplicate: 0
Buckets recovered: 64830, failed: 2

Simple *IDN? query resulted in "Timeout expired before operation completed"

I tried to do a simple query to my LAB instrument by:
>>> import visa
>>> rm = visa.ResourceManager()
>>> viavi = rm.open_resource("TCPIP0::10.0.2.76::5001::SOCKET")
>>> print(viavi.query("*IDN?"))
The result was:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\Python35\lib\site-packages\pyvisa\resources\messagebase
d.py", line 407, in query
return self.read()
File "C:\Program Files\Python35\lib\site-packages\pyvisa\resources\messagebase
d.py", line 332, in read
message = self.read_raw().decode(enco)
File "C:\Program Files\Python35\lib\site-packages\pyvisa\resources\messagebase
d.py", line 306, in read_raw
chunk, status = self.visalib.read(self.session, size)
File "C:\Program Files\Python35\lib\site-packages\pyvisa\ctwrapper\functions.p
y", line 1582, in read
ret = library.viRead(session, buffer, count, byref(return_count))
File "C:\Program Files\Python35\lib\site-packages\pyvisa\ctwrapper\highlevel.p
y", line 188, in _return_handler
raise errors.VisaIOError(ret_value)
pyvisa.errors.VisaIOError: VI_ERROR_TMO (-1073807339): Timeout expired before op
eration completed.
According to what I learned till now (from the experience of others). This timeout error is somehow related to line termination ("\n"). How can I solve this problem ?

I found out that it was all related to the read_termination. My LAB instrument simply terminated its response by a '\n'.
While my script was looking for a '\r' all that time.

Catching python exception in its own class

I'm having a strange problem right now.
In my program I try to catch an exception inside a class, de class is used in a separate python file but the exception bubbles trough to the calling file.
How can I fix this?
Example:
main.py
#from sqlalchemy.exc import SQLAlchemyError, DatabaseError
from os import path
from helpers.db import DB
from models.group import Group
from models.entity import Entity
PROJECT_PATH = path.split(path.abspath(__file__))[0]
DATABASE_PATH = PROJECT_PATH + '/datastore/test.db'
class PasswordManager:
database = DB(DATABASE_PATH)
if __name__ == "__main__":
PasswordManager()
output
Traceback (most recent call last):
trying to build a SQLEngine
File "/home/tom/Projects/test/test/test.py", line 11, in <module>
class test:
File "/home/tom/Projects/test/test/test.py", line 23, in test
database = DB(DATABASE_PATH)
File "/home/tom/Projects/test/test/helpers/db.py", line 19, in __init__
self.buildDatabaseModels()
File "/home/tom/Projects/test/test/helpers/db.py", line 39, in buildDatabaseModels
Base.metadata.create_all(self.SQLEngine, checkfirst=True)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3308, in create_all
tables=tables)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1516, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1164, in _run_visitor
**kwargs).traverse_single(element)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 119, in traverse_single
return meth(obj, **kw)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 696, in visit_metadata
if self._can_create_table(t)]
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 674, in _can_create_table
table.name, schema=table.schema)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 781, in has_table
cursor = _pragma_cursor(connection.execute(statement))
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 709, in execute
return self._execute_text(object, multiparams, params)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 858, in _execute_text
statement, parameters
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 927, in _execute_context
context)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1076, in _handle_dbapi_exception
exc_info
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 185, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 920, in _execute_context
context)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 425, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (DatabaseError) file is encrypted or is not a database u'PRAGMA table_info("groups")' ()
DB.py
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.exc import DatabaseError, SQLAlchemyError
from models import *
from helpers import Base
from helpers.crypt import Crypt
class DB:
SQLEngine = None
SessionMaker = None
Session = None
BuildEngineErrorCount = 0
def __init__(self, db_path):
self.buildEngine(db_path)
self.buildDatabaseModels()
def buildEngine(self, db_path):
try:
self.SQLEngine = create_engine('sqlite:////' + db_path)
print "trying to build a SQLEngine"
except BaseException, er:
print "Errors %s" % er
for error in er.args:
if error == '(DatabaseError) file is encrypted or is not a database':
self.BuildEngineErrorCount += 1
c = Crypt()
c.setKey('Test')
c.decryptDB(db_path)
if self.BuildEngineErrorCount < 5:
self.buildEngine(db_path)
def buildDatabaseModels(self):
if self.SQLEngine is None:
raise Exception("No SQLEngine found")
Base.metadata.create_all(self.SQLEngine, checkfirst=True)
def createSession(self):
if self.SessionMaker is not None or self.Session is not None:
raise Exception("Session already mapped")
self.SessionMaker = sessionmaker(bind=self.SQLEngine)
self.Session = self.SessionMaker()
return self.Session
As you can see I try to catch that exception in the db class, but it isn't catching anything.
Does anybody know what I do wrong?

Your exception is coming from a different method than the one you put the try-except in. Specifically, the exception comes from
Base.metadata.create_all(self.SQLEngine, checkfirst=True)
in buildDatabaseModels, but you have your try-except in buildEngine.