I am trying to understand how to run django tests in parallel with in memory sqlite3.
I have django app with that structure:
gbook
order
...
tests
__init__.py
test_a1.py
test_b1.py
utils.py
test_a1.py and test_b1.py contains same code:
import time
from order import models
from .utils import BackendTestCase
class ATestCase(BackendTestCase):
def test_a(self):
time.sleep(1)
a = models.City.objects.count()
self.assertEqual(a, a)
class BTestCase(BackendTestCase):
def test_b(self):
time.sleep(1)
a = models.City.objects.count()
self.assertEqual(a, a)
utils.py is:
from django.test import TestCase, Client
from order import models
from django.conf import settings
from order.utils import to_hash
class BackendTestCase(TestCase):
fixtures = ['City.json', 'Agency.json']
def setUp(self):
self.client = Client()
self.lang_codes = (i[0] for i in settings.LANGUAGES)
...
settings_test.py:
from .settings import *
DEBUG = False
TEMPLATE_DEBUG = False
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'
PASSWORD_HASHERS = ['django.contrib.auth.hashers.MD5PasswordHasher',] # faster
DATABASES['default'] = {
'ENGINE': 'django.db.backends.sqlite3',
}
When I run test in single process, all goes well (about 4 sec):
python.exe manage.py test order --settings=gbook.settings_test
Then I trying to run tests in parallel:
python.exe manage.py test order --settings=gbook.settings_test --parallel=2
I get this trace (console):
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
System check identified no issues (0 silenced).
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-1:
Traceback (most recent call last):
Traceback (most recent call last):
File "C:\python\Python36-32\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\python\Python36-32\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\python\Python36-32\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\python\Python36-32\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\python\Python36-32\lib\multiprocessing\pool.py", line 108, in worker
task = get()
File "C:\python\Python36-32\lib\multiprocessing\pool.py", line 108, in worker
task = get()
File "C:\python\Python36-32\lib\multiprocessing\queues.py", line 337, in get
return _ForkingPickler.loads(res)
File "C:\python\Python36-32\lib\multiprocessing\queues.py", line 337, in get
return _ForkingPickler.loads(res)
File "C:\kvk\develop\Python\gbook\order\tests\test_a1.py", line 2, in <module>
from order import models
File "C:\kvk\develop\Python\gbook\order\tests\test_a1.py", line 2, in <module>
from order import models
File "C:\kvk\develop\Python\gbook\order\models.py", line 79, in <module>
class Agency(models.Model):
File "C:\kvk\develop\Python\gbook\order\models.py", line 79, in <module>
class Agency(models.Model):
File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 110, in __new__
app_config = apps.get_containing_app_config(module)
File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 110, in __new__
app_config = apps.get_containing_app_config(module)
File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 247, in get_containing_app_config
self.check_apps_ready()
File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 247, in get_containing_app_config
self.check_apps_ready()
File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 125, in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 125, in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
From Pycharm trace is different:
...
Traceback (most recent call last):
File "C:\python\Python36-32\lib\unittest\suite.py", line 163, in _handleClassSetUp
setUpClass()
File "C:\python\venv\gbook\lib\site-packages\django\test\testcases.py", line 1036, in setUpClass
'database': db_name,
File "C:\python\venv\gbook\lib\site-packages\django\core\management\__init__.py", line 131, in call_command
return command.execute(*args, **defaults)
File "C:\python\venv\gbook\lib\site-packages\django\core\management\base.py", line 330, in execute
output = self.handle(*args, **options)
File "C:\python\venv\gbook\lib\site-packages\modeltranslation\management\commands\loaddata.py", line 61, in handle
return super(Command, self).handle(*fixture_labels, **options)
File "C:\python\venv\gbook\lib\site-packages\django\core\management\commands\loaddata.py", line 69, in handle
self.loaddata(fixture_labels)
File "C:\python\venv\gbook\lib\site-packages\django\core\management\commands\loaddata.py", line 109, in loaddata
self.load_label(fixture_label)
File "C:\python\venv\gbook\lib\site-packages\django\core\management\commands\loaddata.py", line 175, in load_label
obj.save(using=self.using)
File "C:\python\venv\gbook\lib\site-packages\django\core\serializers\base.py", line 205, in save
models.Model.save_base(self.object, using=using, raw=True, **kwargs)
File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 838, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 905, in _save_table
forced_update)
File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 955, in _do_update
return filtered._update(values) > 0
File "C:\python\venv\gbook\lib\site-packages\django\db\models\query.py", line 664, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
File "C:\python\venv\gbook\lib\site-packages\django\db\models\sql\compiler.py", line 1204, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "C:\python\venv\gbook\lib\site-packages\django\db\models\sql\compiler.py", line 899, in execute_sql
raise original_exception
File "C:\python\venv\gbook\lib\site-packages\django\db\models\sql\compiler.py", line 889, in execute_sql
cursor.execute(sql, params)
File "C:\python\venv\gbook\lib\site-packages\django\db\backends\utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "C:\python\venv\gbook\lib\site-packages\django\db\utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "C:\python\venv\gbook\lib\site-packages\django\utils\six.py", line 685, in reraise
raise value.with_traceback(tb)
File "C:\python\venv\gbook\lib\site-packages\django\db\backends\utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "C:\python\venv\gbook\lib\site-packages\django\db\backends\sqlite3\base.py", line 328, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: Problem installing fixture 'C:\kvk\develop\Python\gbook\order\fixtures\AirportInfo.json': Could not load order.AirportInfo(pk=2411): no such table: GB_AIRPORT_INFO
It seem like migrations not works for parallel, but why?
Docs says: "--parallel" Runs tests in separate parallel processes. Each process gets its own database. And I do not need to change my code for use it.
Please, help me to understand, what am i doing wrong.
multiprocessing.cpu_count() = 4
Django version 1.11.10
Python 3.6.5
Same issue as above with MacOS and Python 3.8+. You have to explicitly set import multiprocessing; multiprocessing.set_start_method('fork') at the top of your settings.py file. But be sure to understand the side effects before you do!
I ran into a similar issue trying to use the --parallel feature on Windows.
Django's documentation states
This feature isn’t available on Windows. It doesn’t work with the Oracle database backend either.
Running the same command on Linux completed with no issues.
Parallel running is still disabled on Windows as of today. You can track the ticket that keeps progress of this feature here: https://code.djangoproject.com/ticket/31169.
And here's the code block that disables this option on Windows:
def default_test_processes():
"""Default number of test processes when using the --parallel option."""
# The current implementation of the parallel test runner requires
# multiprocessing to start subprocesses with fork().
if multiprocessing.get_start_method() != 'fork':
return 1
try:
return int(os.environ['DJANGO_TEST_PROCESSES'])
except KeyError:
return multiprocessing.cpu_count()
Source: https://github.com/django/django/blob/59b4e99dd00b9c36d56055b889f96885995e4240/django/test/runner.py#L286-L295
In reply to #Menth, this is how I enable for testing only:
# near the top of settings.py
if "test" in sys.argv[1:]:
import multiprocessing
logging.info("Using multiproc for testing.")
multiprocessing.set_start_method("fork")
Related
Context
I have a Django application running inside a Docker container. This application uses Celery and Celery-beat for async and scheduled tasks. One of those tasks scrapes texts from different webs using Scrapy. This task runs every minute looking for new texts on the pages. If there is new information, it creates a new object in MyModel. This logic (querying the database to check if data exists, and create the object or update the info) is performed by a custom Scrapy item pipeline.
Issue
When using Development environment (using locally Docker Compose to turn on one container for the app, one container for PostgreSQL plus other services containers) everything runs smoothly. However, when using Stage environment (one Docker container for the app on a DigitalOcean droplet and a PostgreSQL self-managed cluster) the tasks throws this error:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 237, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 236, in create_cursor
cursor = self.connection.cursor()
psycopg2.InterfaceError: connection already closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/usr/local/lib/python3.8/site-packages/scrapy/utils/defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "/sites/app/services/scraper/news/pipelines.py", line 145, in process_item
existing_article = check_if_existing_article(item)
File "/sites/app/services/scraper/news/pipelines.py", line 124, in check_if_existing_article
if ProcessedArticle.objects.filter(
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 809, in exists
return self.query.has_results(using=self.db)
File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/query.py", line 537, in has_results
return compiler.has_results()
File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1126, in has_results
return bool(self.execute_sql(SINGLE))
File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1154, in execute_sql
cursor = self.connection.cursor()
File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 259, in cursor
return self._cursor()
File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 237, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 237, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 236, in create_cursor
cursor = self.connection.cursor()
django.db.utils.InterfaceError: connection already closed
Additional info
The connection to DB seems stable because there is not other issues in the whole app, and we only use one cluster
If I place in the same script db.connections.close_all() before the queries, it works for a couple of minutes, but fails later.
Once it fails once, it continues failing all the times.
Any clue?
Thanks in advance!
I am trying to play around with deploying Flask application. When I run the code using docker swarm on Amazon's EC2 I started to get following errors:
Error message:
sqlalchemy.exc:OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.[SQL: SELECT visitor."ID" AS "visitor_ID", visitor.username AS visitor_username, visitor.visits AS visitor_visits FROM visitor](Background on this error at: http://sqlalche.me/e/e3q8)
Stack trace
Traceback (most recent call last):
File "/usr/local/bin/gunicorn", line 8, in <module>
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 58, in run
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 228, in run
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 72, in run
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 202, in run
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 140, in init_process
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 123, in run
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 67, in run_for_one
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 29, in accept
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 134, in handle
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 175, in handle_request
File "/usr/local/lib/python3.7/site-packages/newrelic/api/wsgi_application.py", line 665, in _nr_wsgi_application_wrapper_
File "/usr/local/lib/python3.7/site-packages/newrelic/api/wsgi_application.py", line 193, in __init__
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
File "/usr/local/lib/python3.7/site-packages/newrelic/api/wsgi_application.py", line 555, in _nr_wsgi_application_wrapper_
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
File "/usr/local/lib/python3.7/site-packages/newrelic/hooks/framework_flask.py", line 45, in _nr_wrapper_handler_
File "/app/src/views.py", line 5, in index
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3233, in all
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3389, in __iter__
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3414, in _execute_and_instances
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 982, in execute
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
File "/usr/local/lib/python3.7/site-packages/newrelic/hooks/database_psycopg2.py", line 51, in execute
File "/usr/local/lib/python3.7/site-packages/newrelic/hooks/database_dbapi2.py", line 25, in execute
Initially I thought that this might be a result of not closing db connections after using them, but when I added db.session.close() the error still occurs.
Full code can be found here, the most important parts are:
views.py file:
from models import Visitor, db
def index():
visitors = Visitor.query.all()
response = [
{"id": visitor.ID, "username": visitor.username, "visits": visitor.visits}
for visitor in visitors
]
db.session.close()
return {"results": response}
def increment_visits(username: str) -> dict:
if username == "favicon.ico":
return {}
visitor = Visitor.query.filter_by(username=username).first()
if visitor is None:
visitor = Visitor(username=username, visits=1)
db.session.add(visitor)
else:
visitor.visits += 1
db.session.commit()
return {"id": visitor.ID, "username": visitor.username, "visits": visitor.visits}
main.py file:
import sys
from flask import Flask
from config import SQLALCHEMY_DATABASE_URI, HOST, PORT, DEBUG, DB_POOL_SIZE
from models import db
def make_app() -> Flask:
flask_app = Flask(__name__)
flask_app.config['SQLALCHEMY_DATABASE_URI'] = SQLALCHEMY_DATABASE_URI
flask_app.config['SQLALCHEMY_POOL_SIZE'] = DB_POOL_SIZE
add_urls(flask_app)
db.init_app(app=flask_app)
return flask_app
def add_urls(flask_app: Flask):
from views import index, increment_visits
flask_app.add_url_rule('/', view_func=index)
flask_app.add_url_rule('/<username>/', view_func=increment_visits)
app = make_app()
if __name__ == '__main__':
if 'createdb' in sys.argv:
app.app_context().push()
db.create_all()
else:
app.run(host=HOST, port=PORT, debug=DEBUG)
models.py:
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
class Visitor(db.Model):
ID = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True)
visits = db.Column(db.Integer(), default=1)
Docker services:
ubuntu#srv1:~$ docker stack ps --filter "desired-state=running" demo
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
frbdra27cdcb demo_web.1 gonczor/aws-simple-app:prod srv1 Running Running 7 hours ago
0by6yhvr9n4a demo_nginx.1 gonczor/aws-simple-app-nginx:prod srv1 Running Running 3 days ago
ym2t3we6r5b1 demo_db.1 postgres:11 srv1 Running Running 4 days ago
luwgpr3jnsj8 demo_web.2 gonczor/aws-simple-app:prod srv1 Running Running 7 hours ago
I have no clue where this error might come from, so I am happy to provide any further details.
EDIT
Output of requested command:
simple_app=# SELECT sum(numbackends) FROM pg_stat_database;
sum
-----
2
(1 row)
I think I found a solution, though I have very vague why it works. I found an answer in this reddit post, so here we go.
The error mainly occurred after a long idleness. When I was flooding the app with requests during benchmarking nothing happened. It seems that some connections were kept by SQLAlchemy for too long. I lowered the POOL_RECYCLE_TIME to 10 minutes and updated the settings format to avoid deprecation that will be introduced in version 3.0 of Flask-SQLAlchemy.
DB_POOL_SIZE = int(os.environ.get('DB_POOL_SIZE', '10'))
DB_POOL_RECYCLE = int(os.environ.get('DB_POOL_RECYCLE', '60'))
SQLALCHEMY_ENGINE_OPTIONS = {
'pool_size': DB_POOL_SIZE,
'pool_recycle': DB_POOL_RECYCLE,
}
As I say, I don't fully understand why this was a problem, but after introducing those changes no errors occurred. Anyway comments and links to materials explaining this issue are welcome.
when I migrate my database, i see that error.
i did init. (python manager.py db init)
and. migrate (python manager.py db migrate)
i use google app engine, flask, mac, python
what should i do?
$ python manager.py db migrate
/Library/Python/2.7/site-packages/flask_sqlalchemy/__init__.py:839: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning.
'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
Traceback (most recent call last):
File "manager.py", line 3, in <module>
manager.run()
File "/Library/Python/2.7/site-packages/flask_script/__init__.py", line 412, in run
result = self.handle(sys.argv[0], sys.argv[1:])
File "/Library/Python/2.7/site-packages/flask_script/__init__.py", line 383, in handle
res = handle(*args, **config)
File "/Library/Python/2.7/site-packages/flask_script/commands.py", line 216, in __call__
return self.run(*args, **kwargs)
File "/Library/Python/2.7/site-packages/flask_migrate/__init__.py", line 182, in migrate
version_path=version_path, rev_id=rev_id)
File "/Library/Python/2.7/site-packages/alembic/command.py", line 176, in revision
script_directory.run_env()
File "/Library/Python/2.7/site-packages/alembic/script/base.py", line 421, in run_env
util.load_python_file(self.dir, 'env.py')
File "/Library/Python/2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/Library/Python/2.7/site-packages/alembic/util/compat.py", line 75, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "migrations/env.py", line 87, in <module>
run_migrations_online()
File "migrations/env.py", line 70, in run_migrations_online
poolclass=pool.NullPool)
File "/Library/Python/2.7/site-packages/sqlalchemy/engine/__init__.py", line 428, in engine_from_config
return create_engine(url, **options)
File "/Library/Python/2.7/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine
return strategy.create(*args, **kwargs)
File "/Library/Python/2.7/site-packages/sqlalchemy/engine/strategies.py", line 80, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/Library/Python/2.7/site-packages/sqlalchemy/dialects/mysql/gaerdbms.py", line 68, in dbapi
from google.appengine.api import apiproxy_stub_map
ImportError: No module named google.appengine.api
As the Error was suggesting, your runtime environment couldn't find the module. Either install it in your virtualenvironment or globally by following the instruction:
https://cloud.google.com/appengine/downloads#Google_App_Engine_SDK_for_Python
I have a query in a model's method definition:
'content_type_id': ContentType.objects.get_for_model(model).pk,
https://github.com/seperman/django-tagging/blob/develop/tagging/models.py#L107
And it raises "App registry is not ready yet".
Putting the line inside another independent function doesn't help either since it still gets called.
Django docs:
Executing database queries with the ORM at import time in models
modules will also trigger this exception. The ORM cannot function
properly until all models are available.
How can I make it lazy evaluate this line?
I thought about caching its results manually in a file by making a management command. And then reading the cached result but there should be a much better/cleaner solution(?)
./manage.py shell
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute
django.setup()
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/__init__.py", line 21, in setup
apps.populate(settings.INSTALLED_APPS)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
app_config.import_models(all_models)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/var/www/something.com/something/current/apps/riding/models.py", line 32, in <module>
class RidingSection(CategoryBase):
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/mptt/models.py", line 244, in __new__
cls = meta.register(cls)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/mptt/models.py", line 325, in register
obj = getattr(cls, attr)
File "/var/www/something.com/something/current/apps/tagging/fields.py", line 54, in __get__
return edit_string_for_tags(Tag.objects.usage_for_model(owner))
File "/var/www/something.com/something/current/apps/tagging/models.py", line 148, in usage_for_model
usage = self.usage_for_queryset(queryset, counts, min_count)
File "/var/www/something.com/something/current/apps/tagging/models.py", line 185, in usage_for_queryset
return self._get_usage(queryset.model, counts, min_count, extra_joins, extra_criteria, params)
File "/var/www/something.com/something/current/apps/tagging/models.py", line 107, in _get_usage
'content_type_id': ContentType.objects.get_for_model(model).pk,
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/contrib/contenttypes/models.py", line 50, in get_for_model
defaults={'name': smart_text(opts.verbose_name_raw)},
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/manager.py", line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 422, in get_or_create
return self.get(**lookup), False
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 345, in get
clone = self.filter(*args, **kwargs)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 691, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 709, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1287, in add_q
clause, require_inner = self._add_q(where_part, self.used_aliases)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1314, in _add_q
current_negated=current_negated, connector=connector)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1138, in build_filter
lookups, parts, reffed_aggregate = self.solve_lookup_type(arg)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1076, in solve_lookup_type
_, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1339, in names_to_path
field, model, direct, m2m = opts.get_field_by_name(name)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/options.py", line 416, in get_field_by_name
cache = self.init_name_map()
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/options.py", line 445, in init_name_map
for f, model in self.get_all_related_m2m_objects_with_model():
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/options.py", line 563, in get_all_related_m2m_objects_with_model
cache = self._fill_related_many_to_many_cache()
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/db/models/options.py", line 577, in _fill_related_many_to_many_cache
for klass in self.apps.get_models():
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/utils/lru_cache.py", line 101, in wrapper
result = user_function(*args, **kwds)
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/apps/registry.py", line 168, in get_models
self.check_models_ready()
File "/var/www/something.com/something/shared/env/local/lib/python2.7/site-packages/django/apps/registry.py", line 131, in check_models_ready
raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
django-tagging is being a little inconsiderate here.
TagField is a descriptor, meaning if you access YourModel.tags, it runs code which executes a database query.
It just so happens that django-mptt iterates over all the attributes of your MPTTModel classes during startup (before the app cache has loaded), so it happens to trigger this exception.
I don't know how you'd prevent that in django-tagging while maintaining the API (YourModel.tags has to get tags from somewhere). Issues like this are probably the reason why django-tagging is no longer maintained.
However, I'm the maintainer of django-mptt and I've committed a workaround. If you upgrade to the master version of mptt you should find it works now.
You are calling _get_usage before the app has been registered. Wherever you are calling it needs to be changed to a point where the app has been configured. Basically, if you need a model at import time you should use ready().
For example, if you had a signals.py you could do:
# in apps.py
class SomeAppConfig(AppConfig):
def ready(self):
from . import signals
or in your case:
# in apps.py
class SomeAppConfig(AppConfig):
def ready(self):
mgr = TagManager()
mgr._get_usage(#some_args)
As the docs state: ready() ... is called as soon as the registry is fully populated
So I have a Django project that doesn't use a database (the 'DATABASES' setting is commented out). I chose to use Django as there's a chance I will need the database functionality in the future. Anyway, I've been working on and off of tbhis project for a couple of months with no problems. I'm running Linux Mint and have had no troubles using the python manage.py runserver command so far.
Well, today I fired up the app and started the local server with no problems. I then tried to open the app in my browser and received the rather hideous error message:
Traceback (most recent call last):
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/sessions/backends/base.py", line 170, in _get_session
return self._session_cache
AttributeError: 'SessionStore' object has no attribute '_session_cache'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/core/handlers/base.py", line 87, in get_response
response = middleware_method(request)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/auth/middleware.py", line 34, in process_request
if user and hasattr(user, 'get_session_auth_hash'):
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/utils/functional.py", line 224, in inner
self._setup()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/utils/functional.py", line 357, in _setup
self._wrapped = self._setupfunc()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/auth/middleware.py", line 23, in <lambda>
request.user = SimpleLazyObject(lambda: get_user(request))
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/auth/middleware.py", line 11, in get_user
request._cached_user = auth.get_user(request)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/auth/__init__.py", line 151, in get_user
user_id = request.session[SESSION_KEY]
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/sessions/backends/base.py", line 49, in __getitem__
return self._session[key]
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/sessions/backends/base.py", line 175, in _get_session
self._session_cache = self.load()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/sessions/backends/db.py", line 21, in load
expire_date__gt=timezone.now()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/manager.py", line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/query.py", line 351, in get
num = len(clone)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/query.py", line 122, in __len__
self._fetch_all()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/query.py", line 966, in _fetch_all
self._result_cache = list(self.iterator())
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/query.py", line 265, in iterator
for row in compiler.results_iter():
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 700, in results_iter
for rows in self.execute_sql(MULTI):
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 775, in execute_sql
sql, params = self.as_sql()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 100, in as_sql
out_cols, s_params = self.get_columns(with_col_aliases)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 246, in get_columns
col_aliases)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 328, in get_default_columns
r = '%s.%s' % (qn(alias), qn2(column))
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 62, in __call__
r = self.connection.ops.quote_name(name)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/backends/dummy/base.py", line 18, in complain
raise ImproperlyConfigured("settings.DATABASES is improperly configured. "
django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/sessions/backends/base.py", line 170, in _get_session
return self._session_cache
AttributeError: 'SessionStore' object has no attribute '_session_cache'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.3/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/staticfiles/handlers.py", line 64, in __call__
return self.application(environ, start_response)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/core/handlers/wsgi.py", line 187, in __call__
response = self.get_response(request)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/core/handlers/base.py", line 199, in get_response
response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/core/handlers/base.py", line 236, in handle_uncaught_exception
return debug.technical_500_response(request, *exc_info)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/views/debug.py", line 91, in technical_500_response
html = reporter.get_traceback_html()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/views/debug.py", line 349, in get_traceback_html
c = Context(self.get_traceback_data(), use_l10n=False)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/views/debug.py", line 307, in get_traceback_data
frames = self.get_traceback_frames()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/views/debug.py", line 465, in get_traceback_frames
'vars': self.filter.get_traceback_frame_variables(self.request, tb.tb_frame),
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/views/debug.py", line 232, in get_traceback_frame_variables
cleansed[name] = self.cleanse_special_types(request, value)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/views/debug.py", line 187, in cleanse_special_types
if isinstance(value, HttpRequest):
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/utils/functional.py", line 224, in inner
self._setup()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/utils/functional.py", line 357, in _setup
self._wrapped = self._setupfunc()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/auth/middleware.py", line 23, in <lambda>
request.user = SimpleLazyObject(lambda: get_user(request))
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/auth/middleware.py", line 11, in get_user
request._cached_user = auth.get_user(request)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/auth/__init__.py", line 151, in get_user
user_id = request.session[SESSION_KEY]
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/sessions/backends/base.py", line 49, in __getitem__
return self._session[key]
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/sessions/backends/base.py", line 175, in _get_session
self._session_cache = self.load()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/contrib/sessions/backends/db.py", line 21, in load
expire_date__gt=timezone.now()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/manager.py", line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/query.py", line 351, in get
num = len(clone)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/query.py", line 122, in __len__
self._fetch_all()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/query.py", line 966, in _fetch_all
self._result_cache = list(self.iterator())
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/query.py", line 265, in iterator
for row in compiler.results_iter():
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 700, in results_iter
for rows in self.execute_sql(MULTI):
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 775, in execute_sql
sql, params = self.as_sql()
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 100, in as_sql
out_cols, s_params = self.get_columns(with_col_aliases)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 246, in get_columns
col_aliases)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 328, in get_default_columns
r = '%s.%s' % (qn(alias), qn2(column))
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/models/sql/compiler.py", line 62, in __call__
r = self.connection.ops.quote_name(name)
File "/home/peter/.virtualenvs/vis_it/lib/python3.3/site-packages/django/db/backends/dummy/base.py", line 18, in complain
raise ImproperlyConfigured("settings.DATABASES is improperly configured. "
django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.
[28/Nov/2014 13:18:54] "GET / HTTP/1.1" 500 59
I've not touched the app since I last worked on it and it was working fine then. I really have no idea what this is talking about as I have no caches implemented and am not using a database... I've asked a few colleges who are familiar with Django and have no idea what this is either. Any Ideas? I've also tried setting DATABASES to an empty dict {} on the advice of a post online but to no effect. At a bit of a loss.
EDIT: Thought I should mention that in the interim since I last touched this project, I've since started and set up a new Django project that does use a database. Is it possible that this project is somehow cached and breaking the one I'm currently trying to work on?
EDIT2: I should point out that this webapp is currently live and working at http://mrcagney-maps.com. The code is exactly the same (having not touched it since I last pushed to the server). Really weird.
For future Googlers - I ran into this issue and the above solutions did not work for me. What did work for me was clearing/deleting my cookies in Chrome for the 127.0.0.1 URL. So go to Settings or press CMD+, then Cookies and other site data, then find 127.0.0.1 or localhost and delete those cookies. Refresh the local dev host page and the error should be gone. This has something to do with a corrupted session / cookie file.
I'm not sure why I started to get this error, it involved an upgrade though. I just deleted all the sessions and after logging back in all was well.
# from the shell but equivalent sql would work fine too
from django.contrib.sessions.models import Session
Session.objects.all().delete()
The error AttributeError: 'SessionStore' object has no attribute '_session_cache' can stem from the database not having a django_session table. However, since you are not using a table, you would need to make sure that you don't have the 'django.contrib.sessions.middleware.SessionMiddleware' in your MIDDLEWARE_CLASSES in the project's settings file. If it is in there, it will look for a database table which stores the sessions, causing the above error.
Here is what worked for me. Since there is no databases in your application. Admin page looks for the database be it default. So first lets create the default databases.
Shut down your servers and run
python manage.py makemigrations
python manage.py migrate
Now create the admin or superuser for your application. Fill username and password.
python manage.py createsuperuser
Now restart your server and go the admin page
python manage.py runserver
Throwed the same error for me after upgraded my wagtail version... So, for me worked even simpler solution or u didn't noticed already :D Actually I opened the browser > F12 > Storage Tab and Delete all Cookies and all other Cached Data
Users facing this issue via Postman please make sure that the value in the request header for Cache-Control is no-cache.
If not then clear the cookie data for the domain.
In normal case when we use
from django.http import JsonResponse
for sending response we have no need to uses sessions.
But when we use some package such as djangorestframework we force to use sessions.
Sessions need storage place in sqllite,mysql ,...or others
then we need run these commands:
python manage.py makemigrations
python manage.py migrate