I am new to django 1.7 and python3. I am using OSX. As I was following the django 1.7 documentation online,
I tried
python3 manage.py migrate
and it resulted
Operations to perform:
Apply all migrations: auth, contenttypes, sessions, admin
Running migrations:
No migrations to apply.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/contrib/contenttypes/models.py", line 44, in get_for_model
ct = self._get_from_cache(opts)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/contrib/contenttypes/models.py", line 34, in _get_from_cache
return self.__class__._cache[self.db][key]
KeyError: 'default'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/backends/mysql/base.py", line 128, in execute
return self.cursor.execute(query, args)
File "/Users/NAME/Library/Python/3.4/lib/python/site-packages/MySQLdb/cursors.py", line 184, in execute
self.errorhandler(self, exc, value)
File "/Users/NAME/Library/Python/3.4/lib/python/site-packages/MySQLdb/connections.py", line 37, in defaulterrorhandler
raise errorvalue
File "/Users/NAME/Library/Python/3.4/lib/python/site-packages/MySQLdb/cursors.py", line 171, in execute
r = self._query(query)
File "/Users/NAME/Library/Python/3.4/lib/python/site-packages/MySQLdb/cursors.py", line 330, in _query
rowcount = self._do_query(q)
File "/Users/NAME/Library/Python/3.4/lib/python/site-packages/MySQLdb/cursors.py", line 294, in _do_query
db.query(q)
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%s AND `django_content_type`.`model` = %s) LIMIT 21' at line 1")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/contrib/contenttypes/models.py", line 50, in get_for_model
defaults={'name': smart_text(opts.verbose_name_raw)},
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/models/manager.py", line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/models/query.py", line 422, in get_or_create
return self.get(**lookup), False
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/models/query.py", line 351, in get
num = len(clone)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/models/query.py", line 122, in __len__
self._fetch_all()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/models/query.py", line 966, in _fetch_all
self._result_cache = list(self.iterator())
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/models/query.py", line 265, in iterator
for row in compiler.results_iter():
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 700, in results_iter
for rows in self.execute_sql(MULTI):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/backends/utils.py", line 81, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/utils/six.py", line 549, in reraise
raise value.with_traceback(tb)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/db/backends/mysql/base.py", line 128, in execute
return self.cursor.execute(query, args)
File "/Users/NAME/Library/Python/3.4/lib/python/site-packages/MySQLdb/cursors.py", line 184, in execute
self.errorhandler(self, exc, value)
File "/Users/NAME/Library/Python/3.4/lib/python/site-packages/MySQLdb/connections.py", line 37, in defaulterrorhandler
raise errorvalue
File "/Users/NAME/Library/Python/3.4/lib/python/site-packages/MySQLdb/cursors.py", line 171, in execute
r = self._query(query)
File "/Users/NAME/Library/Python/3.4/lib/python/site-packages/MySQLdb/cursors.py", line 330, in _query
rowcount = self._do_query(q)
File "/Users/NAME/Library/Python/3.4/lib/python/site-packages/MySQLdb/cursors.py", line 294, in _do_query
db.query(q)
django.db.utils.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%s AND `django_content_type`.`model` = %s) LIMIT 21' at line 1")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/core/management/__init__.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **options.__dict__)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/core/management/base.py", line 338, in execute
output = self.handle(*args, **options)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/core/management/commands/migrate.py", line 164, in handle
emit_post_migrate_signal(created_models, self.verbosity, self.interactive, connection.alias)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/core/management/sql.py", line 268, in emit_post_migrate_signal
using=db)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/dispatch/dispatcher.py", line 198, in send
response = receiver(signal=self, sender=sender, **named)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/contrib/auth/management/__init__.py", line 83, in create_permissions
ctype = ContentType.objects.db_manager(using).get_for_model(klass)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/contrib/contenttypes/models.py", line 58, in get_for_model
" is migrated before trying to migrate apps individually."
RuntimeError: Error creating new content types. Please make sure contenttypes is migrated before trying to migrate apps individually.`
I don't know what it means and I am sure it is not right.
Is the migration successful?
Please help. Thanks
Here is my /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/django/contrib/contenttypes/models.py
from __future__ import unicode_literals
from django.apps import apps
from django.db import models
from django.db.utils import OperationalError, ProgrammingError
from django.utils.translation import ugettext_lazy as _
from django.utils.encoding import smart_text, force_text
from django.utils.encoding import python_2_unicode_compatible
class ContentTypeManager(models.Manager):
# Cache to avoid re-looking up ContentType objects all over the place.
# This cache is shared by all the get_for_* methods.
_cache = {}
def get_by_natural_key(self, app_label, model):
try:
ct = self.__class__._cache[self.db][(app_label, model)]
except KeyError:
ct = self.get(app_label=app_label, model=model)
self._add_to_cache(self.db, ct)
return ct
def _get_opts(self, model, for_concrete_model):
if for_concrete_model:
model = model._meta.concrete_model
elif model._deferred:
model = model._meta.proxy_for_model
return model._meta
def _get_from_cache(self, opts):
key = (opts.app_label, opts.model_name)
return self.__class__._cache[self.db][key]
def get_for_model(self, model, for_concrete_model=True):
"""
Returns the ContentType object for a given model, creating the
ContentType if necessary. Lookups are cached so that subsequent lookups
for the same model don't hit the database.
"""
opts = self._get_opts(model, for_concrete_model)
try:
ct = self._get_from_cache(opts)
except KeyError:
try:
ct, created = self.get_or_create(
app_label=opts.app_label,
model=opts.model_name,
defaults={'name': smart_text(opts.verbose_name_raw)},
)
except (OperationalError, ProgrammingError):
# It's possible to migrate a single app before contenttypes,
# as it's not a required initial dependency (it's contrib!)
# Have a nice error for this.
raise RuntimeError(
"Error creating new content types. Please make sure contenttypes" +
" is migrated before trying to migrate apps individually."
)
self._add_to_cache(self.db, ct)
return ct
def get_for_models(self, *models, **kwargs):
"""
Given *models, returns a dictionary mapping {model: content_type}.
"""
for_concrete_models = kwargs.pop('for_concrete_models', True)
# Final results
results = {}
# models that aren't already in the cache
needed_app_labels = set()
needed_models = set()
needed_opts = set()
for model in models:
opts = self._get_opts(model, for_concrete_models)
try:
ct = self._get_from_cache(opts)
except KeyError:
needed_app_labels.add(opts.app_label)
needed_models.add(opts.model_name)
needed_opts.add(opts)
else:
results[model] = ct
if needed_opts:
cts = self.filter(
app_label__in=needed_app_labels,
model__in=needed_models
)
for ct in cts:
model = ct.model_class()
if model._meta in needed_opts:
results[model] = ct
needed_opts.remove(model._meta)
self._add_to_cache(self.db, ct)
for opts in needed_opts:
# These weren't in the cache, or the DB, create them.
ct = self.create(
app_label=opts.app_label,
model=opts.model_name,
name=smart_text(opts.verbose_name_raw),
)
self._add_to_cache(self.db, ct)
results[ct.model_class()] = ct
return results
def get_for_id(self, id):
"""
Lookup a ContentType by ID. Uses the same shared cache as get_for_model
(though ContentTypes are obviously not created on-the-fly by get_by_id).
"""
try:
ct = self.__class__._cache[self.db][id]
except KeyError:
# This could raise a DoesNotExist; that's correct behavior and will
# make sure that only correct ctypes get stored in the cache dict.
ct = self.get(pk=id)
self._add_to_cache(self.db, ct)
return ct
def clear_cache(self):
"""
Clear out the content-type cache. This needs to happen during database
flushes to prevent caching of "stale" content type IDs (see
django.contrib.contenttypes.management.update_contenttypes for where
this gets called).
"""
self.__class__._cache.clear()
def _add_to_cache(self, using, ct):
"""Insert a ContentType into the cache."""
# Note it's possible for ContentType objects to be stale; model_class() will return None.
# Hence, there is no reliance on model._meta.app_label here, just using the model fields instead.
key = (ct.app_label, ct.model)
self.__class__._cache.setdefault(using, {})[key] = ct
self.__class__._cache.setdefault(using, {})[ct.id] = ct
#python_2_unicode_compatible
class ContentType(models.Model):
name = models.CharField(max_length=100)
app_label = models.CharField(max_length=100)
model = models.CharField(_('python model class name'), max_length=100)
objects = ContentTypeManager()
class Meta:
verbose_name = _('content type')
verbose_name_plural = _('content types')
db_table = 'django_content_type'
ordering = ('name',)
unique_together = (('app_label', 'model'),)
def __str__(self):
# self.name is deprecated in favor of using model's verbose_name, which
# can be translated. Formal deprecation is delayed until we have DB
# migration to be able to remove the field from the database along with
# the attribute.
#
# We return self.name only when users have changed its value from the
# initial verbose_name_raw and might rely on it.
model = self.model_class()
if not model or self.name != model._meta.verbose_name_raw:
return self.name
else:
return force_text(model._meta.verbose_name)
def model_class(self):
"Returns the Python model class for this type of content."
try:
return apps.get_model(self.app_label, self.model)
except LookupError:
return None
def get_object_for_this_type(self, **kwargs):
"""
Returns an object of this type for the keyword arguments given.
Basically, this is a proxy around this object_type's get_object() model
method. The ObjectNotExist exception, if thrown, will not be caught,
so code that calls this method should catch it.
"""
return self.model_class()._base_manager.using(self._state.db).get(**kwargs)
def get_all_objects_for_this_type(self, **kwargs):
"""
Returns all objects of this type for the keyword arguments given.
"""
return self.model_class()._base_manager.using(self._state.db).filter(**kwargs)
def natural_key(self):
return (self.app_label, self.model)
I believe it is the dependency with mysql. After I switched to postgresql, everything is solved. I found the connector of python with mysql is only up to python 3.3 and I am using python3.4. That is probably the reason and I could not find a connector for mysql and python 3.4.
I assume that there could be some problems with the dependencies.
Instead of running:
python3 manage.py migrate
try with:
python manage.py makemigrations yourapp
python manage.py migrate
if this doesn't work go with:
python manage.py syncdb --all
Related
I have a script that parses data from an xml file into Django models. It works ok but recently we've been encountering files with dozen of thousands of models to create which is incredibly slowing down the process (up to 40 min to upload a single 20MB file).
I would like to try and use multiprocessing to speed it up as much as I can but I am new to this package. This is the current setup:
create_profiles_from_xml: is a function that loops through the data extracted from the xml, called xml_members, popping one element at a time (it has to be like that due to a relationship that can be created with the next element or not).
create_profile: is a just a function that handles the data to match with the model fields and then call the .create() method, that's why I omitted the details of this function here.
def create_profiles_from_xml(xml_members, device):
profiles = []
while len(xml_members) > 0:
parent_member = members.pop(0)
profile = create_profile(parent_member, parent=None)
if profile:
profiles.append(profile)
return profiles
And here is what I tried to do with the multipocessing
from multiprocessing import Pool
def create_profiles_from_xml(xml_members, device):
profiles = []
pool = Pool(processes=cpu_count())
profiles = pool.imap_unordered(create_profile, xml_members)
return profiles
And everytime I try to run it I get this error traceback:
Traceback (most recent call last):
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/base/base.py", line 235, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 223, in create_cursor
cursor = self.connection.cursor()
psycopg2.InterfaceError: connection already closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/axes/middleware.py", line 49, in __call__
failures_since_start = AxesProxyHandler.get_failures(request, credentials)
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/axes/handlers/proxy.py", line 94, in get_failures
return cls.get_implementation().get_failures(request, credentials)
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/axes/handlers/database.py", line 75, in get_failures
attempt_count = max(
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/axes/handlers/database.py", line 77, in <genexpr>
attempts.aggregate(Sum("failures_since_start"))[
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/models/query.py", line 379, in aggregate
return query.get_aggregation(self.db, kwargs)
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/models/sql/query.py", line 489, in get_aggregation
result = compiler.execute_sql(SINGLE)
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1140, in execute_sql
cursor = self.connection.cursor()
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/base/base.py", line 235, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/base/base.py", line 235, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/home/user/.virtualenvs/omnio/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 223, in create_cursor
cursor = self.connection.cursor()
django.db.utils.InterfaceError: connection already closed
Assume a Django application which is supposed to use two MySQL databases:
default - for storing data represented by models A and B (read-write access)
support - for importing data represented by models C and D (read-only access)
The support database is a part of an external application and cannot be modified.
Since the Django application uses the built-in ORM for models A and B I figured it should use the very same ORM for models C and D, even though they map to tables in an external database (support.)
In order to achieve that I defined the models C and D as follows:
from django.db import models
class ExternalModel(models.Model):
class Meta:
managed = False
abstract = True
class ModelC(ExternalModel):
some_field = models.TextField(db_column='some_field')
class Meta(ExternalModel.Meta):
db_table = 'some_table_c'
class ModelD(ExternalModel):
some_other_field = models.TextField(db_column='some_other_field')
class Meta(ExternalModel.Meta):
db_table = 'some_table_d'
Then I defined a database router:
from myapp.myapp.models import ExternalModel
class DatabaseRouter(object):
def db_for_read(self, model, **hints):
if issubclass(model, ExternalModel):
return 'support'
return 'default'
def db_for_write(self, model, **hints):
if issubclass(model, ExternalModel):
return None
return 'default'
def allow_relation(self, obj1, obj2, **hints):
return (isinstance(obj1, ExternalModel) == isinstance(obj2, ExternalModel))
def allow_migrate(self, db, app_label, model_name=None, **hints):
return (db == 'default')
And finally adjusted settings.py:
# (...)
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'OPTIONS': {
'read_default_file': os.path.join(BASE_DIR, 'resources', 'default.cnf'),
},
},
'support': {
'ENGINE': 'django.db.backends.mysql',
'OPTIONS': {
'read_default_file': os.path.join(BASE_DIR, 'resources', 'support.cnf'),
},
},
}
DATABASE_ROUTERS = ['myapp.database_router.DatabaseRouter']
# (...)
The user specified in support.conf for the support database has been assigned read-only privileges.
But when I run python manage.py makemigrations it fails with the following output:
Traceback (most recent call last):
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
return self.cursor.execute(sql)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 112, in execute
return self.cursor.execute(query, args)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 226, in execute
self.errorhandler(self, exc, value)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorvalue
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 217, in execute
res = self._query(query)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 378, in _query
rowcount = self._do_query(q)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 341, in _do_query
db.query(q)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/connections.py", line 280, in query
_mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (1142, "CREATE command denied to user 'somedbuser'#'somehost' for table 'django_migrations'")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/migrations/recorder.py", line 57, in ensure_schema
editor.create_model(self.Migration)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 295, in create_model
self.execute(sql, params or None)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 112, in execute
cursor.execute(sql, params)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/backends/utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
return self.cursor.execute(sql)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 112, in execute
return self.cursor.execute(query, args)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 226, in execute
self.errorhandler(self, exc, value)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorvalue
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 217, in execute
res = self._query(query)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 378, in _query
rowcount = self._do_query(q)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/cursors.py", line 341, in _do_query
db.query(q)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/MySQLdb/connections.py", line 280, in query
_mysql.connection.query(self, query)
django.db.utils.OperationalError: (1142, "CREATE command denied to user 'somedbuser'#'somehost' for table 'django_migrations'")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/core/management/base.py", line 305, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/core/management/base.py", line 356, in execute
output = self.handle(*args, **options)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/core/management/commands/makemigrations.py", line 100, in handle
loader.check_consistent_history(connection)
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/migrations/loader.py", line 276, in check_consistent_history
applied = recorder.applied_migrations()
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/migrations/recorder.py", line 65, in applied_migrations
self.ensure_schema()
File "/Users/username/Development/stuff/myapp/lib/python3.5/site-packages/django/db/migrations/recorder.py", line 59, in ensure_schema
raise MigrationSchemaMissing("Unable to create the django_migrations table (%s)" % exc)
django.db.migrations.exceptions.MigrationSchemaMissing: Unable to create the django_migrations table ((1142, "CREATE command denied to user 'somedbuser'#'somehost' for table 'django_migrations'"))
It appears that Django tries to create the django_migrations table in the read-only database support nevertheless.
Is there any clean way to prevent the migrations mechanism from attempting that? Or do I have to employ another ORM library for this read-only access to the support database?
I encountered the same issue (using Django 1.11) and this question was at the top of my Google results for it.
Your initial solution is only missing one critical piece. You need to tell Django what database models 'C' and 'D' are using. What worked for me:
class ExternalModel(models.Model):
class Meta:
managed = False
abstract = True
app_label = 'support'
Then tell your database router how to behave when it encounters that app_label in the allow_migrate() section:
def allow_migrate(self, db, app_label, model_name=None, **hints):
if app_label == 'support':
return False
return (db == 'default')
I'm not sure that is the most-correct-solution in the eyes of the Django team, but effect is allow_migrate() returning False for any models defined with that app_label attribute value.
The Django documentation on routers doesn't mention this explicitly (or, at least with model code samples that make it clear how the ORM passes the value for 'db' to allow_migrate()), but between the 'app_label' and 'managed' attributes you can get it to work*.
* In my case the default is postgres and the read-only database is Oracle 12 via cx_Oracle.
It seems around the Django 1.10.1 timeframe, Tim Graham (the primary Django maintainer), accepted a patch that suppressed this specific exception but later withdrew the patch in favor of (roughly) the following method to work around this issue and to support read-only databases using the Django ORM.
Define a database router as described in the Django documentation
on routers I've attached an example router below that routes to a
different database based on an 'app' flag in the model meta.
In your routers allow_migrations method, return False for any db argument
that corresponds to a read-only database. This prevents the migration of
the model tables regardless of where they would be routed to.
This next part is a little weird but where the rubber hits the road and
actually answers the original question. To keep makemigrations from
attempting to create the django_migrations table in your read-only
database, the database traffic should not be routed. In the example
router, that means 'read_only' is not in DATABASE_APPS_MAPPING.
So, instead, Read-only databases are accessed explicitly with "using" (e.g. MyReadOnlyModel.objects.using('read_only').all()
Django database apps router
Had the same problem.
Django is trying to create the 'django_migrations' table in all DBs.
This happens even if there are no models associated with the read-only DB
and all routers are pointing a different DB.
I also ended up using peewee.
I have class method in django model which is using json_array_elements function.
In case when it performs by browser it works fine.
But in tests it fails.
python manage.py test
Traceback (most recent call last):
File "path_to_project/dj_server/model_animations/tests.py", line 94, in test_cteating
response_first = model_animations.views.get_animations_list(request, groupid)
File "path_to_project/dj_server/model_animations/views.py", line 37, in get_animations_list
for model_anim in listArray:
File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 1535, in __iter__
query = iter(self.query)
File "/Library/Python/2.7/site-packages/django/db/models/sql/query.py", line 76, in __iter__
self._execute_query()
File "/Library/Python/2.7/site-packages/django/db/models/sql/query.py", line 90, in _execute_query
self.cursor.execute(self.sql, self.params)
File "/Library/Python/2.7/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/Library/Python/2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/Library/Python/2.7/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
ProgrammingError: function json_array_elements(text) does not exist
LINE 1: ...on_name FROM model_animations_model, json_array...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
in models.py
#classmethod
def animations_list(self, group_id):
if group_id:
try:
listArray = ModelAnimationList.objects.raw(('SELECT * FROM model_animations_model, json_array_elements(animated_groups) AS data WHERE \'"%s"\' = data::text' %group_id))
return listArray
except:
pass
return None
in views.py
def get_animations_list(request, group_id):
...
listArray = ModelAnimationList.animations_list(group_id)
if listArray:
for model_anim in listArray:
if model_anim:
anim_dict = { 'a_id' : model_anim.id, 'a_name' : model_anim.animation_name }
result_anim_list.append(anim_dict)
...
in tests.py
request = HttpRequest()
response_first = model_animations.views.get_animations_list(request, groupid)
Installed:
python 2.7
Django 1.7.1
Postgres 9.3.5
psycopg2 2.5.4
Mac 10.10 yosemite
Mistake in query json_array_elements(animated_groups)
Needs change to this:
json_array_elements(animated_groups::json)
Postgres JSON processing functions such as json_array_elements(animated_groups::json), jsonb_array_elements(animated_groups::jsonb) will take json or jsonb values.
Because postgres JSON field(animated_groups) are store as text format or binary format.
so we need to convert the type by using either ::json or ::jsonb it give the json or jsonb format
check here
I'm having a strange problem right now.
In my program I try to catch an exception inside a class, de class is used in a separate python file but the exception bubbles trough to the calling file.
How can I fix this?
Example:
main.py
#from sqlalchemy.exc import SQLAlchemyError, DatabaseError
from os import path
from helpers.db import DB
from models.group import Group
from models.entity import Entity
PROJECT_PATH = path.split(path.abspath(__file__))[0]
DATABASE_PATH = PROJECT_PATH + '/datastore/test.db'
class PasswordManager:
database = DB(DATABASE_PATH)
if __name__ == "__main__":
PasswordManager()
output
Traceback (most recent call last):
trying to build a SQLEngine
File "/home/tom/Projects/test/test/test.py", line 11, in <module>
class test:
File "/home/tom/Projects/test/test/test.py", line 23, in test
database = DB(DATABASE_PATH)
File "/home/tom/Projects/test/test/helpers/db.py", line 19, in __init__
self.buildDatabaseModels()
File "/home/tom/Projects/test/test/helpers/db.py", line 39, in buildDatabaseModels
Base.metadata.create_all(self.SQLEngine, checkfirst=True)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3308, in create_all
tables=tables)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1516, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1164, in _run_visitor
**kwargs).traverse_single(element)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 119, in traverse_single
return meth(obj, **kw)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 696, in visit_metadata
if self._can_create_table(t)]
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 674, in _can_create_table
table.name, schema=table.schema)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 781, in has_table
cursor = _pragma_cursor(connection.execute(statement))
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 709, in execute
return self._execute_text(object, multiparams, params)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 858, in _execute_text
statement, parameters
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 927, in _execute_context
context)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1076, in _handle_dbapi_exception
exc_info
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 185, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 920, in _execute_context
context)
File "/home/tom/Projects/test/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 425, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (DatabaseError) file is encrypted or is not a database u'PRAGMA table_info("groups")' ()
DB.py
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.exc import DatabaseError, SQLAlchemyError
from models import *
from helpers import Base
from helpers.crypt import Crypt
class DB:
SQLEngine = None
SessionMaker = None
Session = None
BuildEngineErrorCount = 0
def __init__(self, db_path):
self.buildEngine(db_path)
self.buildDatabaseModels()
def buildEngine(self, db_path):
try:
self.SQLEngine = create_engine('sqlite:////' + db_path)
print "trying to build a SQLEngine"
except BaseException, er:
print "Errors %s" % er
for error in er.args:
if error == '(DatabaseError) file is encrypted or is not a database':
self.BuildEngineErrorCount += 1
c = Crypt()
c.setKey('Test')
c.decryptDB(db_path)
if self.BuildEngineErrorCount < 5:
self.buildEngine(db_path)
def buildDatabaseModels(self):
if self.SQLEngine is None:
raise Exception("No SQLEngine found")
Base.metadata.create_all(self.SQLEngine, checkfirst=True)
def createSession(self):
if self.SessionMaker is not None or self.Session is not None:
raise Exception("Session already mapped")
self.SessionMaker = sessionmaker(bind=self.SQLEngine)
self.Session = self.SessionMaker()
return self.Session
As you can see I try to catch that exception in the db class, but it isn't catching anything.
Does anybody know what I do wrong?
Your exception is coming from a different method than the one you put the try-except in. Specifically, the exception comes from
Base.metadata.create_all(self.SQLEngine, checkfirst=True)
in buildDatabaseModels, but you have your try-except in buildEngine.
I have an estrange error testing a resourcemodel in tastypie.
I have this test
class LocationResourceTest(ResourceTestCase):
# Use ``fixtures`` & ``urls`` as normal. See Django's ``TestCase``
# documentation for the gory details.
fixtures = ['admin_user.json']
def runTest(self):
super(LocationResourceTest, self).runTest()
def setUp(self):
super(LocationResourceTest, self).setUp()
# Create a user.
self.user = User.objects.create_user(username='johndoe',email='johndoe#example.com',password='password')
# Create an API key for the user:
ApiKey.objects.create(user=self.user)
def get_credentials(self):
return self.create_apikey(username=self.user.username, api_key=self.user.api_key.key)
def test_create_apikey(self):
# Try api key authentication using ResourceTestCase.create_apikey().
credentials = self.get_credentials()
resp = self.api_client.get('/api/v1/location/',authentication=credentials,format='json')
self.assertHttpOK(resp)
def test_get_list_unauthorzied(self):
pass
When I execute the test,I get the following error
======================================================================
ERROR: test_get_list_unauthorzied (geolocation.tests.api.LocationResourceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/phantomis/Memoria/smartaxi_server/geolocation/tests/api.py", line 24, in setUp
ApiKey.objects.create(user=self.user)
File "/Users/phantomis/Virtualenvs/django-memoria/lib/python2.7/site-packages/django/db/models/manager.py", line 137, in create
return self.get_query_set().create(**kwargs)
File "/Users/phantomis/Virtualenvs/django-memoria/lib/python2.7/site-packages/django/db/models/query.py", line 377, in create
obj.save(force_insert=True, using=self.db)
File "/Users/phantomis/Virtualenvs/django-memoria/lib/python2.7/site-packages/django_tastypie-0.9.12_alpha-py2.7.egg/tastypie/models.py", line 47, in save
return super(ApiKey, self).save(*args, **kwargs)
File "/Users/phantomis/Virtualenvs/django-memoria/lib/python2.7/site-packages/django/db/models/base.py", line 463, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/Users/phantomis/Virtualenvs/django-memoria/lib/python2.7/site-packages/django/db/models/base.py", line 551, in save_base
result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
File "/Users/phantomis/Virtualenvs/django-memoria/lib/python2.7/site-packages/django/db/models/manager.py", line 203, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/Users/phantomis/Virtualenvs/django-memoria/lib/python2.7/site-packages/django/db/models/query.py", line 1593, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/Users/phantomis/Virtualenvs/django-memoria/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 912, in execute_sql
cursor.execute(sql, params)
File "/Users/phantomis/Virtualenvs/django-memoria/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 344, in execute
return Database.Cursor.execute(self, query, params)
IntegrityError: column user_id is not unique
The error is in the method "test_get_list_unauthorzied" , is weird because if i comment that method, my test pass (that method is completely empty)
setUp is run for each of the test_* methods you have. When you comment out the empty test case, setUp is only run once. With the other test_* method not commented out, setUp is run twice, and this is how the uniqueness constraint gets violated.
I would create a tearDown method that deletes the user and api key you created in the other test case.