How do you prevent Django unittests from running South migrations?
I have a custom Django app, myapp, that I'm trying to test with manage.py test myapp but when I run it I get the error:
django.db.utils.OperationalError: table "myapp_mymodel" already exists
and sure enough, the traceback shows that South is being executed:
File "/usr/local/myproject/.env/local/lib/python2.7/site-packages/south/management/commands/test.py", line 8, in handle
super(Command, self).handle(*args, **kwargs)
However, in my settings, I've specified:
SOUTH_TESTS_MIGRATE = 0
SKIP_SOUTH_TESTS = 1
which I believe should prevent Django's test framework from executing any South components.
What am I doing wrong?
Edit: I worked around this by simply removing south with:
if 'test' in sys.argv:
INSTALLED_APPS.remove('south')
However, then I got:
ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the NAME value.
For my test database settings, I was using:
DATABASES = {
'default':{
'ENGINE': 'django.db.backends.sqlite3'
}
}
which worked fine in Django 1.4. Now I'm using Django 1.5, and I guess that's not kosher. However, no NAME value I see it to fixes it. They all report none of my tables exist. I've tried:
DATABASES = {
'default':{
'ENGINE': 'django.db.backends.sqlite3',
'NAME': '/dev/shm/test.db',
'TEST_NAME': '/dev/shm/test.db',
}
}
DATABASES = {
'default':{
'ENGINE': 'django.db.backends.sqlite3',
'NAME': ':memory:',
'TEST_NAME': ':memory:',
}
}
DATABASES = {
'default':{
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(os.path.dirname(__file__), 'test.db'),
'TEST_NAME': os.path.join(os.path.dirname(__file__), 'test.db'),
}
}
each seems to create a physical test.db file, which I don't understand, because unittests should be run in-memory. It should never save anything to disk. Presumably, it's failing to run syncdb after creating the file but before it executes the actual unittest. How do I fix this?
Edit: I discovered that, in one of my forms, I was populating field choices by directly querying a model (whereas I should have been doing that inside the form's init), so when Django's test framework imported my model, it was trying to read the table before the sqlite3 database had been created. I fixed that, but now I'm getting the error:
DatabaseError: table "myapp_mythroughmodel" already exists
so I'm back to square one, even though it's throwing a different exception type than initially.
Edit: I had a duplicate through model defined, causing Django to attempt to create it twice, resulting in the error.
This also happened to me with a legacy code but for another reason.
I had two models with db_table referencing the same db table.
I know that is stupid, but it's not my fault )
And I never found anything on the internet that could help me.
I was saved by verbosity set to 3 (manage.py test -v 3)
Hope this helps anyone.
class Bla1(Model):
some_column = ...
class Meta:
db_table = 'some_table'
class Bla2(Model):
some_column = ...
class Meta:
db_table = 'some_table'
This error was the result of several problems. I'll summarize them here to help others who may have stumbled across this.
Ensure your settings.DATABASES is set correctly. Django's docs mention using TEST_NAME, but for clarity, I find it easier to check for the test command and override everything. e.g. at the bottom of my settings.py, I have:
if 'test' in sys.argv:
DATABASES = {
'default':{
'ENGINE': 'django.db.backends.sqlite3',
'NAME': ':memory:',
},
}
Unless you have a good reason, always use :memory: to ensure it runs in memory and doesn't create a physical file that will be bogged down on disk. For some odd reason, a lot of other answers on SO recommend specifying a literal path to a test.db file for testing. This is horrible advice.
Unless you want to test South and/or your South migrations, disable South, because it'll only complicate things:
SOUTH_TESTS_MIGRATE = False
SKIP_SOUTH_TESTS = True
Don't be dumb like me and try to access your models before they're created. This mostly means don't directly refer to models from the fields of other models or forms. e.g.
class MyForm(forms.Form):
somefield = forms.ChoiceField(
required=True,
choices=[(_.id, _.name) for _ in OtherModel.objects.filter(criteria=blah)],
)
This might work in code where your database already exists, but it'll break Django's unittest framework when it tries to load your tests, which load your models.py and forms.py, causing it to read a table that doesn't exist. Instead, set the choices value in the form's __init__().
Related
I have a web application in Python django. I need to import users and display data about them from another database, from another existing application. All I need is the user to be able to login and display information about them. What solutions are?
You can set 2 DATABASES in settings.py.
DATABASES = {
'default': {
...
},
'user_data': {
...
}
}
Then in one database store User models with authentication and stuff, in another rest information. You can connect information about specific User with a field that is storing id of User from another database.
If you have multiple databases and create a model, you should declare on which db it is going to be stored. If you didn't, it will be in default one (if you have it declared).
class UserModel(models.Model):
class Meta:
db_table = 'default'
class UserDataModel(models.Model):
class Meta:
db_table = 'user_data'
the answer from #NixonSparrow was wrong.
_meta.db_table defined only table_name in database and not the database self.
for switch database you can use manager.using('database_name'), for every model, it is good declared here: https://docs.djangoproject.com/en/4.0/topics/db/multi-db/#topics-db-multi-db-routing
in my project i use multiple router.
https://docs.djangoproject.com/en/4.0/topics/db/multi-db/#topics-db-multi-db-routing
it help don't override every manager with using. But in your case:
DATABASES = {
'default': {
...
},
'other_users_data': {
...
}
}
and somethere in views:
other_users = otherUserModel.objects.using('other_users_data')
Probably, otherUserModel should define in meta, which table you want to use db_table = 'other_users_table_name' and also probably it should have managed=False, to hide this model from migration manager.
I am working on one Django project. And I decided to write logic code in PostgreSQL instead of writing in Python. So, I created a stored procedure in PostgreSQL. For example, a stored procedure looks like this:
create or replace procedure close_credit(id_loan int)
language plpgsql
as $$
begin
update public.loan_loan
set sum = 0
where id = id_loan;
commit;
end;$$
Then in settings.py, I made the following changes:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'pawnshop',
'USER': 'admin',
'PASSWORD': password.database_password,
'HOST': 'localhost',
'PORT': '',
}
}
So the question is, How can I call this stored procedure in views.py?
p.s.
Maybe it sounds like a dumb question, but I really couldn't find any solution in Django.
I would recommend storing the procedure definition in a migration file. For example, in the directory myapp/migrations/sql.py:
from django.db import migrations
SQL = """
create procedure close_credit(id_loan int)
language plpgsql
as $$
begin
update public.loan_loan
set sum = 0
where id = id_loan;
commit;
end;$$
"""
class Migration(migrations.Migration):
dependencies = [
('myapp', '0001_initial'),
]
operations = [migrations.RunSQL(SQL)]
Note: you will need to replace myapp with the name of your application, and you will need to include only the most recent migration file for your app as a dependency.
Now you can install the procedure using python3 manage.py migrate.
Once your procedure is defined in the database, you can call it using cursor.callproc:
from django.db import connection
def close_credit(id_loan):
with connection.cursor() as cursor:
cursor.callproc('close_credit', [id_loan])
All that being said, if your procedure is really as trivial as the example you provided, it would cost much less in maintenance to write the equivalent using the ORM:
Loan.objects.filter(id=id_loan).update(sum=0)
Suppose I have a django app on my server, but I wish to do authentication using django.contrib.auth.models where the User and Group models/data are on another server in another database. In Django, my DATABASES setting would be something like this:
DATABASES = {
'default': {},
'auth_db': {
'NAME' : 'my_auth_db',
'ENGINE' : 'django.db.backends.mysql',
'USER' : 'someuser',
'PASSWORD' : 'somepassword',
'HOST' : 'some.host.com',
'PORT' : '3306',
},
'myapp': {
'NAME': 'myapp_db',
'ENGINE': 'django.db.backends.mysql',
'USER': 'localuser',
'PASSWORD': 'localpass',
}
}
DATABASE_ROUTERS = ['pathto.dbrouters.AuthRouter', 'pathto.dbrouters.MyAppRouter']
First question: will this work, ie will it allow me to login to my Django app using users that are stored in the remote DB 'my_auth_db'?
Assuming the answer to the above is yes, what happens if in my local DB (app 'myapp') I have models that have a ForeignKey to User? In other words, my model SomeModel is defined in myapp and should exist in the myapp_db, but it have a ForeignKey to a User in my_auth_db:
class SomeModel(models.model):
user = models.ForeignKey(User, unique=False, null=False)
description = models.CharField(max_length=255, null=True)
dummy = models.CharField(max_length=32, null=True)
etc.
Second question: Is this possible or is it simply not possible for one DB table to have a ForeignKey to a table in another DB?
If I really wanted to make this work, could I replace the ForeignKey field 'user' with an IntegerField 'user_id' and then if I needed somemodel.user I would instead get somemodel.user_id and use models.User.objects.get(pk=somemodel.user_id), where the router knows to query auth_db for the User? Is this a viable approach?
The answer to question 1 is: Yes.
What you will need in any case is a database router (The example in the Django docs is exactly about the auth app, so there's no need to copy this code here).
The answer to question 2 is: Maybe. Not officially. It depends on how you have set up MySQL:
https://docs.djangoproject.com/en/dev/topics/db/multi-db/#limitations-of-multiple-databases
Django doesn’t currently provide any support for foreign key or many-to-many relationships spanning multiple databases.
This is because of referential integrity.
However, if you’re using SQLite or MySQL with MyISAM tables, there is no enforced referential integrity; as a result, you may be able to ‘fake’ cross database foreign keys. However, this configuration is not officially supported by Django.
I have a setup with several legacy MySQL DBs (readonly). This answer shows How to use django models with foreign keys in different DBs?
I later ran into troubles with Django ManyToMany through with multiple databases and the solution (as stated in the accepted answer there) is to set the table name with quotes:
class Meta:
db_table = '`%s`.`table2`' % db2_name
Related questions that might provide some additional information:
How to work around lack of support for foreign keys across databases in Django
How to use django models with foreign keys in different DBs?
It would be nice if somebody would take all this information and put in into the official Django doc :-)
I'm trying to set up a LiveServerTestCase.
For this I'm creating a User within my Testclass with
class ExampleTest(LiveServerTestCase):
user = User.objects.create_superuser('Testuser','test#user.com','1234')
user.save()
.
.
.(rest of the test)
without this line the server and test starts but obviously it can't login because there is no User created before.
But with this line I'm getting a
django.core.exceptions.ImproperlyConfigured: settings.DATABASES
is improperly configured. Please supply the NAME value.
error.
Do I need to set up the Server in settings.py for LiveServerTestCase, and in case yes, with which values or where do I find them?
UPDATE:
I'm running this test with
python manage.py test
so it sets up a database itself which I don't have to define in the settings.py, or am I wrong.
UPDATE2:
I already defined a 'production' database (before I even asked the question), it looks like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST': 'localhost', # 10.1.2.41
'NAME': 'pim_testdatabase',
'USER': 'postgres',
'PASSWORD': '1234',
'PORT': '5432',
'HAS_HSTORE': True,
'TEST':{
'NAME': 'test_pim_testdatabase'
},
},
}
Still the exception appears.
You need to set up the database in your DATABASES settings.
Django sets up a test database corresponding to every database that is
defined in the DATABASES definition in your settings file.
By default the test databases get their names by prepending test_ to
the value of the NAME settings for the databases defined in DATABASES.
If you want to use a different database name, specify NAME in the TEST dictionary for any given database in DATABASES.
An example test database configuration:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'USER': 'mydatabaseuser',
'NAME': 'mydatabase',
'TEST': { # test database settings
'NAME': 'mytestdatabase', # test database name
},
},
}
The problem is that you are creating the user in the class definition. This runs when the test class is loaded, before the database has been created.
class ExampleTest(LiveServerTestCase):
user = User.objects.create_superuser('Testuser','test#user.com','1234')
user.save() # This save isn't required -- it has been saved already
You can fix the problem by moving the user creation into an individual test. Then the user will be created when the test method runs, after the database has been created.
class ExampleTest(LiveServerTestCase):
def test_user(self):
self.user = User.objects.create_superuser('Testuser','test#user.com','1234')
...
Django 1.8 has a setUpTestData method where you can set up initial data once for the entire test case. This is quicker, and less repetitive.
class ExampleTest(LiveServerTestCase):
#classmethod
def setUpTestData(cls):
# Set up data for the whole TestCase
self.user = User.objects.create_superuser('Testuser','test#user.com','1234')
def test_user(self):
# access the user with self.user
...
In earlier versions of Django which don't have setUpTestData, you can create the user in the setUp method.
class ExampleTest(LiveServerTestCase):
def setUp(self):
self.user = User.objects.create_superuser('Testuser','test#user.com','1234')
I'm writing a management command where I want to change the default isolation level. Django and my database will default it to "READ COMITTED" and I need it to be "READ UNCOMMITTED" only for this particular management command.
When running:
./manage.py my_command
I've noticed that Django by default opens a transaction with the default isolation level even if your command don't need any database connection:
from django.core.management.base import BaseCommand
class Command(BaseCommand):
help = "Updates assortment data by refreshing the mviews"
def handle(self, *args, **options):
print "fdkgjldfkjgdlkfjgklj"
This behaviour doesn't fit my problem and I'm asking if there is a way of:
Write a management command where Django doesn't even touch the database leaving all the transaction control completely manual?
Write a management command where you can define transaction characteristics only for it?
Regards
I came across your post on Facebook and thought I might be able to be some help :-)
You can specify database connections with read uncommitted with the following database settings in your settings.py:
DATABASES: {
'default': {...}
'uncommitted_db': {
'ENGINE': ...
'NAME': ...
'USER': '...',
'PASSWORD': '...',
'HOST': '...',
'OPTIONS': {
'init_command': 'SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED' #MySQL
'init_command': 'SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED' #Postgres
}
}
}
With this in place you can access your non-transactional database connection using Django's normal multidatabase syntax:
Model.objects.using('uncommitted_db').all()
Of course, you might not want to have your non-transactional database connection globally available in your entire application, so you'd ideally want a way to have it only available during the execution of this management command. Unfortunately, management commands don't work like that: once you hit the handle method on the Command class, your settings.py has already been parsed and your database connections have already been created. If you can find a way to re-initialise Django with a new set of database settings after runtime, or having a logical split in your settings.py based on your launch conditions, like so:
import sys
if 'some_management_cmd' in sys.argv:
DATABASES['default']['OPTIONS']['init_command'] = 'SET TRANSACTION...'
this could work, but is pretty horrible!