I saw lots of information about using multiple databases with one server but I wasn't able to find contents about sharing one database with multiple servers.
Using Micro Service Architectures, If I define a database and models in a django server, named Account, How can I use the database and models in Account server from another server named like Post??
What I'm thinking is to write same models.py in both servers and use the django commands --fake
Then, type these commands
python manage.py makemigrations
python manage.py migrate
and in another server
python manage.py makemigrations
python manage.py migrate --fake
I'm not sure if this would work and I wonder whether there is any good ways.
I doubt this is the best approach, but if you want two separate Django projects to use the same database you could probably create the first like normal then, in the second project, copy over all of the models.py and migration files. Django creates a database table behind the scenes to track which migrations have been applied, so as long as the apps, models, and migration files are identical in the second app it should work without having to fake any migrations.
That said, this sounds like a mess to maintain going forward. I think what I would do is create a single Django project that talks to the database, then create an API in that first project that all other apps can interface with to communicate with the database. That way you avoid duplicating code or having to worry about keeping multiple projects in sync.
When using additional Django servers with the same database that is already managed by the initial Django server, the tables don't need to be managed by the additional servers.
So you can add into the Meta for the models that managed = False and Django will not need to touch them, but can still use them. You will need to copy your models across to the additional servers, or use inspectdb (see below).
from django.db import models
class ExampleModel(models.Model):
id = models.IntegerField(db_column='db', primary_key=True)
created = models.DateTimeField(auto_now_add=True)
modified = models.DateTimeField(auto_now=True)
class Meta:
managed = False
db_table = 'example_table'
You will probably need to state the name of the table being referenced in the Meta as well, otherwise Django may generate a name that doesn't match the database.
You can even cut down the models when using them unmanaged.
It's not necessary to declare all the fields, just the ones you're using.
You can also use python manage.py inspectdb to automatically generate unmanaged models for all the tables in your database, saving time and ensuring the model fields conform to the actual database setup.
This is detailed in the documentation:
https://docs.djangoproject.com/en/4.0/ref/models/options/#managed
https://docs.djangoproject.com/en/4.0/ref/django-admin/#inspectdb
In my project, I have the same case that I have 2 Django servers and 1 database.
I did that I run on server 1
python manage.py makemigrations
and
python manage.py migrate
and on server 2 I just run:
python manage.py makemigrations
I did not run migrate commands on server 2
Now if there is any change on model then I run makemigrations command on both servers and migrate command on any of one server. I am using only one database
I used inspectdb to import the Models schema from MySQL database connected with MySQL Connector/Python.
When I run the tests, the system shows:
django.db.utils.ProgrammingError: Table 'test_mydb.cards' doesn't exist
But the table name is just mydb.cards, not test_mydb.cards
Why is this prefix being added? My app's name is just container.
Django uses unittest module for testing, also it creates a blank database for testing as tests should be run ideally always on same blank database or fixtures filled database.
Tests that require a database (namely, model tests) will not use your
“real” (production) database. Separate, blank databases are created
for the tests.
Regardless of whether the tests pass or fail, the test databases are
destroyed when all the tests have been executed.
Check out if you are having migration sorted for the table test tries to access, as all tables that don't have migrations aren't accessible to test suite without monkeypatching or workarounds
I found the reason: I had managed property to False in models.py Class Meta for each table. I never set it there myself, it is the default after doing inspectdb.
Please read: Django Models Options: Managed
I recently switched from Django 1.6 to 1.7, and I began using migrations (I never used South).
Before 1.7, I used to load initial data with a fixture/initial_data.json file, which was loaded with the python manage.py syncdb command (when creating the database).
Now, I started using migrations, and this behavior is deprecated :
If an application uses migrations, there is no automatic loading of fixtures.
Since migrations will be required for applications in Django 2.0, this behavior is considered deprecated. If you want to load initial data for an app, consider doing it in a data migration.
(https://docs.djangoproject.com/en/1.7/howto/initial-data/#automatically-loading-initial-data-fixtures)
The official documentation does not have a clear example on how to do it, so my question is :
What is the best way to import such initial data using data migrations :
Write Python code with multiple calls to mymodel.create(...),
Use or write a Django function (like calling loaddata) to load data from a JSON fixture file.
I prefer the second option.
I don't want to use South, as Django seems to be able to do it natively now.
Update: See #GwynBleidD's comment below for the problems this solution can cause, and see #Rockallite's answer below for an approach that's more durable to future model changes.
Assuming you have a fixture file in <yourapp>/fixtures/initial_data.json
Create your empty migration:
In Django 1.7:
python manage.py makemigrations --empty <yourapp>
In Django 1.8+, you can provide a name:
python manage.py makemigrations --empty <yourapp> --name load_intial_data
Edit your migration file <yourapp>/migrations/0002_auto_xxx.py
2.1. Custom implementation, inspired by Django' loaddata (initial answer):
import os
from sys import path
from django.core import serializers
fixture_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../fixtures'))
fixture_filename = 'initial_data.json'
def load_fixture(apps, schema_editor):
fixture_file = os.path.join(fixture_dir, fixture_filename)
fixture = open(fixture_file, 'rb')
objects = serializers.deserialize('json', fixture, ignorenonexistent=True)
for obj in objects:
obj.save()
fixture.close()
def unload_fixture(apps, schema_editor):
"Brutally deleting all entries for this model..."
MyModel = apps.get_model("yourapp", "ModelName")
MyModel.objects.all().delete()
class Migration(migrations.Migration):
dependencies = [
('yourapp', '0001_initial'),
]
operations = [
migrations.RunPython(load_fixture, reverse_code=unload_fixture),
]
2.2. A simpler solution for load_fixture (per #juliocesar's suggestion):
from django.core.management import call_command
fixture_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../fixtures'))
fixture_filename = 'initial_data.json'
def load_fixture(apps, schema_editor):
fixture_file = os.path.join(fixture_dir, fixture_filename)
call_command('loaddata', fixture_file)
Useful if you want to use a custom directory.
2.3. Simplest: calling loaddata with app_label will load fixtures from the <yourapp>'s fixtures dir automatically :
from django.core.management import call_command
fixture = 'initial_data'
def load_fixture(apps, schema_editor):
call_command('loaddata', fixture, app_label='yourapp')
If you don't specify app_label, loaddata will try to load fixture filename from all apps fixtures directories (which you probably don't want).
Run it
python manage.py migrate <yourapp>
Short version
You should NOT use loaddata management command directly in a data migration.
# Bad example for a data migration
from django.db import migrations
from django.core.management import call_command
def load_fixture(apps, schema_editor):
# No, it's wrong. DON'T DO THIS!
call_command('loaddata', 'your_data.json', app_label='yourapp')
class Migration(migrations.Migration):
dependencies = [
# Dependencies to other migrations
]
operations = [
migrations.RunPython(load_fixture),
]
Long version
loaddata utilizes django.core.serializers.python.Deserializer which uses the most up-to-date models to deserialize historical data in a migration. That's incorrect behavior.
For example, supposed that there is a data migration which utilizes loaddata management command to load data from a fixture, and it's already applied on your development environment.
Later, you decide to add a new required field to the corresponding model, so you do it and make a new migration against your updated model (and possibly provide a one-off value to the new field when ./manage.py makemigrations prompts you).
You run the next migration, and all is well.
Finally, you're done developing your Django application, and you deploy it on the production server. Now it's time for you to run the whole migrations from scratch on the production environment.
However, the data migration fails. That's because the deserialized model from loaddata command, which represents the current code, can't be saved with empty data for the new required field you added. The original fixture lacks necessary data for it!
But even if you update the fixture with required data for the new field, the data migration still fails. When the data migration is running, the next migration which adds the corresponding column to the database, is not applied yet. You can't save data to a column which does not exist!
Conclusion: in a data migration, the loaddata command introduces potential inconsistency between the model and the database. You should definitely NOT use it directly in a data migration.
The Solution
loaddata command relies on django.core.serializers.python._get_model function to get the corresponding model from a fixture, which will return the most up-to-date version of a model. We need to monkey-patch it so it gets the historical model.
(The following code works for Django 1.8.x)
# Good example for a data migration
from django.db import migrations
from django.core.serializers import base, python
from django.core.management import call_command
def load_fixture(apps, schema_editor):
# Save the old _get_model() function
old_get_model = python._get_model
# Define new _get_model() function here, which utilizes the apps argument to
# get the historical version of a model. This piece of code is directly stolen
# from django.core.serializers.python._get_model, unchanged. However, here it
# has a different context, specifically, the apps variable.
def _get_model(model_identifier):
try:
return apps.get_model(model_identifier)
except (LookupError, TypeError):
raise base.DeserializationError("Invalid model identifier: '%s'" % model_identifier)
# Replace the _get_model() function on the module, so loaddata can utilize it.
python._get_model = _get_model
try:
# Call loaddata command
call_command('loaddata', 'your_data.json', app_label='yourapp')
finally:
# Restore old _get_model() function
python._get_model = old_get_model
class Migration(migrations.Migration):
dependencies = [
# Dependencies to other migrations
]
operations = [
migrations.RunPython(load_fixture),
]
Inspired by some of the comments (namely n__o's) and the fact that I have a lot of initial_data.* files spread out over multiple apps I decided to create a Django app that would facilitate the creation of these data migrations.
Using django-migration-fixture you can simply run the following management command and it will search through all your INSTALLED_APPS for initial_data.* files and turn them into data migrations.
./manage.py create_initial_data_fixtures
Migrations for 'eggs':
0002_auto_20150107_0817.py:
Migrations for 'sausage':
Ignoring 'initial_data.yaml' - migration already exists.
Migrations for 'foo':
Ignoring 'initial_data.yaml' - not migrated.
See django-migration-fixture for install/usage instructions.
In order to give your database some initial data, write a data migration.
In the data migration, use the RunPython function to load your data.
Don't write any loaddata command as this way is deprecated.
Your data migrations will be run only once. The migrations are an ordered sequence of migrations. When the 003_xxxx.py migrations is run, django migrations writes in the database that this app is migrated until this one (003), and will run the following migrations only.
The solutions presented above didn't work for me unfortunately. I found that every time I change my models I have to update my fixtures. Ideally I would instead write data migrations to modify created data and fixture-loaded data similarly.
To facilitate this I wrote a quick function which will look in the fixtures directory of the current app and load a fixture. Put this function into a migration in the point of the model history that matches the fields in the migration.
On Django 2.1, I wanted to load some models (Like country names for example) with initial data.
But I wanted this to happen automatically right after the execution of initial migrations.
So I thought that it would be great to have a sql/ folder inside each application that required initial data to be loaded.
Then within that sql/ folder I would have .sql files with the required DMLs to load the initial data into the corresponding models, for example:
INSERT INTO appName_modelName(fieldName)
VALUES
("country 1"),
("country 2"),
("country 3"),
("country 4");
To be more descriptive, this is how an app containing a sql/ folder would look:
Also I found some cases where I needed the sql scripts to be executed in a specific order. So I decided to prefix the file names with a consecutive number as seen in the image above.
Then I needed a way to load any SQLs available inside any application folder automatically by doing python manage.py migrate.
So I created another application named initial_data_migrations and then I added this app to the list of INSTALLED_APPS in settings.py file. Then I created a migrations folder inside and added a file called run_sql_scripts.py (Which actually is a custom migration). As seen in the image below:
I created run_sql_scripts.py so that it takes care of running all sql scripts available within each application. This one is then fired when someone runs python manage.py migrate. This custom migration also adds the involved applications as dependencies, that way it attempts to run the sql statements only after the required applications have executed their 0001_initial.py migrations (We don't want to attempt running a SQL statement against a non-existent table).
Here is the source of that script:
import os
import itertools
from django.db import migrations
from YourDjangoProjectName.settings import BASE_DIR, INSTALLED_APPS
SQL_FOLDER = "/sql/"
APP_SQL_FOLDERS = [
(os.path.join(BASE_DIR, app + SQL_FOLDER), app) for app in INSTALLED_APPS
if os.path.isdir(os.path.join(BASE_DIR, app + SQL_FOLDER))
]
SQL_FILES = [
sorted([path + file for file in os.listdir(path) if file.lower().endswith('.sql')])
for path, app in APP_SQL_FOLDERS
]
def load_file(path):
with open(path, 'r') as f:
return f.read()
class Migration(migrations.Migration):
dependencies = [
(app, '__first__') for path, app in APP_SQL_FOLDERS
]
operations = [
migrations.RunSQL(load_file(f)) for f in list(itertools.chain.from_iterable(SQL_FILES))
]
I hope someone finds this helpful, it worked just fine for me!. If you have any questions please let me know.
NOTE: This might not be the best solution since I'm just getting started with Django, however I still wanted to share this "How-to" with you all since I didn't find much information while googling about this.
In my opinion fixtures are a bit bad. If your database changes frequently, keeping them up-to-date will came a nightmare soon. Actually, it's not only my opinion, in the book "Two Scoops of Django" it's explained much better.
Instead I'll write a Python file to provide initial setup. If you need something more I'll suggest you look at Factory boy.
If you need to migrate some data you should use data migrations.
There's also "Burn Your Fixtures, Use Model Factories" about using fixtures.
Although #rockallite's answer is excellent, it does not explain how to handle fixtures that rely on natural keys instead of integer pk values.
Simplified version
First, note that #rockallite's solution can be simplified by using unittest.mock.patch as a context manager, and by patching apps instead of _get_model:
...
from unittest.mock import patch
...
def load_fixture(apps, schema_editor):
with patch('django.core.serializers.python.apps', apps):
call_command('loaddata', 'your_data.json', ...)
...
This works well, as long as your fixtures do not rely on natural keys.
If they do, you're likely to see a DeserializationError: ... value must be an integer....
The problem with natural keys
Under the hood, loaddata uses django.core.serializers.deserialize() to load your fixture objects.
The deserialization of fixtures based on natural keys relies on two things:
the presence of a get_by_natural_key() method on the model's default manager
the presence of a natural_key() method on the model itself
The get_by_natural_key() method is necessary for the deserializer to know how to interpret the natural key, instead of an integer pk value.
Both methods are necessary for the deserializer to get existing objects from the database by natural key, as also explained here.
However, the apps registry which is available in your migrations uses historical models, and these do not have access to custom managers or custom methods such as natural_key().
Possible solution: step 1
The problem of the missing get_by_natural_key() method from our custom model manager is relatively easy to solve:
Just set use_in_migrations=True on your custom manager, as described in the documentation.
This ensures that your historical models can access the current get_by_natural_key() during migrations, and fixture loading should now succeed.
However, your historical models still don't have a natural_key() method. As a result, your fixtures will be treated as new objects, even if they are already present in the database.
This may lead to a variety of errors if the data-migration is ever re-applied, such as:
unique-constraint violations (if your models have unique-constraints)
duplicate fixture objects (if your models do not have unique-constraints)
"get returned multiple objects" errors (due to duplicate fixture objects created previously)
So, effectively, you're still missing out on a kind of get_or_create-like behavior during deserialization.
To experience this, just apply a data-migration as described above (in a test environment), then roll back the same data-migration (without removing the data), then re-apply the data-migration.
Possible solution: step 2
The problem of the missing natural_key() method from the model itself is a bit more difficult to solve.
One solution would be to assign the natural_key() method from the current model to the historical model, for example:
...
from unittest.mock import patch
from django.apps import apps as current_apps
from django.core.management import call_command
...
def load_fixture(apps, schema_editor):
def _get_model_patch(app_label):
""" add natural_key method from current model to historical model """
historical_model = apps.get_model(app_label=app_label)
current_model = current_apps.get_model(app_label=app_label)
historical_model.natural_key = current_model.natural_key
return historical_model
with patch('django.core.serializers.python._get_model', _get_model_patch):
call_command('loaddata', 'your_data.json', ...)
...
Notes:
For clarity, I omitted things like error handling and attribute checking from the example. You should implement those where necessary.
This solution uses the current model's natural_key method, which may still lead to trouble in certain scenarios, but the same goes for Django's use_in_migrations option for model managers.
I used to have a standalone script with some unit tests to test data in our database. I did not use the builtin Django testing tool, as that would create an empty testing database, which is not what I want.
In that script, I created three different classes extending unittest.TestCase containing some test functions that directly executed SQL statements.
Now I would prefer to be able to access the Django ORM directly. The easiest way to do this is via a custom management commant (./manage.py datatests).
In the standalone script, I could call all unit tests via the following function:
if __name__ == '__main__':
unittest.main()
It would discover all tests in the current file and run them.
How can I do an equivalent thing (run some test suites) from within a custom Django management command?
I'm sorry for not having searched for an answer long enough before asking, but I found the solution to this problem myself in another Stackoverflow answer:
How to run django unit-tests on production database?
Essentially, instead of unittest.main() the following code can be used:
suite = unittest.TestLoader().loadTestsFromTestCase(TestCaseClass)
unittest.TextTestRunner(verbosity=2).run(suite)
This will load all tests in the specified TestCaseClass. If you want to load all tests in the current module, creating the suite this way will help:
suite = TestLoader().loadTestsFromName(__name__)
The Stackoverflow answer linked above contains a full example. Furthermore, the Basic Example section of the unittest module docs describes the same thing. For other options to load tests, see Loading and running tests in the docs.
You may want to specify the contents of your start-up db through fixtures. It will load up the context for db for particular test. And you can take a snapshot of db with
$ ./manage.py dumpdata my_app > fixtures/my_pre_test_db.json`
Now in your unit test you will have something like this:
class MyTestCase(TestCase):
fixtures = ['fixtures/my_pre_test_db.json']
def testThisFeature(self):
...
I'm trying to test my Django apps which run on a PostGIS database, by following the info in the Django testing docs.
Normally I create a new database by copying a template:
(as user postgres)
createdb -T template_postgis -O lizard test_geodjango2
When I run ./manage.py test, I get the following message:
Creating test database...
Got an error creating the test database: permission denied to create database
Type 'yes' if you would like to try deleting the test database 'test_geodjango2', or 'no' to > cancel:
What's the best way to let the system create the database?
It may be that your DATABASE_USER doesn't have permissions to create a new database/schema.
Edit
If you read the source for the Django test command, you'll see that it always creates a test database. Further, it modifies your settings to reference this test database.
See this: http://docs.djangoproject.com/en/dev/topics/testing/#id1
What you should do is use fixtures. Here's how we do it.
From your template database, create a "fixture". Use the manage.py dumpdata command to create a JSON file with all of your template data. [Hint, the --indent=2 option gives you readable JSON that you can edit and modify.]
Put this in a fixtures directory under your application.
Reference the fixtures file in your TestCase class definition. This will load the fixture prior to running the test.
class AnimalTestCase(TestCase):
fixtures = ['mammals.json', 'birds']
def testFluffyAnimals(self):
etc.
The fixtures replace your template database. You don't need the template anymore once you have the fixtures.
As S.Lott mentioned, use the standard test command.
Using geodjango with postgis you'll need to add the following to your settings for the spatial templates to be created properly.
settings.py
POSTGIS_SQL_PATH = 'C:\\Program Files\\PostgreSQL\\8.3\\share\\contrib'
TEST_RUNNER='django.contrib.gis.tests.run_tests'
console
manage.py test
Described here:
http://geodjango.org/docs/testing.html?highlight=testing#testing-geodjango-apps
I haven't looked into it yet, but when I do this I get prompted for the database password when it attempts to install the necessary sql.
As of Django 1.11, Django supports the Postgres-only setting settings.DATABASE[whatever]['TEST']['TEMPLATE'] that dictates what template the test database is created from:
https://docs.djangoproject.com/en/dev/ref/settings/#template