Running on Django 3.2 I use dumpdata -o db.json -a to export multiple databases to .json.
Looking into dumpdata.py, it retrieves all objects from a model by calling
queryset = objects.using(using).order_by(model._meta.pk.name)
https://github.com/django/django/blob/main/django/core/management/commands/dumpdata.py, line 185
My problem is that in my case, using is set to 'default' by default, even though I use --all parameter. And later, when calling objects.using(using) it tries to retrieve all objects from default database, even though it's supposed to be 'MIFIR'. What did I do wrong? Have I misconfigured something in my database? I set the app_label in _meta and added my app_label to dbrouter.py, I can see it resolving database name correctly.
Manager, Still tries to use default, Error
Seems that you can use --database to specify a databse.
Also keep in mind that to add new database to Django project you need to create a DBRouter for that database. Not sure, but that might be the problem...
Related
Django 1.7, Python 3.4.
In my models I have several TextFields defined.
When I go to load a JSON fixture (which was generated from an SQLite3 dump), it fails on the second object, which has 515 characters for one of its fields.
The error printed is
psycopg2.DataError: value too long for type character varying(500)
I created a new database (not just a table drop, a whole new db), modified my settings.py file, ran manage.py syncdb on the new database, created a user, and tried to load the data again, getting the same error.
Upon opening pgAdmin3, all columns, both CharField and TextField defined are listed as type character var.
So it seems TextField is being ignored and CharFields are being created instead. The PostgreSQL documentation explicitly lists both text and character types, and defines text as being unlimited in length. Any idea why?
I'm not sure what the exact cause was, but it seems to be related to django's migration tool storing migrations, even on a new database.
What I did to get this behavior:
Create django project, then apps, using CharField
syncdb, run the project's dev server
kill the devserver, modify fields to be TextField
Create a new Postgres database, modify settings.py
Run syncdb, attempt to load fixtures
See the error in question, examine db instance
What fixed the problem:
Create a new database, modify settings.py
delete all migrations in apps/migrations folders
after running syncdb, also run createmigrations and migrate
The last step generated a migration, even though there were none stored in the migrations folder, and there had been no changes to models or data since syncdb was run on the new database, which I found to be odd.
Somewhere in the last two steps this was fixed. Future people stumbling upon this: sorry, I'm not going to keep creating django projects to test the behavior further, but perhaps with this information you can fix your own database problems.
I need to add a couple fields to Group model in django contrib.auth app using:
field_name = models.CharField(...)
field_name.contribute_to_class(Group, 'field_name')
My issue is while creating the migrations with South, since they are created in the "migrations" directory inside the auth app and, since the system is already in production, I'm not allowed to alter the current django installation in the server in order to migrate auth.
Does anyone know how to create and load such migrations?
Thanks in advance for your help.
Django doesn't make it particularly easy to modify the standard models. I wouldn't recommend that you sublass Group, because it's quite annoying to get built-in functionality to reference the new model instead.
The usual thing to do here is to create a GroupProfile model that has a Group as a unique foreign key. It might not be elegant, but it won't have the huge maintenance overhead that comes with forking Django's source code.
Also: if you can't modify the Django code on the server, you aren't going to be able to do this with raw SQL hackery or a clever migration. South isn't going to be the problem -- the problem is that the Django ORM is going to notice that there are fields present in the SQL table that aren't specified in the code, which will cause it to throw an exception.
Since you use a hack to patch up the model, I think you should write a migration manually. Try copying another migration and changing add_column and models first, if it fails - there is always an option named "raw sql" :)
Is it possible to selectively filter which records Django's dumpdata management command outputs? I have a few models, each with millions of rows, and I only want to dump records in one model fitting a specific criteria, as well as all foreign-key linked records referencing any of those records.
Consider this use-case. Say I had a production database where my User model has millions of records. I have several other models (Log, Transaction, Purchase, Bookmarks, etc) all referencing the User model. I want to do development on my Django app, and I want to test using realistic data. However, my production database is so enormous, I can't realistically take a snapshot of the entire thing and load it locally. So ideally, I'd want to use dumpdata to dump 50 random User records, and all related records to JSON, and use that to populate a development database.
Is there an easy way to accomplish this?
I think django-fixture-magic might be worth a look at.
You'll find some additional background info in Scrubbing your Django database.
This snippet might be helpful for you (it follows relationships and serializes them):
http://djangosnippets.org/snippets/918/
You could use also that management command and override the default managers for whichever models you would like to return custom querysets.
This isn't a simple answer to my question, but I found some interesting docs on Django's built-in natural keys feature, which would allow representing serialized records without the primary key. Unfortunately, it doesn't look like this is fully integrated into dumpdata, and there's an old outstanding ticket to fully rely on natural keys.
It also seems the serializers.serialize() function allows serialization of an arbitrary list of specific model instances.
Presumably, if I implemented a natural_key() method on all my models, and then called serializers.serialize([Users.objects.filter(criteria)]), it should come close to accomplishing what I want. I might have to write a function to crawl all the FK references, and include those in the list of objects passed to serialize().
This is a very old question, but I recently wrote a custom management command to do just that. It looks very similar to the existing dumpdata command except that it takes some extra arguments to define how I want to filter the querysets and it overrides the get_objects function to perform the actual filtering:
def get_objects(dump_attributes, dump_values):
qs_1 = ModelClass1.objects.filter(**options["filter_options_for_model_class_1"])
qs_2 = ModelClass2.objects.filter(**options["filter_options_for_model_class_2"])
# ...repeat for as many different model classes you want to dump...
yield from chain(qs_1, qs_2, ...)
I had the same problem but i didn't want to add another package and the snippet still didn't let me to filter my data and i just want a temporary solution
So i thought with my self why not override the default manager apply my filter there, take the dump and then revert my code back. This is of course too hacky and dangerous but in my case made sense.
Yes I had to vim code on live server but you don't need to reload the server since running command through manage.py would run your current code base so the server from the end-user perspective basically remained on-touched.
from django.db.models import Manager
class DahlBookManager(Manager):
def get_queryset(self):
return super().get_queryset().filter(is_edited=False)
class FriendshipQuestion(models.Model):
objects = DahlBookManager()
and then running the dumpdata command did exactly what i needed which was returning all the unedited questions in my case.
Then I git checkout mymodelfile.py to revert it back to the original.
This by no mean is a good solution but it will get somebody either fired or unstuck.
As of Django 3.2, you can use dumpdata to dump a specific app and/or model. For example, for an app named customer:
python manage.py dumpdata customer
or, to dump a model named shoppingcart within the customer app:
python manage.py dumpdata customer.shoppingcart
There are many options with dumpdata, including writing to several output file formats and handling custom managers on models. For example:
python manage.py dumpdata customer --all --indent 4 --output my_fixtures.json
The options:
--all: dumps the records even if you use a custom manager on the model
--indent : amount to indent when writing to file
--output : Send output to a file instead of stdout. Default format is JSON.
See the docs at:
https://docs.djangoproject.com/en/3.2/ref/django-admin/#dumpdata
I have some models I'm working with in a new Django installation. Is it possible to change the fields without losing app data?
I tried changing the field and running python manage.py syncdb. There was no output from this command.
Renavigating to admin pages for editing the changed models caused TemplateSyntaxErrors as Django sought to display fields that didn't exist in the db.
I am using SQLite.
I am able to delete the db file, then re-run python manage.py syncdb, but that is kind of a pain. Is there a better way to do it?
Django does not ever alter an existing database column. Syncdb will create tables, but it does not do 'migrations' as found in Rails, for instance. If you need something like that, check out Django South.
See the docs for syndb:
Syncdb will not alter existing tables
syncdb will only create tables for models which have not yet been installed. It will never issue ALTER TABLE statements to match changes made to a model class after installation. Changes to model classes and database schemas often involve some form of ambiguity and, in those cases, Django would have to guess at the correct changes to make. There is a risk that critical data would be lost in the process.
If you have made changes to a model and wish to alter the database tables to match, use the sql command to display the new SQL structure and compare that to your existing table schema to work out the changes.
You have to change the column names in your DB manually through whatever administration tools sqlite provides. I've done this with MySQL, for instance, and since MySQL lets you change column names without affecting your data, it's no problem.
Of course there is.
Check out South
You'll have to manually update the database schema/layout, if you're only talking about adding/removing columns.
If you're attempting to rename a column, you'll have to find another way.
You can use the python manage.py sql [app name] (http://docs.djangoproject.com/en/dev/ref/django-admin/#sql-appname-appname) command to see what the new SQL should look like, to see what columns, of what type/specification Django would have you add, and then manually run corresponding ALTER TABLE commands.
There are some apps/projects that enable easier model/DB management, but Django doesn't support this out of the box, on purpose/by design.
I'm trying to test my Django apps which run on a PostGIS database, by following the info in the Django testing docs.
Normally I create a new database by copying a template:
(as user postgres)
createdb -T template_postgis -O lizard test_geodjango2
When I run ./manage.py test, I get the following message:
Creating test database...
Got an error creating the test database: permission denied to create database
Type 'yes' if you would like to try deleting the test database 'test_geodjango2', or 'no' to > cancel:
What's the best way to let the system create the database?
It may be that your DATABASE_USER doesn't have permissions to create a new database/schema.
Edit
If you read the source for the Django test command, you'll see that it always creates a test database. Further, it modifies your settings to reference this test database.
See this: http://docs.djangoproject.com/en/dev/topics/testing/#id1
What you should do is use fixtures. Here's how we do it.
From your template database, create a "fixture". Use the manage.py dumpdata command to create a JSON file with all of your template data. [Hint, the --indent=2 option gives you readable JSON that you can edit and modify.]
Put this in a fixtures directory under your application.
Reference the fixtures file in your TestCase class definition. This will load the fixture prior to running the test.
class AnimalTestCase(TestCase):
fixtures = ['mammals.json', 'birds']
def testFluffyAnimals(self):
etc.
The fixtures replace your template database. You don't need the template anymore once you have the fixtures.
As S.Lott mentioned, use the standard test command.
Using geodjango with postgis you'll need to add the following to your settings for the spatial templates to be created properly.
settings.py
POSTGIS_SQL_PATH = 'C:\\Program Files\\PostgreSQL\\8.3\\share\\contrib'
TEST_RUNNER='django.contrib.gis.tests.run_tests'
console
manage.py test
Described here:
http://geodjango.org/docs/testing.html?highlight=testing#testing-geodjango-apps
I haven't looked into it yet, but when I do this I get prompted for the database password when it attempts to install the necessary sql.
As of Django 1.11, Django supports the Postgres-only setting settings.DATABASE[whatever]['TEST']['TEMPLATE'] that dictates what template the test database is created from:
https://docs.djangoproject.com/en/dev/ref/settings/#template