I am having difficulty understanding Django Multiple Databases documentation. Below is what I am trying to achieve.
I have to migrate some data from one database to another in Python.
Both databases have same structure, so I only have one model file.
What I need to do in code is select data from some tables of one database and insert into tables of another database.
How can I do it, i.e., select in Model query which database to use? Also any suggestions and recommendation would be appreciated.
Thanks
The documentation here https://docs.djangoproject.com/en/1.6/topics/db/multi-db/#manually-selecting-a-database is quite clear.
Assuming you have "db_for_read" and "db_for_write" configured in your settings, for the reading:
YourModel.objects.using("db_for_read").all()
For the writing - per instance:
your_model_instance.save(using="db_for_write")
or in batch:
YourModel.objects.using("db_for_write").bulk_create(
[your_model_instance1, your_model_instance2, etc]
)
Related
I'm using SQL alechmy for a project, I've got a system running with some important data, but I would like to update my schemas to make new features. What's the best practice here?
Can schemas be updated without dropping all tables and recreate database? (currently I am running into trouble with
manage.py migrate
Not working. Ie. Not picking up new changes to table fields)
Thanks a lot in advance,
C
EDIT 28/04/2021
Have no chances with my question ; )
No one deals with this problem?
I want to develop an app for automatic data quality monitoring (DQM) based on different queries:
missing data
outliers
inconsitent data in the same models
incosistent data between models
missing form
User should be able to 'customize' its own queries in a structured settings files (in a excel file loaded in a Django model or directly in a Django model with UI interface).
DQM app should read this setting file and run queries with results stored in a model.
User can edit list of query result and make correction in database in order to resolve and improve data quality.
I look for Django package that could already do that but could not find any. So I would appreciate some help in conception.
I have designed the settings files as below :
I've read about data quality with Pandas bt nothing that cover all data quality queries mentionned above.
Nevertheless, Pandas could be used to read excel settings file using dataframe.
Then, I need to 'construct' query based on the setting file. I thought about 2 options:
use raw: concatenate SQL orders with data read from dataframe and passed SQL order in raw()
use Django queryset with unpack dictionnary to provide keyword arguments : qs = mymodel.objects.filter(**kwargs)
Is there a more clean way to achieve data quality?
I have this excel sheet where i can have as many as 3000-5000 or may be more data( email ids) and i need to insert each of these into my db one by one .I have been asked not to make these many write operations in one go .How should i go about solving this so that i don't do so many entries to database ?Any hint will be highly appreciated .
The solution i could think of is this .
https://www.caktusgroup.com/blog/2019/01/09/django-bulk-inserts/
I'd vote for doing bulk operations withithe django model itself
you can for example do the following with your model class :
... Entry.objects.bulk_create([
... Entry(headline='This is a test'),
... Entry(headline='This is only a test'),
... ])
(code from django website)
you can also you transaction dicorator were you can in sure that a database trasaction is atomic (All or Nothing), thisis handy when you want to do long processeing for your entities or transform them
from django.db import transaction
#transaction.atomic
decorate your method that inserts records into the database
I think that the link you included would work fairly well!
Just wanted to throw one other option out there. Many databases have batch/bulk upload capabilities. Depending on what DB you're using, you could potentially connect directly to it, and perform your writes without Django.
I'm updating from an ancient language to Django. I want to keep the data from the old project into the new.
But old project is mySQL. And I'm currently using SQLite3 in dev mode. But read that postgreSQL is most capable. So first question is: Is it better to set up postgreSQL while in development. Or is it an easy transition to postgreSQL from SQLite3?
And for the data in the old project. I am bumping up the table structure from the old mySQL structure. Since it got many relation db's. And this is handled internally with foreignkey and manytomany in SQLite3 (same in postgreSQL I guess).
So I'm thinking about how to transfer the data. It's not really much data. Maybe 3-5.000 rows.
Problem is that I don't want to have same table structure. So a import would be a terrible idea. I want to have the sweet functionality provided by SQLite3/postgreSQL.
One idea I had was to join all the data and create a nested json for each post. And then define into what table so the relations are kept.
But this is just my guessing. So I'm asking you if there is a proper way to do this?
Thanks!
better create the postgres database. write down the python script which take the data from the mysql database and import in postgres database.
I am a little confused with the topic alluded to in the title.
So, when a Flask app is started, does the SQLAlchemy search theSQLALCHEMY_DATABASE_URI for the correct, in my case, MySQL database. Then, does it create the tables if they do not exist already?
What if the database that is programmed into theSQLALCHEMY_DATABASE_URI variable in the config.py file does not exist?
What if that database exists, and only a few of the tables exist (There are more tables coded into the SQLAlchemy code than exist in the actual MySQL database)? Does it erase those tables and then create new tables with the current specs?
And what if those tables do all exist? Do they get erased and re-created?
I am trying to understand how the entire process works so that I (1) Don't lose database information when changes are made to the schema, and (2) can write the necessary code to completely manage how and when the SQLAlchemy talks to the actual Database.
Tables are not created automatically; you need to call the SQLAlchemy.create_all() method to explicitly to have it create tables for you:
db = SQLAlchemy(app)
db.create_all()
You can do this with command-line utility, for example. Or, if you deploy to a PaaS such as Google App Engine, a dedicated admin-only view.
The same applies for database table destruction; use the SQLAlchemy.drop_all() method.
See the Creating and Dropping tables chapter of the documentation, or take a look at the database chapter of the Mega Flask Tutorial.
You can also delegate this task to Flask-Migrate or similar schema versioning tools. These help you record and edit schema creation and migration steps; the database schema of real-life projects is never static and you would want to be able to move existing data between versions or the schema. Creating the initial schema is then just the first step.