I need to migrate information that I created in SQLite3 to a MySQL database on my website. The website is on a hosted server. I have remote access to the MySQL database. I initially thought it would be easy, but I am not finding any good info on it, and everything I read seems to imply that you need to dump the SQLite3 file, convert it to a MySQL dump file using messy scripts, and then import it into the MySQL.
(example: Quick easy way to migrate SQLite3 to MySQL?)
My question: Is it not better to read it and import it straight from the script into MySQL. I haven't used MySQL at all with Python, but it would seem intuitive that it would be better to have less steps for things to be miss-read. I am also trying to save a little time by not having to understand the way that a MySQL dump file works.
If you have a script (Python if possible), tool or link that will help me out that would be great, but my main concern first of all is how to go about doing it. I can't realistically check everything (otherwise I would just do copy and paste), so I want to be sure that I am going about it the right way.
I am using Django, perhaps there is a Django specific method for it, but I haven't been able to find one.
The reason the "messy" scripts are required is that it's generally a difficult problem to solve.
If you are lucky there won't be too many schema incompatibilities between the databases.
These may help
db_dump.py
dbpickle.py
What is your favorite solution for managing database migrations in django?
Have you tried using django-admin's dumpdata and loaddata?
e.g. dump sqllite3 db
manage.py dumpdata
change db to mysql, recreate tables and load
manage.py loaddata mydata.json
dumpdata by default dumps json but you can set format to xml too.
Alternatively, you can just read whole sqlite3 db using python db api or sqlalchemy or django ORM and dump it to MySQL db.
Related
Hello I am creating a small program on Python with the cx_oracle module which allows me to connect to my Oracle database on my Computer. However I would like to send it to a friend and therefore I would like him to be able to handle the same database as me. So I thought of an embeddable database (a bit like a file on SQLite) but with Oracle I did not find such a possibility. I would like to know if there is a way to do it with Oracle or if I am forced to connect to a local database.
First of all, you can export and import Oracle databases, which would help you a lot at the initial sharing. However, if you share your database with a friend and both of you are working on the same database but with different copies, then the two databases will eventually diverge from each other, with ever greater impact. So, you will need to consider your options carefully:
Using a central server
You could use a central server (which could be a remote server or even your computer if you apply port forwarding), ensure that both you and your friend are connected to that database and then both your and his changes will automatically be applied to the same database, without copies.
Versioned dumps
You could use a versioning tool, like git to store the versions of your database dump/structure/data and both you and your friend could use this, maybe storing the versions in a central repositories, so you would not need to send and communicate your database changes again and again. This would ensure schema and data synchronization, albeit you will have frequent merge conflicts and other merge-related problems.
Versioned scripts
You and your friend could write versioned scripts. This would apply on structural changes, so your and your friend's test data would diverge, but the structure would not.
Migration scripts
Some ORMs have automatically generated migration scripts and one can go forwards or backwards some levels. I do not particularly like the idea of automatically generated change scripts, but it is certainly a possible solution.
So I recently changed databases on a Django-based website from SQLite to MySQL.
I did it via dumping the data from the SQLite into a json file and then migrating it to the new MySQL db then loaded it using the python manage.py loaddata command.
The site is hosted on pythonanywhere and the tranistion there went flawless, however, on the local copy of the project I have(used to test implementing new features and debug), the dynamic elements on the front page wont load.
I have a dynamic js table with information for various cryptocurrencies.
This is how the head of the table looks on the live site:
And this is how it looks on the local copy of my machine, with my own hosted MySQL server:
What I've tried:
-Collectstatic
-Running ContentType.objects.all().delete() in python manage.py shell
I also did research on stackoverflow but I could not find a similar issue.
I am also new to Django so it might be something obvious I'm missing
Try with Django fixtures.
Here is the official link for Django fixtures
Solved: I used the datadump.json from pythonanywhere rather than my local copy. Also had some issues with the json utf8 encodings so make sure your datadump file is utf8 encoded and/or matches the encoding of your db.
Is there a way to run a python script that will add data to this SQLite DB then? I mean, a completely separate .py file that I will run in the shell, doing python thefile.py in an SSH connection?
Of course, this file will be on the same server as the whole Django project.
This way, I can easily access the data from the DB from my Django project, and display it the way I want in a pretty HTML/CSS web page.
Thanks
Is there a reason for not using django? you can add a command to the manage.py the way it's described here https://docs.djangoproject.com/en/1.11/howto/custom-management-commands/
I would like to make a Database editable with LibreOffice Base and usable with python. I can't find a way with the normal HSQLDB as it requires Java (I would as less as possible dependencies) and the same thing with SQLite3 as it requires the drivers for LibreOffice.
Be sure to use a split database setup, rather than the default embedded setup. Otherwise it will crash a lot.
One solution that does not require Java is to switch to a different DB engine, for example MySQL. With this setup, see How do I connect to a MySQL Database in Python? LibreOffice Base works well with MySQL.
See also https://wiki.openoffice.org/wiki/FAQ_(Base)#Do_I_need_Java_to_use_Base.3F. Split databases are discussed on that page as well.
I could create tables using the command alembic revision -m 'table_name' and then defining the versions and migrate using alembic upgrade head.
Also, I could create tables in a database by defining a class in models.py (SQLAlchemy).
What is the difference between the two? I'm very confused. Have I messed up the concept?
Also, when I migrate the database using Alembic, why doesn't it form a new class in my models.py? I know the tables have been created because I checked them using a SQLite browser.
I have done all the configurations already. The target for Alembic's database and SQLALCHEMY_DATABASE-URI in config.py are the same .db file.
Yes, you are thinking about it in the wrong way.
Let's say you don't use Alembic or any other migration framework. In that case you create a new database for your application with the following steps:
Write your model classes
Create and configure a brand new database
Run db.create_all(), which looks at your models and creates the corresponding tables in your database.
So now consider the case of an upgrade. For example, let's say you release version 1.0 of your application and now start working on version 2.0, which requires some changes to your database. How can you achieve that? The limitation here is that db.create_all() does not modify tables, it can only create them from scratch. So it goes like this:
Make the necessary changes to your model classes
Now you have two options to transfer those changes to the database:
5.1 Destroy the database so that you can run db.create_all() again to get the updated tables, maybe backing up and restoring the data so that you don't lose it. Unfortunately SQLAlchemy does not help with the data, you'll have to use database tools for that.
5.2 Apply the changes manually, directly to the database. This is error prone, and it would be tedious if the change set is large.
Now consider that you have development and production databases, that means the work needs to be done twice. Also think about how tedious would it be when you have several releases of your application, each with a different database schema and you need to investigate a bug in one of the older releases, for which you need to recreate the database as it was in that release.
See what the problem is when you don't have a migration network?
Using Alembic, you have a little bit of extra work when you start, but it pays off because it simplifies your workflow for your upgrades. The creation phase goes like this:
Write your model classes
Create and configure a brand new database
Generate an initial Alembic migration, either manually or automatically. If you go with automatic migrations, Alembic looks at your models and generates the code that applies those to the database.
Run the upgrade command, which runs the migration script, effectively creating the tables in your database.
Then when you reach the point of doing an upgrade, you do the following:
Make the necessary changes to your model classes
Generate another Alembic migration. If you let Alembic generate this for you, then it compares your models classes against the current schema in your database, and generates the code necessary to make the database match the models.
Run the upgrade command. This applies the changes to the database, without the need to destroy any tables or back up data. You can run this upgrade on all your databases (production, development, etc.).
Important things to consider when using Alembic:
The migration scripts become part of your source code, so they need to be committed to source control along with your own files.
If you use the automatic migration generation, you always have to review the generated migrations. Alembic is not always able to determine the exact changes, so it is possible that the generated script needs some manual fine tuning.
Migration scripts have upgrade and downgrade functions. That means that they not only simplify upgrades, but also downgrades. If you need to sync the database to an old release, the downgrade command does it for you without any additional work on your part!
I can add that for Django there are two commands
makemigrations (which creates migrations files)
migrate (which translates migrations into sql and executes them on database)
I found its great for somebody's understanding to switch between batteries included framework(Django) and other frameworks like Flask/ Falcon.
So using alembic migrations is very convenient, and makes easy to track database changes.