I have a SQL file titled "DreamMarket2017_product.sql". I believe it's MySQL.
How do I read this file into a Jupyter Notebook using PyMySQL? Or, should I use Psycopg2?
I'm much more familiar w/ Psycopg2 than PyMySQL.
Both PyMySQL and Psycopg request a database name. There is no database. I solely have the files.
Do I need to create a database using a GUI like Pgadmin2 and load those the SQL tables into the newly created database?
Also, I'm still waiting to hear from the university that created the dataset.
Yes, u need to create a database and load data into table or import table backup u have
connection = psycopg2.connect(user = "dummy",password = "1234",host = "any",port = "1234",database = "demo")
Related
I'm not sure how to go about doing a one time load of the existing data I have in Oracle to MariaDB. I have DBeaver which I am using to access the databases. I saw an option in DBeaver to migrate the data from Source (Oracle) to Target (MariaDB) with a few clicks, but I'm not sure if that's the best approach.
Is writing a python script a better way of doing it? Should I download another tool to do a one time load? We are using CData Sync to do the incremental loads. Basically, it copies data from one database to another (Oracle to SQL Server for example) and it does incremental loads. I'm not sure if I can use it to do a full time/one time load of all the data I have in my Oracle database to MariaDB. I'm new to this, I've never loaded data before. The thing is, I have over 1100 tables so I can't manually write the schema for each table and do a "CREATE TABLE" statement for all 1100 tables...
Option 1 DBeaver
If DBeaver is willing to try it in a few clicks I'd try and see what it gives for some small tables.
Option 2 MariaDB connect
Alternately there MariaDB connect engine using ODBC or JDBC.
Note you don't need to create table structure for all, but do need the list of table and generate CREATE TABLE t1 ENGINE=CONNECT TABLE_TYPE=ODBC tabname='T1' CONNECTION='DSN=XE;.. for each table.
Then it would be:
create database mariadb_migration;
create table mariadb_migration.t1 like t1;
insert into mariadb_migration.t1 select * from t1;
Option 3 MariaDB Oracle Mode
This uses the Oracle compatibility mode of MariaDB.
Take a SQL dump from Oracle.
Prepend SET SQL_MODE='ORACLE'; to start of the dump.
Import this to MariaDB.
Option 4 SQLines
SQLines offer a Oracle to MariaDB
Small disclaimer, I've not done any of these personally, I just know these options exist.
How to convert a existing postgresql db into a Microsoft Access db with python?
I want to convert my postgresql db into a Microsoft Access db.
There are many possible solutions, like transfer table by table and inside the tables row by row.
But which of the solution mide be the best in terms of performance?
Install the ODBC driver and link the tables from PostgreSQL
Mark the link tables and choose Convert to local table
(Optional) Go to Database Tools, Access Database, and select to split the database to have the tables in an external Access database
I imported a table from sql server database as a dataframe, I am trying to export it as PostgreSQL table
this is what I am doing
from sqlalchemy import create_engine
import psycopg2
engine = create_engine('postgresql://postgres:000000#localhost:5432/sinistrePY')
df.to_sql('table_name3', engine)
and this is the result
the data integration is working fine but
I get the table with read-only privileges
data types are not as I should be
no primary key
I don't need the index column
how can I fix that and control how I want my table to be, from my notebook or directly from PostgreSQL server if needed, thanks.
Using below code to connect to postgres connects me to the default public schema. Where can i mention
the schema name in the connection string? I am trying to insert the data. So when I use dataframe.to_sql('myschema.mytable', engine, if_exists='append', index=False)
It creates a table name with myschema.mytable in public schema instead of inserting the data into mytable which already exist under myschema.
I am using sqlalchemy library in python. Below is my connection string.
engine = create_engine('postgres://user:password#host:5432/dbname')
I tried the jdbc way by appending ?currentSchema=schemaname and ?schema=schemaname but both does not work.
#Belayer, thanks for the response. After some more research, it seems you can mention the schema name while loading the data to database.
dataframe.to_sql('mytable', con=connection, schema='myschema',if_exists='append', index=False)
I have a db file that I would like to open, process the data within and re-save as another db file to be inserted into a MySQLdb database.
I have read that the only way to open a db file is with SQLlite. I'm working in Ubuntu 11.04.
I need to write the process code in Python.
What is the correct conceptual procedure to do this?
I would recommend sqlalchemy for this type of problem. You can use it to
Open your SQLite3 DB and figure out the schema
Save that schema as a sqlalchemy model
< OPTIONAL do any processing you like >
Using the same sqlalchemy model from 1, open a MySQL connection, create the tables and load the data
Note I - you can do all this with the django ORM too - but the SQLAlchemy route will allow you to have less redundant code and more future flexibility.
Note II - sqlautocode can help you with 1.