We Keep Coding

Error when trying to write data to a Datamart database using pyodbc [duplicate]

I am attempting to write a Python script that can take Excel sheets and import them into my SQL Server Express (with Windows Authentication) database as tables. To do this, I am using pandas to read the Excel files into a pandas DataFrame, I then hope to use pandas.to_sql() to import the data into my database. To use this function, however, I need to use sqlalchemy.create_engine().
I am able to connect to my database using pyodbc alone, and run test queries. This conection is done with the followng code:
def create_connection(server_name, database_name):
config = dict(server=server_name, database= database_name)
conn_str = ('SERVER={server};DATABASE={database};TRUSTED_CONNECTION=yes')
return pyodbc.connect(r'DRIVER={ODBC Driver 13 for SQL Server};' + conn_str.format(**config))
...
server = '<MY_SERVER_NAME>\SQLEXPRESS'
db = '<MY_DATABASE_NAME>
connection = create_connection(server, db)
cursor = connection.cursor()
cursor.execute('CREATE VIEW test_view AS SELECT * FROM existing_table')
cursor.commit()
However, this isn't much use as I can't use pandas.to_sql() - to do so I need an engine from sqlalchemy.create_engine(), but I am struggling to figure out how to use my same details in my create_connection() function above to successfully create an engine and connect to the database.
I have tried many, many combinations along the lines of:
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?driver={ODBC Driver 13 for SQL Server}?trusted_connection=yes")
conn = engine.connect().connection
or
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?trusted_connection=yes")
conn = engine.connect().connection

A Pass through exact Pyodbc string works for me:
import pandas as pd
from sqlalchemy import create_engine
from sqlalchemy.engine import URL
connection_string = (
r"Driver=ODBC Driver 17 for SQL Server;"
r"Server=(local)\SQLEXPRESS;"
r"Database=myDb;"
r"Trusted_Connection=yes;"
)
connection_url = URL.create(
"mssql+pyodbc",
query={"odbc_connect": connection_string}
)
engine = create_engine(connection_url)
df = pd.DataFrame([(1, "foo")], columns=["id", "txt"])
pd.to_sql("test_table", engine, if_exists="replace", index=False)

Error when creating sqlalchemy engine: Driver keyword syntax error (IM012) [duplicate]

I am attempting to write a Python script that can take Excel sheets and import them into my SQL Server Express (with Windows Authentication) database as tables. To do this, I am using pandas to read the Excel files into a pandas DataFrame, I then hope to use pandas.to_sql() to import the data into my database. To use this function, however, I need to use sqlalchemy.create_engine().
I am able to connect to my database using pyodbc alone, and run test queries. This conection is done with the followng code:
def create_connection(server_name, database_name):
config = dict(server=server_name, database= database_name)
conn_str = ('SERVER={server};DATABASE={database};TRUSTED_CONNECTION=yes')
return pyodbc.connect(r'DRIVER={ODBC Driver 13 for SQL Server};' + conn_str.format(**config))
...
server = '<MY_SERVER_NAME>\SQLEXPRESS'
db = '<MY_DATABASE_NAME>
connection = create_connection(server, db)
cursor = connection.cursor()
cursor.execute('CREATE VIEW test_view AS SELECT * FROM existing_table')
cursor.commit()
However, this isn't much use as I can't use pandas.to_sql() - to do so I need an engine from sqlalchemy.create_engine(), but I am struggling to figure out how to use my same details in my create_connection() function above to successfully create an engine and connect to the database.
I have tried many, many combinations along the lines of:
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?driver={ODBC Driver 13 for SQL Server}?trusted_connection=yes")
conn = engine.connect().connection
or
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?trusted_connection=yes")
conn = engine.connect().connection

A Pass through exact Pyodbc string works for me:
import pandas as pd
from sqlalchemy import create_engine
from sqlalchemy.engine import URL
connection_string = (
r"Driver=ODBC Driver 17 for SQL Server;"
r"Server=(local)\SQLEXPRESS;"
r"Database=myDb;"
r"Trusted_Connection=yes;"
)
connection_url = URL.create(
"mssql+pyodbc",
query={"odbc_connect": connection_string}
)
engine = create_engine(connection_url)
df = pd.DataFrame([(1, "foo")], columns=["id", "txt"])
pd.to_sql("test_table", engine, if_exists="replace", index=False)

How to use SqlAlchemy to connect Database similar to cx_oracle when we use external authorization like wallets with TNS(net service name)

Consider below code:-
Now the below works
connection = cx_Oracle.connect(dsn = 'DSNAME')
But when I use below format for SqlAlchemy it doesn't work, I get TypeError: Invalid arguments dsn passed:
connection = create_engine('oracle+cx_oracle://' , dsn = 'DSNAME')

SQLAlchemy requires a database connection URI, there is an article about it on their documentation. It requires the format
oracle+cx_oracle://user:pass#host:port/dbname[?key=value&key=value...]
Have you tried the following?
connection = create_engine('oracle+cx_oracle://' + 'DSNAME')

it seems pretty old post, I tried with below code, it works . hope this helps. providing empty username/passwd, they are read from wallet, location is mentioned in sqlnet.ora
tnsnames.ora:
t1 = (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ip)(PORT=1521)(KEY=dbpdb1))(CONNECT_DATA=(SERVICE_NAME=dbsvc1.oracle.com))).
sqlnet.ora:
WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA =
(DIRECTORY = $walletdir)
)
)
SQLNET.WALLET_OVERRIDE = TRUE
from sqlalchemy import create_engine
cstr='oracle://:#t1'
print(cstr)
engine = create_engine(
cstr,
convert_unicode=False,
echo=True
)
s='select * from emp'
conn = engine.connect()
result = conn.execute(s)
for row in result:
print (row)

Does SQLAlchemy support Pervasive PSQL? connection string for it?

I googled and found this sqlalchemy-pervasive dialect. I pipped it, but I have no idea how to use it.
What is connection string to pass to create_engine(.....) for sqlalchemy to connect to pervasive psql? is there any other third party driver or dialect?

After I did a:
pip install sqlalchemy-pervasive
This worked for me:
server='localhost'
database='TEST6'
params = urllib.parse.quote_plus(f'DRIVER=Pervasive ODBC Interface;SERVERNAME={server};DBQ={database}')
engine = create_engine("pervasive:///?odbc_connect=%s" % params)

import urllib
import sqlalchemy as sa
params = urllib.parse.quote_plus(f'DRIVER=Pervasive ODBC Interface;SERVERNAME={server}:{port};DBQ={database}')
engine = sa.create_engine("pervasive:///?odbc_connect=%s" % params)
con = engine.connect()
query = 'SELECT * FROM Clientes'
con.execute(query)

Python pandas to_sql 'append'

I am trying to send monthly data to a MySQL database using Python's pandas to_sql command. My program runs one month of data at a time and I want to append the new data onto the existing database. However, Python gives me an error:
_mysql_exceptions.OperationalError: (1050, "Table 'cps_basic_tabulation' already exists")
Here is my code for connecting and exporting:
conn = MySQLdb.connect(host = config.get('db', 'host'),
user = config.get('db', 'user'),
passwd = config.get('db', 'password'),
db = 'cps_raw')
combined.to_sql(name = "cps_raw.cps_basic_tabulation",
con = conn,
flavor = 'mysql',
if_exists = 'append')
I have also tried using:
from sqlalchemy import create_engine
Replacing conn = MySQLdb.connect... with:
engine = mysql+mysqldb://<user>:<password>#<host>[:<port>]/<dbname>
conn = engine.connect().connection
Any ideas on why I cannot append to a database?
Thanks!

Starting from pandas 0.14, you have to provide directly the sqlalchemy engine, and not the connection object:
engine = create_engine("mysql+mysqldb://<user>:<password>#<host>[:<port>]/<dbname>")
combined.to_sql("cps_raw.cps_basic_tabulation", engine, if_exists='append')

Since I had the same error message and stumbled across this post I leave this here for others to find.
I found two ways to solve the duplicated table creation although I lack the insight as to why this solves it:
Either pass the database name in the url when creating a connection
or pass the database name as a schema in pd.to_sql.
Doing both does not hurt. Also, a few years later it is (again?) possible to pass the pure connection to pandas. My guess would be that in the previous answer by joris the first of my solution cases might have implicitly solved the problem.
```
#create connection to MySQL DB via sqlalchemy & pymysql
user = credentials['user']
password = credentials['password']
port = credentials['port']
host = credentials['hostname']
dialect = 'mysql'
driver = 'pymysql'
db_name = 'test_db'
# setup SQLAlchemy
from sqlalchemy import create_engine
cnx = f'{dialect}+{driver}://{user}:{password}#{host}:{port}/'
engine = create_engine(cnx)
# create database
with engine.begin() as con:
con.execute(f"CREATE DATABASE {db_name}")
############################################################
# either pass the db_name vvvv - HERE- vvvv after creating a database
cnx = f'{dialect}+{driver}://{user}:{password}#{host}:{port}/{db_name}'
############################################################
engine = create_engine(cnx)
table = 'test_table'
col = 'test_col'
with engine.begin() as con:
# this would work here instead of creating a new engine with a new link
# con.execute(f"USE {db_name}")
con.execute(f"CREATE TABLE {table} ({col} CHAR(1));")
# insert into database
import pandas as pd
df = pd.DataFrame({col : ['a','b','c']})
with engine.begin() as con:
# this has no effect here
# con.execute(f"USE {db_name}")
df.to_sql(
name= table,
if_exists='append',
# passing con = cnx here would equally work
con=con,
############################################################
# or pass it as a schema vvvv - HERE - vvvv
#schema=db_name,
############################################################
index=False
)```
Tested with python version 3.8.13, sqlalchemy 1.4.32 and pandas 1.4.2.
Same problem might have appeared here and here.