I can't figure out the right ODBC string I need to pass to the create engine statement.
This works
import pyodbc
import pandas as pd
cnxn=pyodbc.connect('DRIVER=/opt/teradata/client/ODBC_64/lib/tdata.so;DBCName=Server;UID=UN;PWD=PW;Database=myDB')
query = "select top 10 * from TABLE"
df = pd.read_sql(query,cnxn)
This does not work
import urllib
import sqlalchemy
params = urllib.parse.quote_plus('DRIVER=/opt/teradata/client/ODBC_64/lib/tdata.so;DBCName=Server;UID=UN;PWD=PW;Database=myDB')
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
query = "select top 10 * from TABLE"
df = pd.read_sql_query(query, engine)
I can get the pyodbc connection to work but not the sqlalchemy connection. Any help would be appreciated.
I get this error:
InterfaceError: (pyodbc.InterfaceError) ('IM001', '[IM001] [unixODBC][Driver Manager]Driver does not support this function (0) (SQLGetInfo)')
#found the answer here. https://downloads.teradata.com/tools/articles/teradata-sqlalchemy-introduction
from sqlalchemy import create_engine
user = 'sqlalc_user'
pasw=user
host = 'hostname'
port = '1025'
# connect
td_engine = create_engine('teradata://'+ user +':' + pasw + '#'+ host + ':22/')
# execute sql
sql = 'select * from dbc.usersV'
result = td_engine.execute(sql)
Related
I am attempting to write a Python script that can take Excel sheets and import them into my SQL Server Express (with Windows Authentication) database as tables. To do this, I am using pandas to read the Excel files into a pandas DataFrame, I then hope to use pandas.to_sql() to import the data into my database. To use this function, however, I need to use sqlalchemy.create_engine().
I am able to connect to my database using pyodbc alone, and run test queries. This conection is done with the followng code:
def create_connection(server_name, database_name):
config = dict(server=server_name, database= database_name)
conn_str = ('SERVER={server};DATABASE={database};TRUSTED_CONNECTION=yes')
return pyodbc.connect(r'DRIVER={ODBC Driver 13 for SQL Server};' + conn_str.format(**config))
...
server = '<MY_SERVER_NAME>\SQLEXPRESS'
db = '<MY_DATABASE_NAME>
connection = create_connection(server, db)
cursor = connection.cursor()
cursor.execute('CREATE VIEW test_view AS SELECT * FROM existing_table')
cursor.commit()
However, this isn't much use as I can't use pandas.to_sql() - to do so I need an engine from sqlalchemy.create_engine(), but I am struggling to figure out how to use my same details in my create_connection() function above to successfully create an engine and connect to the database.
I have tried many, many combinations along the lines of:
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?driver={ODBC Driver 13 for SQL Server}?trusted_connection=yes")
conn = engine.connect().connection
or
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?trusted_connection=yes")
conn = engine.connect().connection
A Pass through exact Pyodbc string works for me:
import pandas as pd
from sqlalchemy import create_engine
from sqlalchemy.engine import URL
connection_string = (
r"Driver=ODBC Driver 17 for SQL Server;"
r"Server=(local)\SQLEXPRESS;"
r"Database=myDb;"
r"Trusted_Connection=yes;"
)
connection_url = URL.create(
"mssql+pyodbc",
query={"odbc_connect": connection_string}
)
engine = create_engine(connection_url)
df = pd.DataFrame([(1, "foo")], columns=["id", "txt"])
pd.to_sql("test_table", engine, if_exists="replace", index=False)
I connect to a database in MS Access using SQL Alchemy:
import urllib
from sqlalchemy import inspect
from sqlalchemy import create_engine
connection_string = (
r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};'
r'DBQ=C:/Users/mvideo/Desktop/dataBase.accdb;'
r'ExtendedAnsiSQL=1;'
)
connection_uri = f"access+pyodbc:///?odbc_connect={urllib.parse.quote_plus(connection_string)}"
engine = create_engine(connection_uri)
print (engine.table_names())
THe result is
['Air', 'regions', 'Soil', 'water']
I get names of all tables. I want to get query from on of this table (with name of water). How should I solve my problem?
from sqlalchemy import text
t = text("SELECT * FROM water")
connection = engine.connect()
result = connection.execute(t)
I have pyodbc code that I use to connect to a DSN, however for some reason it is no longer working and I cannot figure out why (the drivers are empty even though they are there).
So I want to try and convert everything to use SQLAlchemy instead.
My current code for connecting to the database is:
conn = pyodbc.connect('DSN=QueryBuilder')
cursor = conn.cursor()
stringA = "SELECT GrantInformation.Call FROM GrantInformation"
cursor.execute(stringA)
rows = cursor.fetchall()
How would I get this to do the same in SQLAlchemy, I have checked the documentation and I am still confused.
Many thanks
I used:
from sqlalchemy import create_engine
engine = create_engine("""{}://{}:{}#{}/{}"""
.format(SQL Server,nick,mypassword,myservername,querybuilder))
df = pd.read_sql_query("SELECT GrantInformation.Call FROM GrantInformation")
and I got:
File "<ipython-input-5-f7837462519f>", line 4
.format(SQL Server,nick,mypassword,myservername,querybuilder))
^
SyntaxError: invalid syntax
Also declared the variables before, and I now get:
ArgumentError: Could not parse rfc1738 URL from string 'SQL Server://nick:mypassword#myhost/querybuilder'
from sqlalchemy import create_engine
engine = create_engine("""{}://{}:{}#{}/{}"""
.format(driver,user,password,host,database))
df = pd.read_sql_query("SELECT GrantInformation.Call FROM GrantInformation", engine)
Use one of the below code format to create engine
from sqlalchemy import create_engine
# default
engine = create_engine('mysql://scott:tiger#localhost/foo')
# mysql-python
engine = create_engine('mysql+mysqldb://scott:tiger#localhost/foo')
# MySQL-connector-python
engine = create_engine('mysql+mysqlconnector://scott:tiger#localhost/foo')
# OurSQL
engine = create_engine('mysql+oursql://scott:tiger#localhost/foo')
# query
connection = engine.connect()
result = connection.execute("select username from users")
database name = foo, username = scott, password = tiger, host = localhost
Reference: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html
I am attempting to write a Python script that can take Excel sheets and import them into my SQL Server Express (with Windows Authentication) database as tables. To do this, I am using pandas to read the Excel files into a pandas DataFrame, I then hope to use pandas.to_sql() to import the data into my database. To use this function, however, I need to use sqlalchemy.create_engine().
I am able to connect to my database using pyodbc alone, and run test queries. This conection is done with the followng code:
def create_connection(server_name, database_name):
config = dict(server=server_name, database= database_name)
conn_str = ('SERVER={server};DATABASE={database};TRUSTED_CONNECTION=yes')
return pyodbc.connect(r'DRIVER={ODBC Driver 13 for SQL Server};' + conn_str.format(**config))
...
server = '<MY_SERVER_NAME>\SQLEXPRESS'
db = '<MY_DATABASE_NAME>
connection = create_connection(server, db)
cursor = connection.cursor()
cursor.execute('CREATE VIEW test_view AS SELECT * FROM existing_table')
cursor.commit()
However, this isn't much use as I can't use pandas.to_sql() - to do so I need an engine from sqlalchemy.create_engine(), but I am struggling to figure out how to use my same details in my create_connection() function above to successfully create an engine and connect to the database.
I have tried many, many combinations along the lines of:
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?driver={ODBC Driver 13 for SQL Server}?trusted_connection=yes")
conn = engine.connect().connection
or
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?trusted_connection=yes")
conn = engine.connect().connection
A Pass through exact Pyodbc string works for me:
import pandas as pd
from sqlalchemy import create_engine
from sqlalchemy.engine import URL
connection_string = (
r"Driver=ODBC Driver 17 for SQL Server;"
r"Server=(local)\SQLEXPRESS;"
r"Database=myDb;"
r"Trusted_Connection=yes;"
)
connection_url = URL.create(
"mssql+pyodbc",
query={"odbc_connect": connection_string}
)
engine = create_engine(connection_url)
df = pd.DataFrame([(1, "foo")], columns=["id", "txt"])
pd.to_sql("test_table", engine, if_exists="replace", index=False)
I am trying to connect to teradata server and load a dataframe into a table using python. Here is my code -
import sqlalchemy
engine = sqlalchemy.create_engine("teradata://username:passwor#hostname:port/")
f3.to_sql(con=engine, name='sample', if_exists='replace', schema = 'schema_name')
But I am getting the following error -
InterfaceError: (teradata.api.InterfaceError) ('DRIVER_NOT_FOUND', "No driver found for 'Teradata'. Available drivers: SQL Server,SQL Server Native Client 11.0,ODBC Driver 13 for SQL Server")
Can anybody help me to figure out whats wrong in my approach?
There's is different ways to connect to Teradata in Python. The following list is not exhaustive.
SQLAlchemy
If you wish to use SQLAlchemy, you will also need to install the package SQLAlchemy-Teradata. Here is how you can connect:
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base, DeferredReflection
from sqlalchemy.orm import scoped_session, sessionmaker
[...]
# Connect
engine = create_engine('teradata://' + user + ':' + password + '#' + host + ':22/' + database)
db_session = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=engine))
db_session.execute('SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;') # To avoid locking tables when doing select on tables
db_session.commit()
Base = declarative_base(cls=DeferredReflection)
Base.query = db_session.query_property()
Then you can use db_session to make queries. See SQLAlchemy Session API
Pyodbc
If you wish to use Pyodbc you will first need to install Teradata driver on your machine. Example on mine, after installing Teradata driver I have the following entry in /etc/odbcinst.ini
[Teradata]
Driver=/opt/teradata/client/16.00/odbc_64/lib/tdata.so
APILevel=CORE
ConnectFunctions=YYY
DriverODBCVer=3.51
SQLLevel=1
Then I can connect with the following:
import pyodbc
[...]
#Teradata Connection
connection= pyodbc.connect("driver={Teradata};dbcname=" + host + ";uid=" + user + ";pwd=" + pwd + ";charset=utf8;", autocommit=True)
connection.setdecoding(pyodbc.SQL_CHAR, encoding='utf-8')
connection.setdecoding(pyodbc.SQL_WCHAR, encoding='utf-8')
connection.setdecoding(pyodbc.SQL_WMETADATA, encoding='utf-8')
connection.setencoding(encoding='utf-8')
cursor= n.cursor()
cursor.execute("Select 'Hello World'")
for row in cursor:
print (row)
To connect to a teradata database, you need pyodbc, i also have problems with teradata dialect.
Example:
import pyodbc
user = 'user'
pasw = 'pass'
host = 'host'
connection = pyodbc.connect('DRIVER=Teradata;DBCNAME=' + host +';UID=' + user + ';PWD=' + pasw +';QUIETMODE=YES', autocommit=True,unicode_results=True)
I am not sure why your are using sqlalchemy. But you could explore using Teradata module to connect to Teradata as explained in the other link:
Connecting Python with Teradata using Teradata module
I met a similar problem in airflow, I used jars and jaydebeapi to connect teradata database and execute sql:
[root#myhost transfer]# cat test_conn.py
import jaydebeapi
from contextlib import closing
jclassname='com.teradata.jdbc.TeraDriver'
jdbc_driver_loc = '/opt/spark-2.3.1/jars/terajdbc4-16.20.00.06.jar,/opt/spark-2.3.1/jars/tdgssconfig-16.20.00.06.jar'
jdbc_driver_name = 'com.teradata.jdbc.TeraDriver'
host='my_teradata.address'
url='jdbc:teradata://' + host + '/TMODE=TERA'
login="teradata_user_name"
psw="teradata_passwd"
sql = "SELECT COUNT(*) FROM A_TERADATA_TABLE_NAME where month_key='202009'"
conn = jaydebeapi.connect(jclassname=jdbc_driver_name,
url=url,
driver_args=[login, psw],
jars=jdbc_driver_loc.split(","))
with closing(conn) as conn:
with closing(conn.cursor()) as cur:
cur.execute(sql)
print(cur.fetchall())
[root#myhost transfer]# python test_conn.py
[(7734133,)]
[root#myhost transfer]#