I am trying to connect to teradata server and load a dataframe into a table using python. Here is my code -
import sqlalchemy
engine = sqlalchemy.create_engine("teradata://username:passwor#hostname:port/")
f3.to_sql(con=engine, name='sample', if_exists='replace', schema = 'schema_name')
But I am getting the following error -
InterfaceError: (teradata.api.InterfaceError) ('DRIVER_NOT_FOUND', "No driver found for 'Teradata'. Available drivers: SQL Server,SQL Server Native Client 11.0,ODBC Driver 13 for SQL Server")
Can anybody help me to figure out whats wrong in my approach?
There's is different ways to connect to Teradata in Python. The following list is not exhaustive.
SQLAlchemy
If you wish to use SQLAlchemy, you will also need to install the package SQLAlchemy-Teradata. Here is how you can connect:
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base, DeferredReflection
from sqlalchemy.orm import scoped_session, sessionmaker
[...]
# Connect
engine = create_engine('teradata://' + user + ':' + password + '#' + host + ':22/' + database)
db_session = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=engine))
db_session.execute('SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;') # To avoid locking tables when doing select on tables
db_session.commit()
Base = declarative_base(cls=DeferredReflection)
Base.query = db_session.query_property()
Then you can use db_session to make queries. See SQLAlchemy Session API
Pyodbc
If you wish to use Pyodbc you will first need to install Teradata driver on your machine. Example on mine, after installing Teradata driver I have the following entry in /etc/odbcinst.ini
[Teradata]
Driver=/opt/teradata/client/16.00/odbc_64/lib/tdata.so
APILevel=CORE
ConnectFunctions=YYY
DriverODBCVer=3.51
SQLLevel=1
Then I can connect with the following:
import pyodbc
[...]
#Teradata Connection
connection= pyodbc.connect("driver={Teradata};dbcname=" + host + ";uid=" + user + ";pwd=" + pwd + ";charset=utf8;", autocommit=True)
connection.setdecoding(pyodbc.SQL_CHAR, encoding='utf-8')
connection.setdecoding(pyodbc.SQL_WCHAR, encoding='utf-8')
connection.setdecoding(pyodbc.SQL_WMETADATA, encoding='utf-8')
connection.setencoding(encoding='utf-8')
cursor= n.cursor()
cursor.execute("Select 'Hello World'")
for row in cursor:
print (row)
To connect to a teradata database, you need pyodbc, i also have problems with teradata dialect.
Example:
import pyodbc
user = 'user'
pasw = 'pass'
host = 'host'
connection = pyodbc.connect('DRIVER=Teradata;DBCNAME=' + host +';UID=' + user + ';PWD=' + pasw +';QUIETMODE=YES', autocommit=True,unicode_results=True)
I am not sure why your are using sqlalchemy. But you could explore using Teradata module to connect to Teradata as explained in the other link:
Connecting Python with Teradata using Teradata module
I met a similar problem in airflow, I used jars and jaydebeapi to connect teradata database and execute sql:
[root#myhost transfer]# cat test_conn.py
import jaydebeapi
from contextlib import closing
jclassname='com.teradata.jdbc.TeraDriver'
jdbc_driver_loc = '/opt/spark-2.3.1/jars/terajdbc4-16.20.00.06.jar,/opt/spark-2.3.1/jars/tdgssconfig-16.20.00.06.jar'
jdbc_driver_name = 'com.teradata.jdbc.TeraDriver'
host='my_teradata.address'
url='jdbc:teradata://' + host + '/TMODE=TERA'
login="teradata_user_name"
psw="teradata_passwd"
sql = "SELECT COUNT(*) FROM A_TERADATA_TABLE_NAME where month_key='202009'"
conn = jaydebeapi.connect(jclassname=jdbc_driver_name,
url=url,
driver_args=[login, psw],
jars=jdbc_driver_loc.split(","))
with closing(conn) as conn:
with closing(conn.cursor()) as cur:
cur.execute(sql)
print(cur.fetchall())
[root#myhost transfer]# python test_conn.py
[(7734133,)]
[root#myhost transfer]#
Related
I am attempting to write a Python script that can take Excel sheets and import them into my SQL Server Express (with Windows Authentication) database as tables. To do this, I am using pandas to read the Excel files into a pandas DataFrame, I then hope to use pandas.to_sql() to import the data into my database. To use this function, however, I need to use sqlalchemy.create_engine().
I am able to connect to my database using pyodbc alone, and run test queries. This conection is done with the followng code:
def create_connection(server_name, database_name):
config = dict(server=server_name, database= database_name)
conn_str = ('SERVER={server};DATABASE={database};TRUSTED_CONNECTION=yes')
return pyodbc.connect(r'DRIVER={ODBC Driver 13 for SQL Server};' + conn_str.format(**config))
...
server = '<MY_SERVER_NAME>\SQLEXPRESS'
db = '<MY_DATABASE_NAME>
connection = create_connection(server, db)
cursor = connection.cursor()
cursor.execute('CREATE VIEW test_view AS SELECT * FROM existing_table')
cursor.commit()
However, this isn't much use as I can't use pandas.to_sql() - to do so I need an engine from sqlalchemy.create_engine(), but I am struggling to figure out how to use my same details in my create_connection() function above to successfully create an engine and connect to the database.
I have tried many, many combinations along the lines of:
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?driver={ODBC Driver 13 for SQL Server}?trusted_connection=yes")
conn = engine.connect().connection
or
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?trusted_connection=yes")
conn = engine.connect().connection
A Pass through exact Pyodbc string works for me:
import pandas as pd
from sqlalchemy import create_engine
from sqlalchemy.engine import URL
connection_string = (
r"Driver=ODBC Driver 17 for SQL Server;"
r"Server=(local)\SQLEXPRESS;"
r"Database=myDb;"
r"Trusted_Connection=yes;"
)
connection_url = URL.create(
"mssql+pyodbc",
query={"odbc_connect": connection_string}
)
engine = create_engine(connection_url)
df = pd.DataFrame([(1, "foo")], columns=["id", "txt"])
pd.to_sql("test_table", engine, if_exists="replace", index=False)
I am attempting to write a Python script that can take Excel sheets and import them into my SQL Server Express (with Windows Authentication) database as tables. To do this, I am using pandas to read the Excel files into a pandas DataFrame, I then hope to use pandas.to_sql() to import the data into my database. To use this function, however, I need to use sqlalchemy.create_engine().
I am able to connect to my database using pyodbc alone, and run test queries. This conection is done with the followng code:
def create_connection(server_name, database_name):
config = dict(server=server_name, database= database_name)
conn_str = ('SERVER={server};DATABASE={database};TRUSTED_CONNECTION=yes')
return pyodbc.connect(r'DRIVER={ODBC Driver 13 for SQL Server};' + conn_str.format(**config))
...
server = '<MY_SERVER_NAME>\SQLEXPRESS'
db = '<MY_DATABASE_NAME>
connection = create_connection(server, db)
cursor = connection.cursor()
cursor.execute('CREATE VIEW test_view AS SELECT * FROM existing_table')
cursor.commit()
However, this isn't much use as I can't use pandas.to_sql() - to do so I need an engine from sqlalchemy.create_engine(), but I am struggling to figure out how to use my same details in my create_connection() function above to successfully create an engine and connect to the database.
I have tried many, many combinations along the lines of:
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?driver={ODBC Driver 13 for SQL Server}?trusted_connection=yes")
conn = engine.connect().connection
or
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?trusted_connection=yes")
conn = engine.connect().connection
A Pass through exact Pyodbc string works for me:
import pandas as pd
from sqlalchemy import create_engine
from sqlalchemy.engine import URL
connection_string = (
r"Driver=ODBC Driver 17 for SQL Server;"
r"Server=(local)\SQLEXPRESS;"
r"Database=myDb;"
r"Trusted_Connection=yes;"
)
connection_url = URL.create(
"mssql+pyodbc",
query={"odbc_connect": connection_string}
)
engine = create_engine(connection_url)
df = pd.DataFrame([(1, "foo")], columns=["id", "txt"])
pd.to_sql("test_table", engine, if_exists="replace", index=False)
I can't figure out the right ODBC string I need to pass to the create engine statement.
This works
import pyodbc
import pandas as pd
cnxn=pyodbc.connect('DRIVER=/opt/teradata/client/ODBC_64/lib/tdata.so;DBCName=Server;UID=UN;PWD=PW;Database=myDB')
query = "select top 10 * from TABLE"
df = pd.read_sql(query,cnxn)
This does not work
import urllib
import sqlalchemy
params = urllib.parse.quote_plus('DRIVER=/opt/teradata/client/ODBC_64/lib/tdata.so;DBCName=Server;UID=UN;PWD=PW;Database=myDB')
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
query = "select top 10 * from TABLE"
df = pd.read_sql_query(query, engine)
I can get the pyodbc connection to work but not the sqlalchemy connection. Any help would be appreciated.
I get this error:
InterfaceError: (pyodbc.InterfaceError) ('IM001', '[IM001] [unixODBC][Driver Manager]Driver does not support this function (0) (SQLGetInfo)')
#found the answer here. https://downloads.teradata.com/tools/articles/teradata-sqlalchemy-introduction
from sqlalchemy import create_engine
user = 'sqlalc_user'
pasw=user
host = 'hostname'
port = '1025'
# connect
td_engine = create_engine('teradata://'+ user +':' + pasw + '#'+ host + ':22/')
# execute sql
sql = 'select * from dbc.usersV'
result = td_engine.execute(sql)
I am attempting to write a Python script that can take Excel sheets and import them into my SQL Server Express (with Windows Authentication) database as tables. To do this, I am using pandas to read the Excel files into a pandas DataFrame, I then hope to use pandas.to_sql() to import the data into my database. To use this function, however, I need to use sqlalchemy.create_engine().
I am able to connect to my database using pyodbc alone, and run test queries. This conection is done with the followng code:
def create_connection(server_name, database_name):
config = dict(server=server_name, database= database_name)
conn_str = ('SERVER={server};DATABASE={database};TRUSTED_CONNECTION=yes')
return pyodbc.connect(r'DRIVER={ODBC Driver 13 for SQL Server};' + conn_str.format(**config))
...
server = '<MY_SERVER_NAME>\SQLEXPRESS'
db = '<MY_DATABASE_NAME>
connection = create_connection(server, db)
cursor = connection.cursor()
cursor.execute('CREATE VIEW test_view AS SELECT * FROM existing_table')
cursor.commit()
However, this isn't much use as I can't use pandas.to_sql() - to do so I need an engine from sqlalchemy.create_engine(), but I am struggling to figure out how to use my same details in my create_connection() function above to successfully create an engine and connect to the database.
I have tried many, many combinations along the lines of:
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?driver={ODBC Driver 13 for SQL Server}?trusted_connection=yes")
conn = engine.connect().connection
or
engine = create_engine("mssql+pyodbc://#C<MY_SERVER_NAME>\SQLEXPRESS/<MY_DATABASE_NAME>?trusted_connection=yes")
conn = engine.connect().connection
A Pass through exact Pyodbc string works for me:
import pandas as pd
from sqlalchemy import create_engine
from sqlalchemy.engine import URL
connection_string = (
r"Driver=ODBC Driver 17 for SQL Server;"
r"Server=(local)\SQLEXPRESS;"
r"Database=myDb;"
r"Trusted_Connection=yes;"
)
connection_url = URL.create(
"mssql+pyodbc",
query={"odbc_connect": connection_string}
)
engine = create_engine(connection_url)
df = pd.DataFrame([(1, "foo")], columns=["id", "txt"])
pd.to_sql("test_table", engine, if_exists="replace", index=False)
When I am trying to connect python with SQL Server, following error occurred.
"pyodbc.Error: ('08001', '[08001] [Microsoft][ODBC SQL Server
Driver][DBNETLIB]SQL Server does not exist or access denied. (17)
(SQLDriverConnect)')"
Following is the my code.
import pyodbc
connection = pyodbc.connect("Driver={SQL Server}; Server=localhost;
Database=emotionDetection; uid=uname ;pwd=pw;Trusted_Connection=yes")
cursor = connection.cursor()
SQLCommand = ("INSERT INTO emotion" "(happy, sad, angry) "
"VALUES (?,?,?)")
Values = ['smile','cry','blame']
cursor.execute(SQLCommand,Values)
connection.commit()
connection.close()
This is my first attempt to connect Python with sql server. I don't have an idea what would be the driver name, server name, username and password.Do you have any idea of what should be my configuration. Please help me.
CONNECTION FROM WINDOWS TO MS SQL SERVER DATABASE:
Here you have an example I use myself to connect to MS SQL database table with a Python script:
import pyodbc
server = 'ip_database_server'
database = 'database_name'
username = 'user_name'
password = 'user_password'
driver = '{SQL Server}' # Driver you need to connect to the database
port = '1433'
cnn = pyodbc.connect('DRIVER='+driver+';PORT=port;SERVER='+server+';PORT=1443;DATABASE='+database+';UID='+username+
';PWD='+password)
cursor = cnn.cursor()
'User' and 'password' and 'table_name' are attibutes defined by the DB administrator, and he should give them to you. The port to connect to is also defined by the admin. If you are trying to connect from a Windows device to the DB, go to ODBC Data Source Administrator from Windows, and check if you have installed the driver:
Where is the ODBC data source administrator in a Windows machine.
The image is in spanish, but you only have to click on 'Drivers' tab, and check if the driver is there as in the image.
CONNECTION FROM LINUX/UNIX TO MS SQL SERVER DATABASE:
If you are working in Linux/Unix, then you shoud install a ODBC manager like 'FreeTDS' and 'unixODBC'. To configure them, you have some examples in the following links:
Example: Connecting to Microsoft SQL Server from Linux/Unix
Example: Installing and Configuring ODBC
I think you should check out this.
stackoverflow answer about odbc
Also, what sql server do you use?
The library pymssql doesnot require any drivers and works on both Windows as well as Ubunutu.
import pymssql
import pandas as pd
server = 'yourusername'
username = 'yourusername'
password = 'yourpassword'
database = 'yourdatabase'
table_name = 'yourtablename'
conn = pymssql.connect(host=server,user=username,password=password,database=database)
dat = pd.read_sql("select * from table_name,conn)
Try pyodbc with SQLalchemy
try this:
import sqlalchemy
import pyodbc
from sqlalchemy import create_engine
engine = create_engine("mssql+pyodbc://user:password#host:port/databasename?driver=ODBC+Driver+17+for+SQL+Server")
cnxn = engine.connect()
Use your corresponding driver
It works for me
Luck!
Working Examples Work Best For Me:
Need Mac ODBC Drivers?
If you need the mac driver I used homebrew and found the commands here
Detail
I personally learn best by reverse enginerring, with that said I am sharing one of my examples, it may be a bit crude but I'm growing my Python skills.
My script I created allows me to connect my Mac OS to a AWS RDS instance.
The whole script is a copy paste with a little modification for you about your server info, and you are off and running.
just modify these lines to connect.
server = 'yourusername'
username = 'yourusername'
password = 'yourforgottencomplicatedpassword'
database = 'yourdatabase'
Then Run the file: python3 ~/Your/path/pyodbc_mssqldbtest.py and you should be set.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# =============================================================================
# Created By : Jeromie Kirchoff
# Created Date: Mon July 31 22:32:00 PDT 2018
# FILENAME: pyodbc_mssqldbtest.py
# =============================================================================
"""The Module Has Been Build for Interaction with MSSQL DBs To Test the con."""
# =============================================================================
# Thanks to this post for headers https://stackoverflow.com/q/12704305/1896134
# Answer to an SO question: https://stackoverflow.com/q/42433408/1896134
# =============================================================================
import pyodbc
def runningwithqueries(query):
"""The Module Has Been Build to {Open, Run & Close} query connection."""
print("\nRunning Query: " + str(query) + "\nResult :\n")
crsr = cnxn.execute(query)
columns = [column[0] for column in crsr.description]
print(columns)
for row in crsr.fetchall():
print(row)
crsr.close()
# =============================================================================
# SET VARIABLES NEEDED FOR SERVER CONNECTION
# =============================================================================
server = 'yourusername'
username = 'yourusername'
password = 'yourforgottencomplicatedpassword'
database = 'yourdatabase'
connStr = (r'DRIVER={ODBC Driver 17 for SQL Server};' +
r"Integrated Security=True;" +
r'SERVER=' + server +
r';UID=' + username +
r';PWD=' + password +
r';DSN=MSSQL-PYTHON' +
r';DATABASE=' + database + ';'
)
print("Your Connection String:\n" + str(connStr) + "\n\n")
# =============================================================================
# CONNECT TO THE DB
# =============================================================================
cnxn = pyodbc.connect(connStr, autocommit=True)
# =============================================================================
# SET QUERIES TO VARIABLES
# =============================================================================
SQLQUERY1 = ("SELECT ##VERSION;")
SQLQUERY2 = ("SELECT * FROM sys.schemas;")
SQLQUERY3 = ("SELECT * FROM INFORMATION_SCHEMA.TABLES;")
SQLQUERY4 = ("SELECT * FROM INFORMATION_SCHEMA.COLUMNS;")
SQLQUERY5 = ("SELECT * FROM INFORMATION_SCHEMA.CHECK_CONSTRAINTS;")
SQLQUERY6 = ("EXEC sp_databases;")
SQLQUERY7 = ("EXEC sp_who2 'active';")
# =============================================================================
# RUN QUERIES
# YOU CAN RUN AS MANY QUERIES AS LONG AS THE CONNECTION IS OPEN TO THE DB
# =============================================================================
runningwithqueries(SQLQUERY1)
runningwithqueries(SQLQUERY2)
runningwithqueries(SQLQUERY3)
runningwithqueries(SQLQUERY4)
runningwithqueries(SQLQUERY5)
runningwithqueries(SQLQUERY6)
runningwithqueries(SQLQUERY7)
# =============================================================================
# CLOSE THE CONNECTION TO THE DB
# =============================================================================
cnxn.close()
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};' 'Server=**SERVER NAME**;' 'Database=**DATABASE NAME**;' 'Trusted_Connection=yes;')
cursor = conn.cursor()
cursor.execute('SELECT * FROM Output3')
This works just check you specify the Respective Driver, Server and the Database names correctively!