I am trying to connect to teradata server and load a dataframe into a table using python. Here is my code -
import sqlalchemy
engine = sqlalchemy.create_engine("teradata://username:passwor#hostname:port/")
f3.to_sql(con=engine, name='sample', if_exists='replace', schema = 'schema_name')
But I am getting the following error -
InterfaceError: (teradata.api.InterfaceError) ('DRIVER_NOT_FOUND', "No driver found for 'Teradata'. Available drivers: SQL Server,SQL Server Native Client 11.0,ODBC Driver 13 for SQL Server")
Can anybody help me to figure out whats wrong in my approach?
There's is different ways to connect to Teradata in Python. The following list is not exhaustive.
SQLAlchemy
If you wish to use SQLAlchemy, you will also need to install the package SQLAlchemy-Teradata. Here is how you can connect:
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base, DeferredReflection
from sqlalchemy.orm import scoped_session, sessionmaker
[...]
# Connect
engine = create_engine('teradata://' + user + ':' + password + '#' + host + ':22/' + database)
db_session = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=engine))
db_session.execute('SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;') # To avoid locking tables when doing select on tables
db_session.commit()
Base = declarative_base(cls=DeferredReflection)
Base.query = db_session.query_property()
Then you can use db_session to make queries. See SQLAlchemy Session API
Pyodbc
If you wish to use Pyodbc you will first need to install Teradata driver on your machine. Example on mine, after installing Teradata driver I have the following entry in /etc/odbcinst.ini
[Teradata]
Driver=/opt/teradata/client/16.00/odbc_64/lib/tdata.so
APILevel=CORE
ConnectFunctions=YYY
DriverODBCVer=3.51
SQLLevel=1
Then I can connect with the following:
import pyodbc
[...]
#Teradata Connection
connection= pyodbc.connect("driver={Teradata};dbcname=" + host + ";uid=" + user + ";pwd=" + pwd + ";charset=utf8;", autocommit=True)
connection.setdecoding(pyodbc.SQL_CHAR, encoding='utf-8')
connection.setdecoding(pyodbc.SQL_WCHAR, encoding='utf-8')
connection.setdecoding(pyodbc.SQL_WMETADATA, encoding='utf-8')
connection.setencoding(encoding='utf-8')
cursor= n.cursor()
cursor.execute("Select 'Hello World'")
for row in cursor:
print (row)
To connect to a teradata database, you need pyodbc, i also have problems with teradata dialect.
Example:
import pyodbc
user = 'user'
pasw = 'pass'
host = 'host'
connection = pyodbc.connect('DRIVER=Teradata;DBCNAME=' + host +';UID=' + user + ';PWD=' + pasw +';QUIETMODE=YES', autocommit=True,unicode_results=True)
I am not sure why your are using sqlalchemy. But you could explore using Teradata module to connect to Teradata as explained in the other link:
Connecting Python with Teradata using Teradata module
I met a similar problem in airflow, I used jars and jaydebeapi to connect teradata database and execute sql:
[root#myhost transfer]# cat test_conn.py
import jaydebeapi
from contextlib import closing
jclassname='com.teradata.jdbc.TeraDriver'
jdbc_driver_loc = '/opt/spark-2.3.1/jars/terajdbc4-16.20.00.06.jar,/opt/spark-2.3.1/jars/tdgssconfig-16.20.00.06.jar'
jdbc_driver_name = 'com.teradata.jdbc.TeraDriver'
host='my_teradata.address'
url='jdbc:teradata://' + host + '/TMODE=TERA'
login="teradata_user_name"
psw="teradata_passwd"
sql = "SELECT COUNT(*) FROM A_TERADATA_TABLE_NAME where month_key='202009'"
conn = jaydebeapi.connect(jclassname=jdbc_driver_name,
url=url,
driver_args=[login, psw],
jars=jdbc_driver_loc.split(","))
with closing(conn) as conn:
with closing(conn.cursor()) as cur:
cur.execute(sql)
print(cur.fetchall())
[root#myhost transfer]# python test_conn.py
[(7734133,)]
[root#myhost transfer]#
How do I connect to a MySQL database using a python program?
Connecting to MYSQL with Python 2 in three steps
1 - Setting
You must install a MySQL driver before doing anything. Unlike PHP, Only the SQLite driver is installed by default with Python. The most used package to do so is MySQLdb but it's hard to install it using easy_install. Please note MySQLdb only supports Python 2.
For Windows user, you can get an exe of MySQLdb.
For Linux, this is a casual package (python-mysqldb). (You can use sudo apt-get install python-mysqldb (for debian based distros), yum install MySQL-python (for rpm-based), or dnf install python-mysql (for modern fedora distro) in command line to download.)
For Mac, you can install MySQLdb using Macport.
2 - Usage
After installing, Reboot. This is not mandatory, But it will prevent me from answering 3 or 4 other questions in this post if something goes wrong. So please reboot.
Then it is just like using any other package :
#!/usr/bin/python
import MySQLdb
db = MySQLdb.connect(host="localhost", # your host, usually localhost
user="john", # your username
passwd="megajonhy", # your password
db="jonhydb") # name of the data base
# you must create a Cursor object. It will let
# you execute all the queries you need
cur = db.cursor()
# Use all the SQL you like
cur.execute("SELECT * FROM YOUR_TABLE_NAME")
# print all the first cell of all the rows
for row in cur.fetchall():
print row[0]
db.close()
Of course, there are thousand of possibilities and options; this is a very basic example. You will have to look at the documentation. A good starting point.
3 - More advanced usage
Once you know how it works, You may want to use an ORM to avoid writing SQL manually and manipulate your tables as they were Python objects. The most famous ORM in the Python community is SQLAlchemy.
I strongly advise you to use it: your life is going to be much easier.
I recently discovered another jewel in the Python world: peewee. It's a very lite ORM, really easy and fast to setup then use. It makes my day for small projects or stand alone apps, Where using big tools like SQLAlchemy or Django is overkill :
import peewee
from peewee import *
db = MySQLDatabase('jonhydb', user='john', passwd='megajonhy')
class Book(peewee.Model):
author = peewee.CharField()
title = peewee.TextField()
class Meta:
database = db
Book.create_table()
book = Book(author="me", title='Peewee is cool')
book.save()
for book in Book.filter(author="me"):
print book.title
This example works out of the box. Nothing other than having peewee (pip install peewee) is required.
Here's one way to do it, using MySQLdb, which only supports Python 2:
#!/usr/bin/python
import MySQLdb
# Connect
db = MySQLdb.connect(host="localhost",
user="appuser",
passwd="",
db="onco")
cursor = db.cursor()
# Execute SQL select statement
cursor.execute("SELECT * FROM location")
# Commit your changes if writing
# In this case, we are only reading data
# db.commit()
# Get the number of rows in the resultset
numrows = cursor.rowcount
# Get and display one row at a time
for x in range(0, numrows):
row = cursor.fetchone()
print row[0], "-->", row[1]
# Close the connection
db.close()
Reference here
If you do not need MySQLdb, but would accept any library, I would very, very much recommend MySQL Connector/Python from MySQL: http://dev.mysql.com/downloads/connector/python/.
It is one package (around 110k), pure Python, so it is system independent, and dead simple to install. You just download, double-click, confirm license agreement and go. There is no need for Xcode, MacPorts, compiling, restarting …
Then you connect like:
import mysql.connector
cnx = mysql.connector.connect(user='scott', password='tiger',
host='127.0.0.1',
database='employees')
try:
cursor = cnx.cursor()
cursor.execute("""
select 3 from your_table
""")
result = cursor.fetchall()
print result
finally:
cnx.close()
Oracle (MySQL) now supports a pure Python connector. That means no binaries to install: it's just a Python library. It's called "Connector/Python".
http://dev.mysql.com/downloads/connector/python/
After installations, you can see some usage examples here
Stop Using MySQLDb if you want to avoid installing mysql headers just to access mysql from python.
Use pymysql. It does all of what MySQLDb does, but it was implemented purely in Python with NO External Dependencies. This makes the installation process on all operating systems consistent and easy. pymysql is a drop in replacement for MySQLDb and IMHO there is no reason to ever use MySQLDb for anything... EVER! - PTSD from installing MySQLDb on Mac OSX and *Nix systems, but that's just me.
Installation
pip install pymysql
That's it... you are ready to play.
Example usage from pymysql Github repo
import pymysql.cursors
import pymysql
# Connect to the database
connection = pymysql.connect(host='localhost',
user='user',
password='passwd',
db='db',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
try:
with connection.cursor() as cursor:
# Create a new record
sql = "INSERT INTO `users` (`email`, `password`) VALUES (%s, %s)"
cursor.execute(sql, ('webmaster#python.org', 'very-secret'))
# connection is not autocommit by default. So you must commit to save
# your changes.
connection.commit()
with connection.cursor() as cursor:
# Read a single record
sql = "SELECT `id`, `password` FROM `users` WHERE `email`=%s"
cursor.execute(sql, ('webmaster#python.org',))
result = cursor.fetchone()
print(result)
finally:
connection.close()
ALSO - Replace MySQLdb in existing code quickly and transparently
If you have existing code that uses MySQLdb, you can easily replace it with pymysql using this simple process:
# import MySQLdb << Remove this line and replace with:
import pymysql
pymysql.install_as_MySQLdb()
All subsequent references to MySQLdb will use pymysql transparently.
Try using MySQLdb. MySQLdb only supports Python 2.
There is a how to page here: http://www.kitebird.com/articles/pydbapi.html
From the page:
# server_version.py - retrieve and display database server version
import MySQLdb
conn = MySQLdb.connect (host = "localhost",
user = "testuser",
passwd = "testpass",
db = "test")
cursor = conn.cursor ()
cursor.execute ("SELECT VERSION()")
row = cursor.fetchone ()
print "server version:", row[0]
cursor.close ()
conn.close ()
Run this command in your terminal to install mysql connector:
pip install mysql-connector-python
And run this in your python editor to connect to MySQL:
import mysql.connector
mydb = mysql.connector.connect(
host="localhost",
user="username",
passwd="password",
database="database_name"
)
Samples to execute MySQL Commands (in your python edior):
mycursor = mydb.cursor()
mycursor.execute("CREATE TABLE customers (name VARCHAR(255), address VARCHAR(255))")
mycursor.execute("SHOW TABLES")
mycursor.execute("INSERT INTO customers (name, address) VALUES ('John', 'Highway 21')")
mydb.commit() # Use this command after insert, update, delete commands
For more commands: https://www.w3schools.com/python/python_mysql_getstarted.asp
For newer versions of Python (>=3.6)
Use either mysqlclient or pymysql (recommended).
For older versions of Python (<3.7, 2.4 <= Python <= 2.7)
If you are working on an older version of Python (unfortunately), then you could also try out -> oursql.
Please note however, that the project is no longer maintained, and bug fixes are not being pushed either.
As a db driver, there is also oursql. Some of the reasons listed on that link, which say why oursql is better:
oursql has real parameterization, sending the SQL and data to MySQL completely separately.
oursql allows text or binary data to be streamed into the database and streamed out of the database, instead of requiring everything to be buffered in the client.
oursql can both insert rows lazily and fetch rows lazily.
oursql has unicode support on by default.
oursql supports python 2.4 through 2.7 without any deprecation warnings on 2.6+ (see PEP 218) and without completely failing on 2.7 (see PEP 328).
oursql runs natively on python 3.x.
So how to connect to mysql with oursql?
Very similar to mysqldb:
import oursql
db_connection = oursql.connect(host='127.0.0.1',user='foo',passwd='foobar',db='db_name')
cur=db_connection.cursor()
cur.execute("SELECT * FROM `tbl_name`")
for row in cur.fetchall():
print row[0]
The tutorial in the documentation is pretty decent.
And of course for ORM SQLAlchemy is a good choice, as already mentioned in the other answers.
SqlAlchemy
SQLAlchemy is the Python SQL toolkit and Object Relational Mapper that
gives application developers the full power and flexibility of SQL.
SQLAlchemy provides a full suite of well known enterprise-level
persistence patterns, designed for efficient and high-performing
database access, adapted into a simple and Pythonic domain language.
Installation
pip install sqlalchemy
RAW query
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, scoped_session
engine = create_engine("mysql://<user_name>:<password>#<host_name>/<db_name>")
session_obj = sessionmaker(bind=engine)
session = scoped_session(session_obj)
# insert into database
session.execute("insert into person values(2, 'random_name')")
session.flush()
session.commit()
ORM way
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, scoped_session
Base = declarative_base()
engine = create_engine("mysql://<user_name>:<password>#<host_name>/<db_name>")
session_obj = sessionmaker(bind=engine)
session = scoped_session(session_obj)
# Bind the engine to the metadata of the Base class so that the
# declaratives can be accessed through a DBSession instance
Base.metadata.bind = engine
class Person(Base):
__tablename__ = 'person'
# Here we define columns for the table person
# Notice that each column is also a normal Python instance attribute.
id = Column(Integer, primary_key=True)
name = Column(String(250), nullable=False)
# insert into database
person_obj = Person(id=12, name="name")
session.add(person_obj)
session.flush()
session.commit()
Best way to connect to MySQL from python is to Use MySQL Connector/Python because it is official Oracle driver for MySQL for working with Python and it works with both Python 3 and Python 2.
follow the steps mentioned below to connect MySQL
install connector using pip
pip install mysql-connector-python
or you can download the installer from https://dev.mysql.com/downloads/connector/python/
Use connect() method of mysql connector python to connect to MySQL.pass the required argument to connect() method. i.e. Host, username, password, and database name.
Create cursor object from connection object returned by connect()method to execute SQL queries.
close the connection after your work completes.
Example:
import mysql.connector
from mysql.connector import Error
try:
conn = mysql.connector.connect(host='hostname',
database='db',
user='root',
password='passcode')
if conn.is_connected():
cursor = conn.cursor()
cursor.execute("select database();")
record = cursor.fetchall()
print ("You're connected to - ", record)
except Error as e :
print ("Print your error msg", e)
finally:
#closing database connection.
if(conn.is_connected()):
cursor.close()
conn.close()
Reference - https://pynative.com/python-mysql-database-connection/
Important API of MySQL Connector Python
For DML operations - Use cursor.execute() and cursor.executemany() to run query. and after this use connection.commit() to persist your changes to DB
To fetch data - Use cursor.execute() to run query and cursor.fetchall(), cursor.fetchone(), cursor.fetchmany(SIZE) to fetch data
Despite all answers above, in case you do not want to connect to a specific database upfront, for example, if you want to create the database still (!), you can use connection.select_db(database), as demonstrated in the following.
import pymysql.cursors
connection = pymysql.connect(host='localhost',
user='mahdi',
password='mahdi',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
cursor = connection.cursor()
cursor.execute("CREATE DATABASE IF NOT EXISTS "+database)
connection.select_db(database)
sql_create = "CREATE TABLE IF NOT EXISTS "+tablename+(timestamp DATETIME NOT NULL PRIMARY KEY)"
cursor.execute(sql_create)
connection.commit()
cursor.close()
Even though some of you may mark this as a duplicate and get upset that I am copying someone else's answer, I would REALLY like to highlight an aspect of Mr. Napik's response. Because I missed this, I caused nationwide website downtime (9min). If only someone shared this information, I could have prevented it!
Here is his code:
import mysql.connector
cnx = mysql.connector.connect(user='scott', password='tiger',
host='127.0.0.1',
database='employees')
try:
cursor = cnx.cursor()
cursor.execute("""select 3 from your_table""")
result = cursor.fetchall()
print(result)
finally:
cnx.close()
The important thing here is the Try and Finally clause. This allows connections to ALWAYS be closed, regardless of what happens in the cursor/sqlstatement portion of the code. A lot of active connections cause DBLoadNoCPU to spike and could crash a db server.
I hope this warning helps to save servers and ultimately jobs! :D
MySQLdb is the straightforward way. You get to execute SQL queries over a connection. Period.
My preferred way, which is also pythonic, is to use the mighty SQLAlchemy instead. Here is a query related tutorial, and here is a tutorial on ORM capabilities of SQLALchemy.
for Python3.6 I found two driver: pymysql and mysqlclient. I tested the performance between them and got the result: the mysqlclient is faster.
below is my test process(need install python lib profilehooks to analyze time elapse：
raw sql: select * from FOO;
immediatly execute in mysql terminal:
46410 rows in set (0.10 sec)
pymysql (2.4s):
from profilehooks import profile
import pymysql.cursors
import pymysql
connection = pymysql.connect(host='localhost', user='root', db='foo')
c = connection.cursor()
#profile(immediate=True)
def read_by_pymysql():
c.execute("select * from FOO;")
res = c.fetchall()
read_by_pymysql()
here's the pymysql profile:
mysqlclient (0.4s)
from profilehooks import profile
import MySQLdb
connection = MySQLdb.connect(host='localhost', user='root', db='foo')
c = connection.cursor()
#profile(immediate=True)
def read_by_mysqlclient():
c.execute("select * from FOO;")
res = c.fetchall()
read_by_mysqlclient()
here's the mysqlclient profile:
So, it seems that mysqlclient is much faster than pymysql
Just a modification in above answer.
Simply run this command to install mysql for python
sudo yum install MySQL-python
sudo apt-get install MySQL-python
remember! It is case sensitive.
mysqlclient is the best as others only provide support to specific versions of python
pip install mysqlclient
example code
import mysql.connector
import _mysql
db=_mysql.connect("127.0.0.1","root","umer","sys")
#db=_mysql.connect(host,user,password,db)
# Example of how to insert new values:
db.query("""INSERT INTO table1 VALUES ('01', 'myname')""")
db.store_result()
db.query("SELECT * FROM new1.table1 ;")
#new1 is scheme table1 is table mysql
res= db.store_result()
for i in range(res.num_rows()):
print(result.fetch_row())
see https://github.com/PyMySQL/mysqlclient-python
Also take a look at Storm. It is a simple SQL mapping tool which allows you to easily edit and create SQL entries without writing the queries.
Here is a simple example:
from storm.locals import *
# User will be the mapped object; you have to create the table before mapping it
class User(object):
__storm_table__ = "user" # table name
ID = Int(primary=True) #field ID
name= Unicode() # field name
database = create_database("mysql://root:password#localhost:3306/databaseName")
store = Store(database)
user = User()
user.name = u"Mark"
print str(user.ID) # None
store.add(user)
store.flush() # ID is AUTO_INCREMENT
print str(user.ID) # 1 (ID)
store.commit() # commit all changes to the database
To find and object use:
michael = store.find(User, User.name == u"Michael").one()
print str(user.ID) # 10
Find with primary key:
print store.get(User, 1).name #Mark
For further information see the tutorial.
This is Mysql DB connection
from flask import Flask, render_template, request
from flask_mysqldb import MySQL
app = Flask(__name__)
app.config['MYSQL_HOST'] = 'localhost'
app.config['MYSQL_USER'] = 'root'
app.config['MYSQL_PASSWORD'] = 'root'
app.config['MYSQL_DB'] = 'MyDB'
mysql = MySQL(app)
#app.route('/', methods=['GET', 'POST'])
def index():
if request.method == "POST":
details = request.form
cur = mysql.connection.cursor()
cur.execute ("_Your query_")
mysql.connection.commit()
cur.close()
return 'success'
return render_template('index.html')
if __name__ == '__main__':
app.run()
PyMySQL 0.10.1 - Released: Sep 10, 2020, has support for python3 as well.
python3 -m pip install PyMySQL
Simple code:
import pymysql
# Connect to the database
conn = pymysql.connect(host='127.0.0.1',user='root',passwd='root',db='fax')
# Create a Cursor object
cur = conn.cursor()
# Execute the query
cur.execute("SELECT * FROM fax.student")
# Read and print records
for row in cur.fetchall():
print(row)
output:
(1, 'Petar', 'Petrovic', 1813, 'Njegusi')
(2, 'Donald', 'Tramp', 1946, 'New York')
(3, 'Bill', 'Gates', 1955, 'Seattle')
you can connect your python code to mysql in this way.
import MySQLdb
db = MySQLdb.connect(host="localhost",
user="appuser",
passwd="",
db="onco")
cursor = db.cursor()
First step to get The Library:
Open terminal and execute pip install mysql-python-connector.
After the installation go the second step.
Second Step to import the library:
Open your python file and write the following code:
import mysql.connector
Third step to connect to the server:
Write the following code:
conn = mysql.connector.connect(host=you host name like localhost or 127.0.0.1,
username=your username like root,
password = your password)
Third step Making the cursor:
Making a cursor makes it easy for us to run queries.
To make the cursor use the following code:
cursor = conn.cursor()
Executing queries:
For executing queries you can do the following:
cursor.execute(query)
If the query changes any thing in the table you need to add the following code after the execution of the query:
conn.commit()
Getting values from a query:
If you want to get values from a query then you can do the following:
cursor.excecute('SELECT * FROM table_name') for i in cursor: print(i) #Or for i in cursor.fetchall(): print(i)
The fetchall() method returns a list with many tuples that contain the values that you requested ,row after row .
Closing the connection:
To close the connection you should use the following code:
conn.close()
Handling exception:
To Handel exception you can do it Vai the following method:
try: #Logic pass except mysql.connector.errors.Error: #Logic pass
To use a database:
For example you are a account creating system where you are storing the data in a database named blabla, you can just add a database parameter to the connect() method ,like
mysql.connector.connect(database = database name)
don't remove other informations like host,username,password.
Python does not come with an inbuilt Library to interact with MySQL, so in order to make a connection between the MySQL database and Python we need to install the MySQL driver or module for our Python Environment.
pip install mysql-connector-python
the mysql-connecter-python is an open source Python library that can connect your python code to the MySQL data base in a few lines of code. And it is very compatible with the latest version of Python.
After install the mysql-connector-python, you can connect to your MySQL database using the the following code snippet.
import mysql.connector
Hostname = "localhost"
Username = "root"
Password ="admin" #enter your MySQL password
#set connection
set_db_conn = mysql.connector.connect(host= Hostname, user=Username, password=Password)
if set_db_conn:
print("The Connection between has been set and the Connection ID is:")
#show connection id
print(set_db_conn.connection_id)
Connect Django with MySQL
In Django, to connect your model or project to the MySQL data base, you need to install the mysqlclient library.
pip install mysqlclient
And to configure your Django setting so your project can connect to the MySQL database, you can use the following setting.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'database_name',
'USER': 'username',
'PASSWORD': 'databasepassword#123',
'HOST': 'localhost', # Or an IP Address that your DB is hosted on
'PORT': '3306',
}
I have written a dedicated Python tutorial on my blog that covers how you can connect to a MySQL database and create tables using Python. To know more about it, click here.
For python 3.3
CyMySQL
https://github.com/nakagami/CyMySQL
I have pip installed on my windows 7, just
pip install cymysql
(you don't need cython)
quick and painless
first install the driver
pip install MySQL-python
Then a basic code goes like this:
#!/usr/bin/python
import MySQLdb
try:
db = MySQLdb.connect(host="localhost", # db server, can be a remote one
db="mydb" # database
user="mydb", # username
passwd="mydb123", # password for this username
)
# Create a Cursor object
cur = db.cursor()
# Create a query string. It can contain variables
query_string = "SELECT * FROM MY_TABLE"
# Execute the query
cur.execute(query_string)
# Get all the rows present the database
for each_row in cur.fetchall():
print each_row
# Close the connection
db.close()
except Exception, e:
print 'Error ', e
First install the driver (Ubuntu)
sudo apt-get install python-pip
sudo pip install -U pip
sudo apt-get install python-dev libmysqlclient-dev
sudo apt-get install MySQL-python
MySQL database connection codes
import MySQLdb
conn = MySQLdb.connect (host = "localhost",user = "root",passwd = "pass",db = "dbname")
cursor = conn.cursor ()
cursor.execute ("SELECT VERSION()")
row = cursor.fetchone ()
print "server version:", row[0]
cursor.close ()
conn.close ()
First, install python-mysql connector from https://dev.mysql.com/downloads/connector/python/
on Python console enter:
pip install mysql-connector-python-rf
import mysql.connector