I've successfully setup a connection from a stock raspberry pi 4 to our local database running on windows 10.
In the terminal if I type:
tsql -S *servername* -U *username* -P *password*
select * from testlist_table
go
I get the result for the query and all is good for that test. It's probably worth mentioning I recieve back after the first tsql line:
default encoding is en_us.utf-8 or something odd like this and then:
using default encoding utf-8,
When using python code if I type
import pyodbc
cnxn = pyodbc.connect("Driver={freeTDS};Server=*servername*;DATABASE=*databasename*;UID=*userid*;pwd=*pwd*")
cursor = cnxn.cursor()
cursor.execute("Select * from TestList_Table")
I get the following error:
Unicode data in a Unicode-only collation or ntext data cannot be sent to clients using DB-Library (such as ISQL) or ODBC version 3.7 or earlier.
Any ideas? I've tried setting encoding / decoding but that hasn't help. If I select a particular field so far that's worked. Not excited to try and change all the code as I have a lot of columns and am referencing by column number sometimes in a big mess of code.
I'm going to assume you're running a relatively recent version of FreeTDS (version 0.95 or above; you can find your version with tsql -C). If you're using an earlier version and this doesn't work, trying changing the TDS_Version to 7.2 or 7.1.
Give this a try:
import pyodbc
cnxn = pyodbc.connect(
"Driver={freeTDS};Server=*servername*;DATABASE=*databasename*;"
"UID=*userid*;pwd=*pwd*;TDS_Version=7.3;CHARSET=UTF-8;"
)
cursor = cnxn.cursor()
cursor.execute("Select * from TestList_Table")
I was looking for a better way to try to read files and use them to create tables for MySQL, and I stumbled across agate, so I'm trying to see if it will work for my purposes.
I have created the table from a csv file using:
table=agate.Table.from_csv('testsheet.csv')
That worked just fine, and I saw agate option to save a db to sql using agatesql and the command:
table.to_sql('postgresql:///database', 'output_table')
Is there any way to use this command or to make this module work with MySQL, or will it only work with postgresql? Thanks in advance for any help
As the Agate SQL API docs indicate, the .to_sql() method can be any valid SQLAlchemy connection string or SQLAlchemy connection (Postgres was just the example used):
from sqlalchemy import create_engine
...
# CONNECTION STRING
table.to_sql('mysql://user:pwd#hostname:port/database', 'output_table')
# CONNECTION OBJECT
my_engine = create_engine('mysql://user:pwd#hostname:port/database')
table.to_sql(my_engine, 'output_table')
If needed you can interface sqlAlchemy with an available DB-API like pymsql
import pymysql
from sqlalchemy import create_engine
...
# CONNECTION OBJECT
my_engine = create_engine("mysql+pymysql://user:pwd#hostname:port/database")
table.to_sql(my_engine, 'output_table')
Below is my code to connect to Teradata through a JDBC connection using Jaydebeapi. When i run the query using the razorsql GUI, it takes only 15 seconds. When i run it through the code below, it took over 20 minutes just to run query1.
Is there anything wrong with the Jaydebeapi or can i make it faster by optimizing my query/using Jpype?
#-*- coding: utf-8 -*-
import jaydebeapi
import jpype
import pandas as pd
import numpy as np
import collections
query_dict=collections.OrderedDict()
connection = jaydebeapi.connect('com.teradata.jdbc.TeraDriver', ['my_db_name','my_username','my_password'], ['/Applications/drivers/tdgssconfig.jar','/Applications/drivers/terajdbc4.jar'],)
cur = connection.cursor()
query_name_list=['query1','query2']
query1= """select ......"""
query2= """ select ....."""
for i in query_list:
query_dict[i]=locals()[i]
print query_dict.keys()
for index in range(len(query_list)):
tera_query=query_dict.values()[index]
cur.execute(tera_query)
print "executing ... "
result=cur.fetchall()
print "fetching results ... "
I've already posted about some performance considerations.
Here again:
...fetching of large result sets with the JPype implementation causes some JNI calls for every single cell value which causes lot's of overhead. ...
Minimize the size of your resultset. Do aggregations using SQL functions.
Give the newest implementation of JPype1 a try. There have been some performance improvements.
Switch your runtime to Jython (JayDeBeApi works on Jython as well)
Implement the db queries and data extraction directly in Java and call the logic using JPype but with a interface not returning a large data set.
Try to improve JPype and JayDeBeApi code
BTW: The subject ".. if 2 driver files" is a bit misleading. The number of driver files is definitely not related to performance issues.
Is there a way to connect Python to Db2?
The documentation is difficult to find, and once you find it, it's pretty abysmal. Here's what I've found over the past 3 hours.
You need to install ibm_db using pip, as follows:
pip install ibm_db
You'll want to create a connection object. The documentation is here.
Here's what I wrote:
from ibm_db import connect
# Careful with the punctuation here - we have 3 arguments.
# The first is a big string with semicolons in it.
# (Strings separated by only whitespace, newlines included,
# are automatically joined together, in case you didn't know.)
# The last two are emptry strings.
connection = connect('DATABASE=<database name>;'
'HOSTNAME=<database ip>;' # 127.0.0.1 or localhost works if it's local
'PORT=<database port>;'
'PROTOCOL=TCPIP;'
'UID=<database username>;'
'PWD=<username password>;', '', '')
Next you should know that commands to ibm_db never actually give you results. Instead, you need to call one of the fetch methods on the command, repeatedly, to get the results. I wrote this helper function to deal with that.
def results(command):
from ibm_db import fetch_assoc
ret = []
result = fetch_assoc(command)
while result:
# This builds a list in memory. Theoretically, if there's a lot of rows,
# we could run out of memory. In practice, I've never had that happen.
# If it's ever a problem, you could use
# yield result
# Then this function would become a generator. You lose the ability to access
# results by index or slice them or whatever, but you retain
# the ability to iterate on them.
ret.append(result)
result = fetch_assoc(command)
return ret # Ditch this line if you choose to use a generator.
Now with that helper function defined, you can easily do something like get the information on all the tables in your database with the following:
from ibm_db import tables
t = results(tables(connection))
If you'd like to see everything in a given table, you could do something like this now:
from ibm_db import exec_immediate
sql = 'LIST * FROM ' + t[170]['TABLE_NAME'] # Using our list of tables t from before...
rows = results(exec_immediate(connection, sql))
And now rows contains a list of rows from the 170th table in your database, where every row contains a dict of column name: value.
Hope this all helps.
After lots of digging I discovered how to connect with DB2 using ibm_db.
First off, if you use a python version higher than 3.2 use
pip install ibm_db==2.0.8a
version 2.0.8 (the latest) will fail to install.
then use the following to connect
import ibm_db_dbi as db
conn = db.connect("DATABASE=name;HOSTNAME=host;PORT=60000;PROTOCOL=TCPIP;UID=username;PWD=password;", "", "")
list tables with
for t in conn.tables():
print(t)
and execute SQL with
cursor = conn.cursor()
cursor.execute("SELECT * FROM Schema.Table")
for r in cursor.fetchall():
print(r)
check this link for official not so accurate documentation
ibm-db, the official DB2 driver for Python and Django is here:
https://code.google.com/p/ibm-db/
Here's a recent tutorial for how to install everything on Ubuntu Linux:
http://programmingzen.com/2011/05/12/installing-python-django-and-db2-on-ubuntu-11-04/
I should mention that there were several older unofficial DB2 drivers for Python. ibm-db is the one you should be using.
In addition to #prof1990 response:
Since 2.0.9 (Aug 16th 2018), also with Python 3 you can simply use:
pip install ibm_db
Reference:
https://github.com/ibmdb/python-ibmdb#updated-ibm_db
Example of connection here:
import ibm_db
ibm_db.connect("DATABASE=<dbname>;HOSTNAME=<host>;PORT=<60000>;PROTOCOL=TCPIP;UID=<username>;PWD=<password>;", "", "")
Full API documentation here:
https://github.com/ibmdb/python-ibmdb/wiki/APIs
You can connect to db2 from python using jaydeapi
First install library running pip install jaydeapi
download db2jcc4.jar
Then you can connect using below code :
by passing hostname,portno, userid,password database name
import jaydebeapi
conn_src = jaydebeapi.connect(
'com.ibm.db2.jcc.DB2Driver',
['YourHostName:PortNo/DatabaseName','userid','password'],'C:/db2jcc4.jar'
)
cursor=conn_src.cursor()
sql = 'Select * from schemaname.TableName fetch first 100 rows only '
cursor.execute(sql)
print("fetchall:")
result = cursor.fetchall()
for r in result:
print(r)
There is a way in which one can connect to IBM db2 using nothing but Python requests library. Worked for me.
STEP 1:
Go to IBM CLOUD Dashboard -> Navigate to your IBM db2 instance -> Click on 'Service Credentials'
A default one should be there, if not, create one. This service credential is a dictionary. Copy the service credentials.
STEP 2:
db2id = { // service credential dictionary here //}
api = "/dbapi/v3"
host = db2id['https_url']+api
userinfo = {"userid":db2id['username'],"password":db2id['password']}
service = '/auth/tokens'
r = requests.post(host+service,json=userinfo)
access_token = r.json()['token']
auth_header = {"Authorization": "Bearer "+access_token}
// Connection to database established
STEP 3
Now you can run SELECT, INSERT, DELETE, UPDATE queries
The format for INSERT, DELETE, UPDATE queries is the same. After an INSERT, DELETE, UPDATE query, a COMMIT query has to be sent, else changes aren't reflected. (You should commit your changes otherwise also)
INSERT / UPDATE / DELETE QUERIES
sql = " your insert/update/delete query here "
sql_command = {"commands":sql,"limit":1000,"separator":";","stop_on_error":"yes"}
service = "/sql_jobs"
r = requests.post(host+service,headers=auth_header,json=sql_command)
sql_command = {"commands":"COMMIT","limit":1000,"separator":";","stop_on_error":"yes"}
service = "/sql_jobs"
r = requests.post(host+service,headers=auth_header,json=sql_command)
You can use the variable r to check status of your request
SELECT QUERIES
sql = " your select query here "
service = "/sql_jobs"
r = requests.post(host+service,headers=auth_header,json=sql_command)
jobid = r.json()['id']
r = requests.get(host+service+"/"+jobid,headers=auth_header)
results = r.json()['results']
rows = results[0]['rows']
The variable rows will have the results of your query. Use it as per your convenience.
I didn't use any DDL queries. But I think they should work like the DML queries. Not sure though!
IBM's Db2 is available for various platforms. If you are trying to connect to a Db2 which lives on an IBM i server (formerly known as AS/400, iSeries, or System i), then ibm_db requires a product called Db2 Connect, which is rather expensive. Most people who use Python to connect to Db2 for i use ODBC (usually through PyODBC).
I'm not completely sure about the situation with Db2 on their z (mainframe) servers, but I would think it also requires Db2 Connect.
There are many ways to connect from Python to Db2. I am trying to provide a summary of options. Note that in many environments SSL/TLS is enforced now which requires additional parameters (see below).
Db2 and Python drivers
Db2 does not offer one, but four drivers (clients) for Python. The Db2 documentation page "Python, SQLAlchemy, and Django Framework application development for IBM Database servers" provides a good overview about the four drivers:
ibm_db is based on the IBM-defined API,
ibm_db_dbi is a driver for the Python database API (DBI),
ibm_db_sa implements the Python SQLAlchemy interface and
ibm_db_django serves as Db2 driver in the Django Framework.
Note that there are additional Python database interfaces which make use of existing JDBC or ODBC drivers which can be used to connect to Db2. You can use SQLAlchemy (ibm_db_sa) with the popular Flask framework. To use Db2 with pandas utilize ibm_db_dbi. All of the above Db2 drivers are available on GitHub and are based on the CLI (Call Level Interface / ODBC). There are additional ways to connect to Db2, e.g., by using 3rd party ODBC-based wrappers and more.
Db2 connections
Typical connection information is made up of the Db2 server (hostname), the port, the database name and username / password information. If nothing else is specified, most drivers assume that the connection is not encrypted. Thus, to connect over an encrypted connection more parameters are needed. They depend on the Db2 version, the type of Db2 product and some more. Let's start easy.
Newer Db2 versions simplified the use of SSL/TLS because certificates are now part of the package. A typical connection string would then look like this:
conn_str='database=MYDB;hostname=db2host.example.com;port=50001;protocol=tcpip;uid=db2inst1;pwd=secret;security=SSL'
ibm_db_conn = ibm_db.connect(conn_str,'','')
An important parameter is "security=SSL" to tell the driver to use encryption for the data in transit.
Db2 connection strings can have even more options. It depends on what security plugin is enabled. See this blog post on connecting from Python to Db2 for more links and discussions.
SQL Alchemy connection
When using Db2 with SQLAlchemy, pass an URI similar to
ibm_db_sa://user:password#hostname:port/database?Security=SSL
to get the connection established.
You can use ibm_db library to connect DB2.
query_str = "SELECT COUNT(*) FROM table_name"
conn = ibm_db.pconnect("dsn=write","usrname","secret")
query_stmt = ibm_db.prepare(conn, query_str)
ibm_db.execute(query_stmt)
This is for future reference:
Official installation docs say:
Python 2.5 or later, excluding Python 3.X.
pip install ibm_db
It only worked on Python 2.7 for me; it didn't for 3.X. Also, I had to make Python 2.7 default (instead of Python 3) so that the installation would work (otherwise, there would be installation errors).
Official docs sample usage:
import ibm_db
ibm_db.connect("DATABASE=name;HOSTNAME=host;PORT=60000;PROTOCOL=TCPIP;UID=username; PWD=password;", "", "")
Version: ibm-db 3.0.2 - ibm-db==3.0.2
pip install ibm-db
Released: Jun 17, 2020
Connect to a local or cataloged database:
import ibm_db
conn = ibm_db.connect("database","username","password")
Connect to an uncataloged database:
import ibm_db
ibm_db.connect("DATABASE=name;HOSTNAME=host;PORT=60000;PROTOCOL=TCPIP;UID=username;
PWD=password;", "", "")
How I managed to do in 2021.
What you will need:
Python 3.7
PipEnv
Ibm-db
Ibm-db version is not important but this lib only works with Python 3.7 (current python version is 3.9).
Install Python 3.7.6 in your machine (this is the version that worked).
In your IDE create a new python file.
Let' create a Virtual Enviroment to make sure we will use Python 3.7
pip install pipenv
After installing
pipenv install --python 3.7
Activate the Virtual Environment
pipenv shell
You can use pip list to verify if you are in the new Virtual Enviroment - if list only shows 3 or 4 libs, it's because you are
Now you can download Ibm_db
pip install ibm-db
You may add this to your code to confirm what is the version you are using
from platform import python_version
print(python_version())
Now accessing the DB2
import ibm_db_dbi as db
# Connect to DB2B1 (keep Protocol as TCPIP)
conn = db.connect("DATABASE=DBNAME;HOSTNAME=hostname;PORT=port;PROTOCOL=TCPIP;UID=Your User;PWD=Your Password;", "", "")
Checking all tables available
for t in conn.tables():
print(t)
Your SQL code:
sql_for_df = """SELECT *
FROM TABLE
WHERE ..."""
Visualizing as DataFrame
First install pandas as it will not be present in your Virtual Environment
pip install pandas
After that import to your code and play around
import pandas as pd
df = pd.read_sql(sql_for_df, conn)
df.head()
To exit the VIrtual Enviroment just write exit in your terminal.
If you want to remove the Virtual Enviroment write in the terminal pipenv --rm
That's pretty much all I could learn so far.
I hope it helps you all.
# Install : ibm_db package
# Command : pip install ibm_db
import ibm_db
import sys
def get_connection():
db_name = ""
db_host_name = ""
db_port = ""
db_protocol = ""
db_username = ""
db_password = ""
try:
conn = ibm_db.connect(
f"DATABASE = {db_name}; HOSTNAME = {db_host_name}; PORT = {db_port}; PROTOCOL = {db_protocol}; "
f"UID = {db_username}; PWD = {db_password};", "", "")
return conn
except:
print("no connection:", ibm_db.conn_errormsg())
sys.exit(1)
get_connection()