Is there a way to connect Python to Db2?
The documentation is difficult to find, and once you find it, it's pretty abysmal. Here's what I've found over the past 3 hours.
You need to install ibm_db using pip, as follows:
pip install ibm_db
You'll want to create a connection object. The documentation is here.
Here's what I wrote:
from ibm_db import connect
# Careful with the punctuation here - we have 3 arguments.
# The first is a big string with semicolons in it.
# (Strings separated by only whitespace, newlines included,
# are automatically joined together, in case you didn't know.)
# The last two are emptry strings.
connection = connect('DATABASE=<database name>;'
'HOSTNAME=<database ip>;' # 127.0.0.1 or localhost works if it's local
'PORT=<database port>;'
'PROTOCOL=TCPIP;'
'UID=<database username>;'
'PWD=<username password>;', '', '')
Next you should know that commands to ibm_db never actually give you results. Instead, you need to call one of the fetch methods on the command, repeatedly, to get the results. I wrote this helper function to deal with that.
def results(command):
from ibm_db import fetch_assoc
ret = []
result = fetch_assoc(command)
while result:
# This builds a list in memory. Theoretically, if there's a lot of rows,
# we could run out of memory. In practice, I've never had that happen.
# If it's ever a problem, you could use
# yield result
# Then this function would become a generator. You lose the ability to access
# results by index or slice them or whatever, but you retain
# the ability to iterate on them.
ret.append(result)
result = fetch_assoc(command)
return ret # Ditch this line if you choose to use a generator.
Now with that helper function defined, you can easily do something like get the information on all the tables in your database with the following:
from ibm_db import tables
t = results(tables(connection))
If you'd like to see everything in a given table, you could do something like this now:
from ibm_db import exec_immediate
sql = 'LIST * FROM ' + t[170]['TABLE_NAME'] # Using our list of tables t from before...
rows = results(exec_immediate(connection, sql))
And now rows contains a list of rows from the 170th table in your database, where every row contains a dict of column name: value.
Hope this all helps.
After lots of digging I discovered how to connect with DB2 using ibm_db.
First off, if you use a python version higher than 3.2 use
pip install ibm_db==2.0.8a
version 2.0.8 (the latest) will fail to install.
then use the following to connect
import ibm_db_dbi as db
conn = db.connect("DATABASE=name;HOSTNAME=host;PORT=60000;PROTOCOL=TCPIP;UID=username;PWD=password;", "", "")
list tables with
for t in conn.tables():
print(t)
and execute SQL with
cursor = conn.cursor()
cursor.execute("SELECT * FROM Schema.Table")
for r in cursor.fetchall():
print(r)
check this link for official not so accurate documentation
ibm-db, the official DB2 driver for Python and Django is here:
https://code.google.com/p/ibm-db/
Here's a recent tutorial for how to install everything on Ubuntu Linux:
http://programmingzen.com/2011/05/12/installing-python-django-and-db2-on-ubuntu-11-04/
I should mention that there were several older unofficial DB2 drivers for Python. ibm-db is the one you should be using.
In addition to #prof1990 response:
Since 2.0.9 (Aug 16th 2018), also with Python 3 you can simply use:
pip install ibm_db
Reference:
https://github.com/ibmdb/python-ibmdb#updated-ibm_db
Example of connection here:
import ibm_db
ibm_db.connect("DATABASE=<dbname>;HOSTNAME=<host>;PORT=<60000>;PROTOCOL=TCPIP;UID=<username>;PWD=<password>;", "", "")
Full API documentation here:
https://github.com/ibmdb/python-ibmdb/wiki/APIs
You can connect to db2 from python using jaydeapi
First install library running pip install jaydeapi
download db2jcc4.jar
Then you can connect using below code :
by passing hostname,portno, userid,password database name
import jaydebeapi
conn_src = jaydebeapi.connect(
'com.ibm.db2.jcc.DB2Driver',
['YourHostName:PortNo/DatabaseName','userid','password'],'C:/db2jcc4.jar'
)
cursor=conn_src.cursor()
sql = 'Select * from schemaname.TableName fetch first 100 rows only '
cursor.execute(sql)
print("fetchall:")
result = cursor.fetchall()
for r in result:
print(r)
There is a way in which one can connect to IBM db2 using nothing but Python requests library. Worked for me.
STEP 1:
Go to IBM CLOUD Dashboard -> Navigate to your IBM db2 instance -> Click on 'Service Credentials'
A default one should be there, if not, create one. This service credential is a dictionary. Copy the service credentials.
STEP 2:
db2id = { // service credential dictionary here //}
api = "/dbapi/v3"
host = db2id['https_url']+api
userinfo = {"userid":db2id['username'],"password":db2id['password']}
service = '/auth/tokens'
r = requests.post(host+service,json=userinfo)
access_token = r.json()['token']
auth_header = {"Authorization": "Bearer "+access_token}
// Connection to database established
STEP 3
Now you can run SELECT, INSERT, DELETE, UPDATE queries
The format for INSERT, DELETE, UPDATE queries is the same. After an INSERT, DELETE, UPDATE query, a COMMIT query has to be sent, else changes aren't reflected. (You should commit your changes otherwise also)
INSERT / UPDATE / DELETE QUERIES
sql = " your insert/update/delete query here "
sql_command = {"commands":sql,"limit":1000,"separator":";","stop_on_error":"yes"}
service = "/sql_jobs"
r = requests.post(host+service,headers=auth_header,json=sql_command)
sql_command = {"commands":"COMMIT","limit":1000,"separator":";","stop_on_error":"yes"}
service = "/sql_jobs"
r = requests.post(host+service,headers=auth_header,json=sql_command)
You can use the variable r to check status of your request
SELECT QUERIES
sql = " your select query here "
service = "/sql_jobs"
r = requests.post(host+service,headers=auth_header,json=sql_command)
jobid = r.json()['id']
r = requests.get(host+service+"/"+jobid,headers=auth_header)
results = r.json()['results']
rows = results[0]['rows']
The variable rows will have the results of your query. Use it as per your convenience.
I didn't use any DDL queries. But I think they should work like the DML queries. Not sure though!
IBM's Db2 is available for various platforms. If you are trying to connect to a Db2 which lives on an IBM i server (formerly known as AS/400, iSeries, or System i), then ibm_db requires a product called Db2 Connect, which is rather expensive. Most people who use Python to connect to Db2 for i use ODBC (usually through PyODBC).
I'm not completely sure about the situation with Db2 on their z (mainframe) servers, but I would think it also requires Db2 Connect.
There are many ways to connect from Python to Db2. I am trying to provide a summary of options. Note that in many environments SSL/TLS is enforced now which requires additional parameters (see below).
Db2 and Python drivers
Db2 does not offer one, but four drivers (clients) for Python. The Db2 documentation page "Python, SQLAlchemy, and Django Framework application development for IBM Database servers" provides a good overview about the four drivers:
ibm_db is based on the IBM-defined API,
ibm_db_dbi is a driver for the Python database API (DBI),
ibm_db_sa implements the Python SQLAlchemy interface and
ibm_db_django serves as Db2 driver in the Django Framework.
Note that there are additional Python database interfaces which make use of existing JDBC or ODBC drivers which can be used to connect to Db2. You can use SQLAlchemy (ibm_db_sa) with the popular Flask framework. To use Db2 with pandas utilize ibm_db_dbi. All of the above Db2 drivers are available on GitHub and are based on the CLI (Call Level Interface / ODBC). There are additional ways to connect to Db2, e.g., by using 3rd party ODBC-based wrappers and more.
Db2 connections
Typical connection information is made up of the Db2 server (hostname), the port, the database name and username / password information. If nothing else is specified, most drivers assume that the connection is not encrypted. Thus, to connect over an encrypted connection more parameters are needed. They depend on the Db2 version, the type of Db2 product and some more. Let's start easy.
Newer Db2 versions simplified the use of SSL/TLS because certificates are now part of the package. A typical connection string would then look like this:
conn_str='database=MYDB;hostname=db2host.example.com;port=50001;protocol=tcpip;uid=db2inst1;pwd=secret;security=SSL'
ibm_db_conn = ibm_db.connect(conn_str,'','')
An important parameter is "security=SSL" to tell the driver to use encryption for the data in transit.
Db2 connection strings can have even more options. It depends on what security plugin is enabled. See this blog post on connecting from Python to Db2 for more links and discussions.
SQL Alchemy connection
When using Db2 with SQLAlchemy, pass an URI similar to
ibm_db_sa://user:password#hostname:port/database?Security=SSL
to get the connection established.
You can use ibm_db library to connect DB2.
query_str = "SELECT COUNT(*) FROM table_name"
conn = ibm_db.pconnect("dsn=write","usrname","secret")
query_stmt = ibm_db.prepare(conn, query_str)
ibm_db.execute(query_stmt)
This is for future reference:
Official installation docs say:
Python 2.5 or later, excluding Python 3.X.
pip install ibm_db
It only worked on Python 2.7 for me; it didn't for 3.X. Also, I had to make Python 2.7 default (instead of Python 3) so that the installation would work (otherwise, there would be installation errors).
Official docs sample usage:
import ibm_db
ibm_db.connect("DATABASE=name;HOSTNAME=host;PORT=60000;PROTOCOL=TCPIP;UID=username; PWD=password;", "", "")
Version: ibm-db 3.0.2 - ibm-db==3.0.2
pip install ibm-db
Released: Jun 17, 2020
Connect to a local or cataloged database:
import ibm_db
conn = ibm_db.connect("database","username","password")
Connect to an uncataloged database:
import ibm_db
ibm_db.connect("DATABASE=name;HOSTNAME=host;PORT=60000;PROTOCOL=TCPIP;UID=username;
PWD=password;", "", "")
How I managed to do in 2021.
What you will need:
Python 3.7
PipEnv
Ibm-db
Ibm-db version is not important but this lib only works with Python 3.7 (current python version is 3.9).
Install Python 3.7.6 in your machine (this is the version that worked).
In your IDE create a new python file.
Let' create a Virtual Enviroment to make sure we will use Python 3.7
pip install pipenv
After installing
pipenv install --python 3.7
Activate the Virtual Environment
pipenv shell
You can use pip list to verify if you are in the new Virtual Enviroment - if list only shows 3 or 4 libs, it's because you are
Now you can download Ibm_db
pip install ibm-db
You may add this to your code to confirm what is the version you are using
from platform import python_version
print(python_version())
Now accessing the DB2
import ibm_db_dbi as db
# Connect to DB2B1 (keep Protocol as TCPIP)
conn = db.connect("DATABASE=DBNAME;HOSTNAME=hostname;PORT=port;PROTOCOL=TCPIP;UID=Your User;PWD=Your Password;", "", "")
Checking all tables available
for t in conn.tables():
print(t)
Your SQL code:
sql_for_df = """SELECT *
FROM TABLE
WHERE ..."""
Visualizing as DataFrame
First install pandas as it will not be present in your Virtual Environment
pip install pandas
After that import to your code and play around
import pandas as pd
df = pd.read_sql(sql_for_df, conn)
df.head()
To exit the VIrtual Enviroment just write exit in your terminal.
If you want to remove the Virtual Enviroment write in the terminal pipenv --rm
That's pretty much all I could learn so far.
I hope it helps you all.
# Install : ibm_db package
# Command : pip install ibm_db
import ibm_db
import sys
def get_connection():
db_name = ""
db_host_name = ""
db_port = ""
db_protocol = ""
db_username = ""
db_password = ""
try:
conn = ibm_db.connect(
f"DATABASE = {db_name}; HOSTNAME = {db_host_name}; PORT = {db_port}; PROTOCOL = {db_protocol}; "
f"UID = {db_username}; PWD = {db_password};", "", "")
return conn
except:
print("no connection:", ibm_db.conn_errormsg())
sys.exit(1)
get_connection()
Related
I've successfully setup a connection from a stock raspberry pi 4 to our local database running on windows 10.
In the terminal if I type:
tsql -S *servername* -U *username* -P *password*
select * from testlist_table
go
I get the result for the query and all is good for that test. It's probably worth mentioning I recieve back after the first tsql line:
default encoding is en_us.utf-8 or something odd like this and then:
using default encoding utf-8,
When using python code if I type
import pyodbc
cnxn = pyodbc.connect("Driver={freeTDS};Server=*servername*;DATABASE=*databasename*;UID=*userid*;pwd=*pwd*")
cursor = cnxn.cursor()
cursor.execute("Select * from TestList_Table")
I get the following error:
Unicode data in a Unicode-only collation or ntext data cannot be sent to clients using DB-Library (such as ISQL) or ODBC version 3.7 or earlier.
Any ideas? I've tried setting encoding / decoding but that hasn't help. If I select a particular field so far that's worked. Not excited to try and change all the code as I have a lot of columns and am referencing by column number sometimes in a big mess of code.
I'm going to assume you're running a relatively recent version of FreeTDS (version 0.95 or above; you can find your version with tsql -C). If you're using an earlier version and this doesn't work, trying changing the TDS_Version to 7.2 or 7.1.
Give this a try:
import pyodbc
cnxn = pyodbc.connect(
"Driver={freeTDS};Server=*servername*;DATABASE=*databasename*;"
"UID=*userid*;pwd=*pwd*;TDS_Version=7.3;CHARSET=UTF-8;"
)
cursor = cnxn.cursor()
cursor.execute("Select * from TestList_Table")
Hi everyone,
I'm trying to wrap my head around microsoft server 2017 and python script.
In general - I'm trying to store a table I took from a website (using bs4),
storing it in a panda df , and then simply put the results in a temp sql table.
I entered the following code (I'm skipping parts of the code because the python script
does work in python. Keep in mind I'm calling the script from microsoft sql server 2017):
CREATE PROC OTC
AS
BEGIN
EXEC sp_execute_external_script
#language = N'Python',
#script = N'
import bs4 as bs
import pandas as pd
import requests
....
r = requests.get(url, verify = False)
html = r.text
soup = bs.BeautifulSoup(html, "html.parser")
data_date = str(soup.find(id="ctl00_SPWebPartManager1_g_4be2cf24_5a47_472d_a6ab_4248c8eb10eb_ctl00_lDate").contents)
t_tab1 = soup.find(id="ctl00_SPWebPartManager1_g_4be2cf24_5a47_472d_a6ab_4248c8eb10eb_ctl00_NiaROGrid1_DataGrid1")
df = parse_html_table(1,t_tab1)
print(df)
OutputDataSet=df
'
I tried the microsoft tutorials and simply couldn't understand how to
handle the inputs/outputs to get the result as a sql table.
Furthermore, I get the error
"
import bs4 as bs
ImportError: No module named 'bs4'
"
I'm obviously missing a lot here.
What am I to add to the sql code?
does the sql server even supports bs4? or only pandas?
and then I need to find another solution like write as csv?
Thanks for any help or advice you can offer
To use pip to install a Python package on SQL Server 2017:
On the server, open a command prompt as administrator.
Then cd to {instance directory}\PYTHON_SERVICES\Scripts
(for example: C:\Program Files\Microsoft SQL Server\MSSQL14.SQL2017\PYTHON_SERVICES\Scripts).
Then execute pip install {package name}.
One you have the necessary package(s) installed and the script executes successfully, simply setting variable OutputDataSet to a pandas data frame will result in the contents of that data frame being returned as a result set from the stored procedure.
If you want to capture that result set in a table (perhaps a temporary table), you can use INSERT...EXEC (e.g. INSERT MyTable(Col1, Col2) EXEC sp_execute_external_script ...).
I am trying to make the best out of an aws server and had the idea to use an in memory database across multiple threads(using SQLite 3 in python) I found this command online:
conn = sqlite3.connect('file::memory:?cache=shared')
but then I get this vague error:
sqlite3.OperationalError: unable to open database file
Is it even possible to do this anymore?
It is still possible. I just verified against Python 3.6.0 and Python 2.7.13 on MacOS.
sqlite3.connect("file::memory:?cache=shared") is indeed the correct way to connect to DB.
import sqlite3
p = sqlite3.connect("file::memory:?cache=shared")
p.execute('CREATE TABLE foo (bar, baz)')
p.execute("INSERT INTO foo VALUES ('apple', 'orange')")
p.commit()
and in another python shell
import sqlite3
q = sqlite3.connect("file::memory:?cache=shared")
list(q.execute('SELECT * FROM foo'))
my output is [(u'apple', u'orange')]
To you answer your question "Is it even possible to do this anymore?", the answer is yes. So the problem lies in your system, as you confirmed it works on aws (in the comments below).
In Python 3.4+ and SQLite 3.7.13+, you can use this approach:
sqlite3.connect("file:memory?cache=shared&mode=memory", uri=True)
Hi I am new to python (programming and stackoverflow). First let me start by saying a little bit about what I am doing and trying to do.
I am using an internal XML API to pull data from an internal database
I parse/format xml result into a txt document (automated this to occur at a
set interval)
I want to write or import the contents of this document to an oracle database
How would i go about importing or writing this document into an existing oracle database? I can't seem to find much in the way of documentation with regards to the cx_Oracle module that i am using to establish a connection with my database. Could any of you kind folk point me in a direction / resource to accomplish this?
SHORT ANSWER:
query = """
insert into TABLE(FIELD1, FIELD2, ...) values (VAL1, VAL2, ...)
"""
cur = con.cursor()
cur.execute(query)
cur.commit()
I strongly suggest you to use prepared statements.
LONG ANSWER:
This is for Linux, in particular for Red Hat. Only the Python code at the end can be used on every OS. Try to adapt these steps to your OS.
0: Install the packages libaio and python-dev (or python-devel, check your distro)
1: If you don't have pip, install it
2: Install oracle instantclient-basic and instantclient-sdk (or instantclient-devel) from Oracle site
3: Launch these commands using bash. If you don't have /etc/profile.d/, check your distro.
echo 'ORACLE_HOME="/usr/lib/oracle/12.1/client64"' | \
sudo tee /etc/profile.d/cx_oracle.sh
pip install cx_Oracle
4: Logout and login again
5: Before using cx_Oracle, you have to set LD_LIBRARY_PATH I recommend you to NOT set it globally:
export LD_LIBRARY_PATH="$ORACLE_HOME/lib"
6: Finally, the Python code:
import cx_Oracle
os.environ["NLS_LANG"] = "AMERICAN_AMERICA.UTF8"
con_str = "USERNAME/PASSWORD#HOST:PORT/DBNAME"
con = cx_Oracle.connect(con_str)
query = """
select 1 from dual
"""
cur = con.cursor()
cur.execute(query)
rows = cur.fetchall()
for row in rows:
print(row) # it should print "1"
con.close()
You have to change the con_str with your username, password etc. The line that sets utf-8 encoding is optional and adaptable to your needs, but recommended.
7: If you want to insert rows:
query = """
insert into TABLE(FIELD1, FIELD2, ...) values (VAL1, VAL2, ...)
"""
cur = con.cursor()
cur.execute(query)
cur.commit()
I strongly suggest you to use prepared statements if you can't trust the source of the data.
Sources:
http://chilipuppy.blogspot.it/2008/10/purpose-im-working-on-building-python.html
http://agiletesting.blogspot.it/2005/05/installing-and-using-cxoracle-on-unix.html
http://cx-oracle.readthedocs.org/en/latest/index.html
Personal hassle
Your question is basically "how do I get started with cx_Oracle?"
There's some snippets here:
http://markharrison.net/cx-oracle-demos
and your simplest cx_Oracle program is something like this:
import cx_Oracle
conn = cx_Oracle.connect('scott/tiger')
curs = conn.cursor()
curs.execute('select 2+2 from dual')
print curs.fetchall()
curs.execute('insert into mytable(x) values(3)')
conn.commit()
curs.execute('select * from mytable')
for row in curs:
print row
conn.close()
I'm new to python. I'm trying to query a MSSQL database.
import pymssql
conn = pymssql.connect(host='hostname', user='username', password='password', database='dbname')
cursor = conn.cursor()
sql = "select count(*) from T_Email with (nolock) where Transmit is null"
cursor.execute(sql)
results = cursor.fetchall()
for row in results:
print (row)
The query successfully runs is Microsoft SQL Server Management Studio, but my python script always returns nothing.
I verified I have network connectivity. I verified the username, password and database name. If I change the password, then the script will give an error.
I have tried results = cursor.fetchone(), but that didn't help.
Any suggestions?
If you're using Ubuntu you may have used (like me) apt-get to install pymssql package.
This is a known bug of the bundled version: https://bugs.launchpad.net/ubuntu/+source/pymssql/+bug/918896
Try to install it manually using easy_install.
I had the same issue on Ubuntu 12.04, indeed the fix is doing the sequence:
$ apt-get purge python-pymssql
$ apt-get install freetds-dev
$ pip install Cython
$ pip install pymssql
Try adding a conn.commit() to your query
Without sufficient information to reproduce the example, it's hard to say the specific problem you're having. However, here are a few guesses I have as for possible problems:
Maybe your actual column name (presuming your example above was just a mock up) is too long. See: http://code.google.com/p/pymssql/wiki/FAQ (look for Column names get silently truncated to 30 characters. (1.x only) ) This is a common one to trip folks up because it SILENTLY fails!!
If you are creating tables before querying into them, or other things that require commits, it can mess things up, even with autocommit turned on ( autocommit(1) ). See my answer to myself :) at pymssql ( python module ) unable to use temporary tables
Good luck!
Mike
import pymssql
conn = pymssql.connect(
server="server",
port=port,
user="user",
password=password,
database="database")
conn
cursor = conn.cursor()
cursor.execute("select count(*) from T_Email with (nolock) where Transmit is null")
for row in cursor.fetchall():
print ("%s\n" % str(row))
conn.close()
Change the following lines in your code snippet
results = cursor.fetchall()
for row in results:
print (row)
into
# results = cursor.fetchall()
for row in cursor:
print (row)
pymssql has bug in cursor.fetchall()
For reference https://github.com/pymssql/pymssql/issues/141