We Keep Coding

Using freeTDS to connect to SQL Server database working with TSQL but not completely in python

I've successfully setup a connection from a stock raspberry pi 4 to our local database running on windows 10.
In the terminal if I type:
tsql -S *servername* -U *username* -P *password*
select * from testlist_table
go
I get the result for the query and all is good for that test. It's probably worth mentioning I recieve back after the first tsql line:
default encoding is en_us.utf-8 or something odd like this and then:
using default encoding utf-8,
When using python code if I type
import pyodbc
cnxn = pyodbc.connect("Driver={freeTDS};Server=*servername*;DATABASE=*databasename*;UID=*userid*;pwd=*pwd*")
cursor = cnxn.cursor()
cursor.execute("Select * from TestList_Table")
I get the following error:
Unicode data in a Unicode-only collation or ntext data cannot be sent to clients using DB-Library (such as ISQL) or ODBC version 3.7 or earlier.
Any ideas? I've tried setting encoding / decoding but that hasn't help. If I select a particular field so far that's worked. Not excited to try and change all the code as I have a lot of columns and am referencing by column number sometimes in a big mess of code.

I'm going to assume you're running a relatively recent version of FreeTDS (version 0.95 or above; you can find your version with tsql -C). If you're using an earlier version and this doesn't work, trying changing the TDS_Version to 7.2 or 7.1.
Give this a try:
import pyodbc
cnxn = pyodbc.connect(
"Driver={freeTDS};Server=*servername*;DATABASE=*databasename*;"
"UID=*userid*;pwd=*pwd*;TDS_Version=7.3;CHARSET=UTF-8;"
)
cursor = cnxn.cursor()
cursor.execute("Select * from TestList_Table")

Running simple query through python: No results

I am trying to learn how to get Microsoft SQL query results using python and pyodbc module and have run into an issue in returning the same results using the same query that I use in Microsoft SQL Management Studio.
I've looked at the pyodbc documentation and set up my connection correctly... at least I'm not getting any connection errors at execution. The only issue seems to be returning the table data
import pyodbc
import sys
import csv
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=<server>;DATABASE=<db>;UID=<uid>;PWD=<PWD>')
cursor = cnxn.cursor()
cursor.execute("""
SELECT request_id
From audit_request request
where request.reception_datetime between '2019-08-18' and '2019-08-19' """)
rows = cursor.fetchall()
for row in cursor:
print(row.request_id)
When I run the above code i get this in the python terminal window:
Process returned 0 (0x0) execution time : 0.331 s
Press any key to continue . . .
I tried this same query in SQL Management Studio and it returns the results I am looking for. There must be something I'm missing as far as displaying the results using python.

You're not actually setting your cursor up to be used. You should have something like this before executing:
cursor = cnxn.cursor()
Learn more here: https://github.com/mkleehammer/pyodbc/wiki/Connection#cursor

importing data to oracle using python cx_oracle

Hi I am new to python (programming and stackoverflow). First let me start by saying a little bit about what I am doing and trying to do.
I am using an internal XML API to pull data from an internal database
I parse/format xml result into a txt document (automated this to occur at a
set interval)
I want to write or import the contents of this document to an oracle database
How would i go about importing or writing this document into an existing oracle database? I can't seem to find much in the way of documentation with regards to the cx_Oracle module that i am using to establish a connection with my database. Could any of you kind folk point me in a direction / resource to accomplish this?

SHORT ANSWER:
query = """
insert into TABLE(FIELD1, FIELD2, ...) values (VAL1, VAL2, ...)
"""
cur = con.cursor()
cur.execute(query)
cur.commit()
I strongly suggest you to use prepared statements.
LONG ANSWER:
This is for Linux, in particular for Red Hat. Only the Python code at the end can be used on every OS. Try to adapt these steps to your OS.
0: Install the packages libaio and python-dev (or python-devel, check your distro)
1: If you don't have pip, install it
2: Install oracle instantclient-basic and instantclient-sdk (or instantclient-devel) from Oracle site
3: Launch these commands using bash. If you don't have /etc/profile.d/, check your distro.
echo 'ORACLE_HOME="/usr/lib/oracle/12.1/client64"' | \
sudo tee /etc/profile.d/cx_oracle.sh
pip install cx_Oracle
4: Logout and login again
5: Before using cx_Oracle, you have to set LD_LIBRARY_PATH I recommend you to NOT set it globally:
export LD_LIBRARY_PATH="$ORACLE_HOME/lib"
6: Finally, the Python code:
import cx_Oracle
os.environ["NLS_LANG"] = "AMERICAN_AMERICA.UTF8"
con_str = "USERNAME/PASSWORD#HOST:PORT/DBNAME"
con = cx_Oracle.connect(con_str)
query = """
select 1 from dual
"""
cur = con.cursor()
cur.execute(query)
rows = cur.fetchall()
for row in rows:
print(row) # it should print "1"
con.close()
You have to change the con_str with your username, password etc. The line that sets utf-8 encoding is optional and adaptable to your needs, but recommended.
7: If you want to insert rows:
query = """
insert into TABLE(FIELD1, FIELD2, ...) values (VAL1, VAL2, ...)
"""
cur = con.cursor()
cur.execute(query)
cur.commit()
I strongly suggest you to use prepared statements if you can't trust the source of the data.
Sources:
http://chilipuppy.blogspot.it/2008/10/purpose-im-working-on-building-python.html
http://agiletesting.blogspot.it/2005/05/installing-and-using-cxoracle-on-unix.html
http://cx-oracle.readthedocs.org/en/latest/index.html
Personal hassle

Your question is basically "how do I get started with cx_Oracle?"
There's some snippets here:
http://markharrison.net/cx-oracle-demos
and your simplest cx_Oracle program is something like this:
import cx_Oracle
conn = cx_Oracle.connect('scott/tiger')
curs = conn.cursor()
curs.execute('select 2+2 from dual')
print curs.fetchall()
curs.execute('insert into mytable(x) values(3)')
conn.commit()
curs.execute('select * from mytable')
for row in curs:
print row
conn.close()

python script hangs when calling cursor.fetchall() with large data set

I have a query that returns over 125K rows.
The goal is to write a script the iterates through the rows, and for each, populate a second table with data processed from the result of the query.
To develop the script, I created a duplicate database with a small subset of the data (4126 rows)
On the small database, the following code works:
import os
import sys
import random
import mysql.connector
cnx = mysql.connector.connect(user='dbuser', password='thePassword',
host='127.0.0.1',
database='db')
cnx_out = mysql.connector.connect(user='dbuser', password='thePassword',
host='127.0.0.1',
database='db')
ins_curs = cnx_out.cursor()
curs = cnx.cursor(dictionary=True)
#curs = cnx.cursor(dictionary=True,buffered=True) #fail
with open('sql\\getRawData.sql') as fh:
sql = fh.read()
curs.execute(sql, params=None, multi=False)
result = curs.fetchall() #<=== script stops at this point
print len(result) #<=== this line never executes
print curs.column_names
curs.close()
cnx.close()
cnx_out.close()
sys.exit()
The line curs.execute(sql, params=None, multi=False) succeeds on both the large and small databases.
If I use curs.fetchone() in a loop, I can read all records.
If I alter the line:
curs = cnx.cursor(dictionary=True)
to read:
curs = cnx.cursor(dictionary=True,buffered=True)
The script hangs at curs.execute(sql, params=None, multi=False).
I can find no documentation on any limits to fetchall(), nor can I find any way to increase the buffer size, and no way to tell how large a buffer I even need.
There are no exceptions raised.
How can I resolve this?

I was having this same issue, first on a query that returned ~70k rows and then on one that only returned around 2k rows (and for me RAM was also not the limiting factor). I switched from using mysql.connector (i.e. the mysql-connector-python package) to MySQLdb (i.e. the mysql-python package) and then was able to fetchall() on large queries with no problem. Both packages seem to follow the python DB API, so for me MySQLdb was a drop-in replacement for mysql.connector, with no code changes necessary beyond the line that sets up the connection. YMMV if you're leveraging something specific about mysql.connector.
Pragmatically speaking, if you don't have a specific reason to be using mysql.connector the solution to this is just to switch to a package that works better!

How to connect Python to Db2

Is there a way to connect Python to Db2?

The documentation is difficult to find, and once you find it, it's pretty abysmal. Here's what I've found over the past 3 hours.
You need to install ibm_db using pip, as follows:
pip install ibm_db
You'll want to create a connection object. The documentation is here.
Here's what I wrote:
from ibm_db import connect
# Careful with the punctuation here - we have 3 arguments.
# The first is a big string with semicolons in it.
# (Strings separated by only whitespace, newlines included,
# are automatically joined together, in case you didn't know.)
# The last two are emptry strings.
connection = connect('DATABASE=<database name>;'
'HOSTNAME=<database ip>;' # 127.0.0.1 or localhost works if it's local
'PORT=<database port>;'
'PROTOCOL=TCPIP;'
'UID=<database username>;'
'PWD=<username password>;', '', '')
Next you should know that commands to ibm_db never actually give you results. Instead, you need to call one of the fetch methods on the command, repeatedly, to get the results. I wrote this helper function to deal with that.
def results(command):
from ibm_db import fetch_assoc
ret = []
result = fetch_assoc(command)
while result:
# This builds a list in memory. Theoretically, if there's a lot of rows,
# we could run out of memory. In practice, I've never had that happen.
# If it's ever a problem, you could use
# yield result
# Then this function would become a generator. You lose the ability to access
# results by index or slice them or whatever, but you retain
# the ability to iterate on them.
ret.append(result)
result = fetch_assoc(command)
return ret # Ditch this line if you choose to use a generator.
Now with that helper function defined, you can easily do something like get the information on all the tables in your database with the following:
from ibm_db import tables
t = results(tables(connection))
If you'd like to see everything in a given table, you could do something like this now:
from ibm_db import exec_immediate
sql = 'LIST * FROM ' + t[170]['TABLE_NAME'] # Using our list of tables t from before...
rows = results(exec_immediate(connection, sql))
And now rows contains a list of rows from the 170th table in your database, where every row contains a dict of column name: value.
Hope this all helps.

After lots of digging I discovered how to connect with DB2 using ibm_db.
First off, if you use a python version higher than 3.2 use
pip install ibm_db==2.0.8a
version 2.0.8 (the latest) will fail to install.
then use the following to connect
import ibm_db_dbi as db
conn = db.connect("DATABASE=name;HOSTNAME=host;PORT=60000;PROTOCOL=TCPIP;UID=username;PWD=password;", "", "")
list tables with
for t in conn.tables():
print(t)
and execute SQL with
cursor = conn.cursor()
cursor.execute("SELECT * FROM Schema.Table")
for r in cursor.fetchall():
print(r)
check this link for official not so accurate documentation

ibm-db, the official DB2 driver for Python and Django is here:
https://code.google.com/p/ibm-db/
Here's a recent tutorial for how to install everything on Ubuntu Linux:
http://programmingzen.com/2011/05/12/installing-python-django-and-db2-on-ubuntu-11-04/
I should mention that there were several older unofficial DB2 drivers for Python. ibm-db is the one you should be using.

In addition to #prof1990 response:
Since 2.0.9 (Aug 16th 2018), also with Python 3 you can simply use:
pip install ibm_db
Reference:
https://github.com/ibmdb/python-ibmdb#updated-ibm_db
Example of connection here:
import ibm_db
ibm_db.connect("DATABASE=<dbname>;HOSTNAME=<host>;PORT=<60000>;PROTOCOL=TCPIP;UID=<username>;PWD=<password>;", "", "")
Full API documentation here:
https://github.com/ibmdb/python-ibmdb/wiki/APIs

You can connect to db2 from python using jaydeapi
First install library running pip install jaydeapi
download db2jcc4.jar
Then you can connect using below code :
by passing hostname,portno, userid,password database name
import jaydebeapi
conn_src = jaydebeapi.connect(
'com.ibm.db2.jcc.DB2Driver',
['YourHostName:PortNo/DatabaseName','userid','password'],'C:/db2jcc4.jar'
)
cursor=conn_src.cursor()
sql = 'Select * from schemaname.TableName fetch first 100 rows only '
cursor.execute(sql)
print("fetchall:")
result = cursor.fetchall()
for r in result:
print(r)

There is a way in which one can connect to IBM db2 using nothing but Python requests library. Worked for me.
STEP 1:
Go to IBM CLOUD Dashboard -> Navigate to your IBM db2 instance -> Click on 'Service Credentials'
A default one should be there, if not, create one. This service credential is a dictionary. Copy the service credentials.
STEP 2:
db2id = { // service credential dictionary here //}
api = "/dbapi/v3"
host = db2id['https_url']+api
userinfo = {"userid":db2id['username'],"password":db2id['password']}
service = '/auth/tokens'
r = requests.post(host+service,json=userinfo)
access_token = r.json()['token']
auth_header = {"Authorization": "Bearer "+access_token}
// Connection to database established
STEP 3
Now you can run SELECT, INSERT, DELETE, UPDATE queries
The format for INSERT, DELETE, UPDATE queries is the same. After an INSERT, DELETE, UPDATE query, a COMMIT query has to be sent, else changes aren't reflected. (You should commit your changes otherwise also)
INSERT / UPDATE / DELETE QUERIES
sql = " your insert/update/delete query here "
sql_command = {"commands":sql,"limit":1000,"separator":";","stop_on_error":"yes"}
service = "/sql_jobs"
r = requests.post(host+service,headers=auth_header,json=sql_command)
sql_command = {"commands":"COMMIT","limit":1000,"separator":";","stop_on_error":"yes"}
service = "/sql_jobs"
r = requests.post(host+service,headers=auth_header,json=sql_command)
You can use the variable r to check status of your request
SELECT QUERIES
sql = " your select query here "
service = "/sql_jobs"
r = requests.post(host+service,headers=auth_header,json=sql_command)
jobid = r.json()['id']
r = requests.get(host+service+"/"+jobid,headers=auth_header)
results = r.json()['results']
rows = results[0]['rows']
The variable rows will have the results of your query. Use it as per your convenience.
I didn't use any DDL queries. But I think they should work like the DML queries. Not sure though!

IBM's Db2 is available for various platforms. If you are trying to connect to a Db2 which lives on an IBM i server (formerly known as AS/400, iSeries, or System i), then ibm_db requires a product called Db2 Connect, which is rather expensive. Most people who use Python to connect to Db2 for i use ODBC (usually through PyODBC).
I'm not completely sure about the situation with Db2 on their z (mainframe) servers, but I would think it also requires Db2 Connect.

There are many ways to connect from Python to Db2. I am trying to provide a summary of options. Note that in many environments SSL/TLS is enforced now which requires additional parameters (see below).
Db2 and Python drivers
Db2 does not offer one, but four drivers (clients) for Python. The Db2 documentation page "Python, SQLAlchemy, and Django Framework application development for IBM Database servers" provides a good overview about the four drivers:
ibm_db is based on the IBM-defined API,
ibm_db_dbi is a driver for the Python database API (DBI),
ibm_db_sa implements the Python SQLAlchemy interface and
ibm_db_django serves as Db2 driver in the Django Framework.
Note that there are additional Python database interfaces which make use of existing JDBC or ODBC drivers which can be used to connect to Db2. You can use SQLAlchemy (ibm_db_sa) with the popular Flask framework. To use Db2 with pandas utilize ibm_db_dbi. All of the above Db2 drivers are available on GitHub and are based on the CLI (Call Level Interface / ODBC). There are additional ways to connect to Db2, e.g., by using 3rd party ODBC-based wrappers and more.
Db2 connections
Typical connection information is made up of the Db2 server (hostname), the port, the database name and username / password information. If nothing else is specified, most drivers assume that the connection is not encrypted. Thus, to connect over an encrypted connection more parameters are needed. They depend on the Db2 version, the type of Db2 product and some more. Let's start easy.
Newer Db2 versions simplified the use of SSL/TLS because certificates are now part of the package. A typical connection string would then look like this:
conn_str='database=MYDB;hostname=db2host.example.com;port=50001;protocol=tcpip;uid=db2inst1;pwd=secret;security=SSL'
ibm_db_conn = ibm_db.connect(conn_str,'','')
An important parameter is "security=SSL" to tell the driver to use encryption for the data in transit.
Db2 connection strings can have even more options. It depends on what security plugin is enabled. See this blog post on connecting from Python to Db2 for more links and discussions.
SQL Alchemy connection
When using Db2 with SQLAlchemy, pass an URI similar to
ibm_db_sa://user:password#hostname:port/database?Security=SSL
to get the connection established.

You can use ibm_db library to connect DB2.
query_str = "SELECT COUNT(*) FROM table_name"
conn = ibm_db.pconnect("dsn=write","usrname","secret")
query_stmt = ibm_db.prepare(conn, query_str)
ibm_db.execute(query_stmt)

This is for future reference:
Official installation docs say:
Python 2.5 or later, excluding Python 3.X.
pip install ibm_db
It only worked on Python 2.7 for me; it didn't for 3.X. Also, I had to make Python 2.7 default (instead of Python 3) so that the installation would work (otherwise, there would be installation errors).
Official docs sample usage:
import ibm_db
ibm_db.connect("DATABASE=name;HOSTNAME=host;PORT=60000;PROTOCOL=TCPIP;UID=username; PWD=password;", "", "")

Version: ibm-db 3.0.2 - ibm-db==3.0.2
pip install ibm-db
Released: Jun 17, 2020
Connect to a local or cataloged database:
import ibm_db
conn = ibm_db.connect("database","username","password")
Connect to an uncataloged database:
import ibm_db
ibm_db.connect("DATABASE=name;HOSTNAME=host;PORT=60000;PROTOCOL=TCPIP;UID=username;
PWD=password;", "", "")

How I managed to do in 2021.
What you will need:
Python 3.7
PipEnv
Ibm-db
Ibm-db version is not important but this lib only works with Python 3.7 (current python version is 3.9).
Install Python 3.7.6 in your machine (this is the version that worked).
In your IDE create a new python file.
Let' create a Virtual Enviroment to make sure we will use Python 3.7
pip install pipenv
After installing
pipenv install --python 3.7
Activate the Virtual Environment
pipenv shell
You can use pip list to verify if you are in the new Virtual Enviroment - if list only shows 3 or 4 libs, it's because you are
Now you can download Ibm_db
pip install ibm-db
You may add this to your code to confirm what is the version you are using
from platform import python_version
print(python_version())
Now accessing the DB2
import ibm_db_dbi as db
# Connect to DB2B1 (keep Protocol as TCPIP)
conn = db.connect("DATABASE=DBNAME;HOSTNAME=hostname;PORT=port;PROTOCOL=TCPIP;UID=Your User;PWD=Your Password;", "", "")
Checking all tables available
for t in conn.tables():
print(t)
Your SQL code:
sql_for_df = """SELECT *
FROM TABLE
WHERE ..."""
Visualizing as DataFrame
First install pandas as it will not be present in your Virtual Environment
pip install pandas
After that import to your code and play around
import pandas as pd
df = pd.read_sql(sql_for_df, conn)
df.head()
To exit the VIrtual Enviroment just write exit in your terminal.
If you want to remove the Virtual Enviroment write in the terminal pipenv --rm
That's pretty much all I could learn so far.
I hope it helps you all.

# Install : ibm_db package
# Command : pip install ibm_db
import ibm_db
import sys
def get_connection():
db_name = ""
db_host_name = ""
db_port = ""
db_protocol = ""
db_username = ""
db_password = ""
try:
conn = ibm_db.connect(
f"DATABASE = {db_name}; HOSTNAME = {db_host_name}; PORT = {db_port}; PROTOCOL = {db_protocol}; "
f"UID = {db_username}; PWD = {db_password};", "", "")
return conn
except:
print("no connection:", ibm_db.conn_errormsg())
sys.exit(1)
get_connection()