I have been at this for a while, trying all kinds of different packages from openSource, IBM, and many others. I have not yet found one that works without some sort of confusing install method that I can not get to work, or some sort of integration with other third-party pieces that I can not seem to get working.
I am simply trying to perform SQL statements on a Informix Server using Python. No different than mySQL and other tools. Using cursors or full result dumps, really do not care. I want to be able to formalize a query string statically or dynamically and then tell whatever tools/module to execute said query and return results (if any).
I have tried:
ibm_db 2.0.5.1 (https://pypi.python.org/pypi/ibm_db)
IBM Informix Client SDK
pymssql
unixODBC
Looked at but do not want to use Jython (JPython).
What I have managed:
I have been able to install and get the IBM Informix Client SDK installed and working. I can connect to my Informix DB server and perform queries.
I have mySQL working and connecting and querying.
I have written a Java program to perform queries using a Java driver, compiled it, combined it with a bash script to perform queries and email results.
I am just stumped. Looking for assistance on what to download (URLs), how to go about installing it (tips and tricks, environment variables, where to install it, etc..) I want to have something that does not depend on Java or writing Java, etc. I am looking for a solution that may will give me the ability to write Python to query, insert, update, and delete from an Informix database and tables. I want to combine my previously written Java and Bash script into a Python script.
Frustrated and looking for any assistance.
Thank you for listening and please ask questions if you do not understand my plea.
Informix on Linux is a bag of pain. My personal setup to get Informix-connect to work with CPython3 is stacking the Informix Client SDK with unixODBC and pyodbc. There are some hoops to jump through, none of which are documented. Almost all the setup is completely useless yet required to prevent some parts of the Informix-driver to bail out. Note that some options are case- and space-sensitive (Description=Informix != description = Informix).
Install the Informix Client SDK. You don't need all the garbage that comes in the package, just Informix Connect. I assume you use the default path /opt/IBM/informix
Add /opt/IBM/informix/lib/cli and /opt/IBM/informix/lib/esql to your dynamic linker lookup paths. On Fedora you can do this by putting them in a new file /etc/ld.so.conf.d/informix.conf
Create a new /etc/odbc.ini and add the following:
[ODBC Data Sources]
Infdrv1=IBM INFORMIX ODBC DRIVER
[Infdrv1]
Driver=/opt/IBM/informix/lib/cli/iclit09b.so
Description=Informix
Database=WHATEVER_YOUR_DB_NAME_IS
Servername=WHATEVER_YOUR_SERVER_NAME_IS
CLIENT_LOCALE=en_us.8859-1 # MAY BE DIFFERENT
DB_LOCALE=en_us.819 # MAY BE DIFFERENT
[ODBC]
UNICODE=UCS-2
Create a new /etc/odbcinst.ini and add the following
[IBM INFORMIX ODBC DRIVER]
Description=Informix Driver
Driver=libifcli.so
You need to set the environment variables INFORMIXDIR and ODBCINI. On Fedora you may add a new file /etc/profile.d/informix.sh and add
export INFORMIXDIR=/opt/IBM/informix
export ODBCINI=/etc/odbc.ini
Edit /opt/IBM/informix/etc/sqlhosts and put your basic connection information there. In the most simple case it has only one line that reads
YOUR_SERVER_NAME\tonsoctcp\tYOUR_DB_NAME\tpdap-np
Note that pdap-np is actually port 1526 which is also the Informix "Turbo"-Driver tcp port. See your /etc/services
Create an empty .odbc.ini in your $HOME e.g. by touch $HOME/.odbc.ini. It needs to be there. It needs to be 0 bytes. I love this part.
Install unixODBC and pyodbc from your favorite repository.
Remember to get your env-changes going, e.g. via reboot. You can now connect like this:
import pyodbc
DRIVER = 'IBM INFORMIX ODBC DRIVER'
SERVER = 'YOUR_SERVER_NAME'
DATABASE = 'YOUR_DB_NAME'
constr = 'DRIVER={%s};SERVER=%s;DATABASE=%s;UID=%s;PWD=%s' % (DRIVER, SERVER, DATABASE, USER, PASS)
con = pyodbc.connect(constr, autocommit=False)
From there on you can get your cursor, execute queries, fetch results and such. Note that there are numerous bugs in quirks in IBM's ODBC-driver, out of my head:
Rows that contain NULLs may cause a segfault as the IBM driver puts a 32bit int where a 64bit int is expected to signal the value being null. In case you are affected by this, you need to patch unixODBC for all possible column types to deal with this.
Columns without names cause the driver to segfault (e.g. SELECT COUNT(*) FROM foobar needs to be SELECT COUNT(*) AS c FROM foobar).
Make sure your encoding actually works as expected. UTF8 is something not enterprise-enough for IBM and UCS-2 is the only thing I got to work.
Related
I found this website and it suggest that I might be able to connect to my Oracle ADW cloud database using python. I tried running the below code but keep running into the same error. Anyone have any insight on how to resolve this? Note: Password is changed for obvious reasons.
Code in Jupyter Notebooks:
import cx_Oracle as cx
import pandas as pd
import warnings
warnings.filterwarnings('ignore')
pswd = 'ABC'
#Connect to Autonomous Data Warehouse
con = cx.connect(user = 'ADMIN', password = pswd)
query = 'SELECT * from TEST123'
data_train = pd.read_sql(query, con=con)
Error:
DatabaseError: Error while trying to retrieve text for error ORA-01804
I get the same error when I run the below code:
...
#Connect to Autonomous Data Warehouse
con = cx.connect('ADMIN',pswd,"mltest_high")
query = 'SELECT * from TEST123'
data_train = pd.read_sql(query, con=con)
So this took a lot of education in order to figure out especially when it came to how Oracle wallets work inline with SLQNET.ora and TNS_NAMES.ora file in conjunction with system environmental variables but this website did get my python (in .ipynb) in visual studio code to work in being able to connect with Oracle's cloud ADW system. It is almost exactly what I did to get it to work on my machine but I didn't do the virtual environment. I had to figure out a work around with the items stated above but was able to instead use the system link to my wallet for the directory.
It is important to know that you need to do these things to get it to work. When you download the wallet from ADW, you need to copy the high/medium/low lines from the TNS_NAMES and paste that in your Oracle/network/admin/tns_names.ora file. You will also need to take the wallet information and ssl server from the sqlnet.ora file and put it in the sqlnet.ora file in the Oracle/network/admin/ directory as well. If you chose not to use the virtual environment as demonstrated in the post, to get the directory link, for the wallet information line, to work, you'll need to set said directory to the directory of the wallet folder. I unzipped mine; unsure if needed or not.
Lastly, you will need to set your system environmental variables for TNS_NAMES to wherever your tns_names.ora and sqlnet.ora system files are (not the ones that come in the wallet download folder), likely in Oracle\network\admin.
Below is the code that worked for me. I hope this helps someone else and that they don't have to go through the same hoops that I had to in order to figure it out.
import cx_Oracle
import os
import pandas as pd
os.environ.get('TNS_ADMIN')
connection = cx_Oracle.connect('<Oracle ADW Username', '<Oracle ADW Password>', '<TNS_NAME entry (high/med/low)>')
cursor = connection.cursor()
rs = cursor.execute("SELECT * FROM TEST123")
df = pd.DataFrame(rs.fetchall())
df
At a guess from the error number and fact the message text wasn't found, cx_Oracle is using Oracle Instant Client libraries, but you have the ORACLE_HOME environment variable set to some other software. If so, unset ORACLE_HOME. Or perhaps you are only using libraries included in a local Oracle DB install and haven't fully set the Oracle environment variables e.g. haven't set ORACLE_HOME. Or perhaps you might need a more recent version of the Oracle client libraries - get 19c libraries e.g Oracle Instant Client. Also check other StackOverflow questions about ORA-1804. If you update your question with information about what Oracle software you have installed on the computer running Python, a more detailed answer might be possible.
It sounds like you have got the cloud wallet sorted out for connection, but here are references for people coming to this question after reading your heading:
A blog post How to connect to Oracle Autonomous Cloud Databases
cx_Oracle documentation Connecting to Autononmous Databases
Oracle ADW documentation: Connect with Python, Node.js, and other Scripting Languages
Currently I am trying to make a connection with a foxpro database using the python win32com module.
The Python code currently looks like this:
import win32com.client
conn = win32com.client.Dispatch('ADODB.Connection')
dsn = 'Provider=vfpoledb;Data Source=C:\MyDbFolder\MyDbContainer.dbc;'
conn.Open(dsn)
print('ok')
However it says that it could not find the provider; even-tough I have successfully installed the latest version of the Microsoft OLE DB Provider for Visual FoxPro 9.0 from the microsoft website.
'Provider cannot be found. It may not be properly installed.'
I have tried this both with python 32 bit as well as the 64 bit version on different pc's. If you use 32 bit python it works. However if one needs to use 64bit python, it seems that this should as well.
Did anyone get this working without issues?
One possible workaround for the lack of a 64-bit VFPOLEDB driver might be setting up the VFP database as a linked server in a 32-bit instance of MS SQL Server (Express is free and should work). SQL Server 2014 seems to be the last version for which a 32-bit edition is available. There are plenty 64-bit OLEDB drivers for SQL Server, and they don't care about the bitness of the instance.
There are step-by-step instructions in How to successfully connect to Foxpro database files using MSSQL Linked Server feature and ODBC? over on ServerFault
Note: using Fox data via a linked server is severely limiting and nowhere near as powerful as using Fox directly or via VFPOLEDB. However, sometimes limited access is better than no access at all.
The queries have to use SQL server syntax and they are limited by it. For example, boolean fields get mapped to the bit data type (0 or 1) because SQL Server has no concept of booleans. But inside an OpenQuery call you can use full Fox syntax. Assuming the linked server is called FOX and table StoffPZN has a boolean field op:
select * from FOX...StoffPZN where op = 1; -- T-SQL rules
select * from openquery(FOX, 'sele * from StoffPZN wher op'); -- Fox rules
I currently use PYODBC to connect to MS SQL Server and MYSQL, but now need to access an Oracle database as well.
I have Oracle SQL Developer installed on my work comp (but there doesn't seem to be a separate Net Manager client per other SO posts), which I can use to access the DB.
Ideally, I would run what I need to in python, but am having difficulties. As it stands, I have created a linked server object to the Oracle DB in a MS SQL Server DB as a work around, but this isn't ideal.
What do I need to do to get PYODBC (or substitute) to connect to Oracle? Thanks very kindly.
I ran into the same issue where I could connect to a database via Oracle SQL Developer but not via pyodbc. Someone else did most of the database setup, so I wasn't sure of the proper connection parameters. I'll run you through how I was able to connect on a Windows computer.
In the Start Menu I typed "odbc" and selected "Microsoft ODBC Administrator". Under the "System DSN" tab I found my DSN name (we'll call it myDSN) and corresponding driver (mine was "Oracle in OraClient11g_home2"). I also have to specify a username and password for my database so my connection line now looks like this:
cnxn = pyodbc.connect(driver='{Oracle in OraClient11g_home2}', dsn='myDSN', uid='HODOR', pwd='hodor')
Maybe at this point it will work for you, but I still wasn't able to connect. This computer is a mess of 32 and 64 bit drivers so I figured I was pointing to the wrong one. So once again into the Start Menu, where under All Programs I found a folder called "Oracle in OraClient11g_home2" and right under it, one called "Oracle in OraClient11g_home32Bit". I changed my connection line in Python to the following:
cnxn = pyodbc.connect(driver='{Oracle in OraClient11g_home32Bit}', dsn='myDSN', uid='HODOR', pwd='hodor')
And it connected.
I'm attempting to connect to a SQL Server instance from a Windows box using pymssql (version 2.0.0b1-dev-20111019 with Python 2.7.1). I've tried the most basic approach from the console:
import pymssql
c = pymssql.connect(host = r'servername\instance',
user = 'username',
password = 'userpassword')
In response to this, I get the very helpful error: InterfaceError: Connection to the database failed for an unknown reason.
I am reasonably confident that the connection information is correct, as it works when I use adodbapi, with the following commands:
import adodbapi
c = adodbapi.connect(r'Provider=sqloledb;Data Source=servername\instance;User ID=username;password=userpassword;'
c.close
I've tried adding the port number to the host parameter, with the same result. Does anyone have a suggestion on how to go about resolving this issue?
Incidentally, I've read the responses at "Unable to connect to SQL Server via pymssql". The OP eventually resolved his issue by correctly configuring FreeTDS, which, from what I can tell, is not used by pymssql on Windows.
Based on #cha0site's recommendation, I have tried using just the hostname, rather than the hostname and instance. This resulted in the same error, but it seemed to take longer to generate the error (though the traceback still indicates the same line). The reason I have been specifying the instance is that I was not able to connect using SSMS unless I specified the instance, so I assumed that it would be necessary for other connections.
I've now also tried pymssql.connect(host='servername', user='username', password='userpassword', database='instance') with the same result (based on #Sid's comment). Based on the pymssql documentation, I believe the database parameter is used to specify the initial database that the user is to be connected to, rather than the instance.
Just to clarify, "instance" is the name provided during installation of SQL Server, not a database within that installation. It occurs to me that it's possible that pymssql does not support this notation, so I will look into re-configuring the SQL Server instance so that it is not required.
I've now re-installed SQL Server as a default instance, rather than a named instance, which allows me to connect without specifying the instance name. adodbapi still works (without /instance), but pymssql still returns the same error. I've also removed and re-installed pymssql from a freshly downloaded archive (still the same version).
Check your freetds.conf file and see if you have set the port 1219., then check again the connexion:
DB = pymssql.connect(host='DB',user='youruser',password='yourpwd',database='yourDBname')
Edit: example of my freetds.conf file Python:
host = 'IP'
port = 1219
To specify host=servername\instance or server=servername\instance, the SQL Server Browser service must be on the SQL Server machine.
I am working on a program to automate parsing data from XML files and storing it into several databases. (Specifically the USGS realtime water quality service, if anyone's interested, at http://waterservices.usgs.gov/rest/WaterML-Interim-REST-Service.html) It's written in Python 2.5.1 using LXML and PYODBC. The databases are in Microsoft Access 2000.
The connection function is as follows:
def get_AccessConnection(db):
connString = 'DRIVER={Microsoft Access Driver (*.mdb)};DBQ=' + db
cnxn = pyodbc.connect(connString, autocommit=False)
cursor = cnxn.cursor()
return cnxn, cursor
where db is the filepath to the database.
The program:
a) opens the connection to the database
b) parses 2 to 8 XML files for that database and builds the values from them into a series of records to insert into the database (using a nested dictionary structure, not a user-defined type)
c) loops through the series of records, cursor.execute()-ing an SQL query for each one
d) commits and closes the database connection
If the cursor.execute() call throws an error, it writes the traceback and the query to the log file and moves on.
When my coworker runs it on his machine, for one particular database, specific records will simply not be there, with no errors recorded. When I run the exact same code on the exact same copy of the database over the exact same network path from my machine, all the data that should be there is there.
My coworker and I are both on Windows XP computers with Microsoft Access 2000 and the same versions of Python, lxml, and pyodbc installed. I have no idea how to check whether we have the same version of the Microsoft ODBC drivers. I haven't been able to find any difference between the records that are there and the records that aren't. I'm in the process of testing whether the same problem happens with the other databases, and whether it happens on a third coworker's computer as well.
What I'd really like to know is ANYTHING anyone can think of that would cause this, because it doesn't make sense to me. To summarize: Python code executing SQL queries will silently fail half of them on one computer and work perfectly on another.
Edit:
No more problem. I just had my coworker run it again, and the database was updated completely with no missing records. Still no idea why it failed in the first place, nor whether or not it will happen again, but "problem solved."
I have no idea how to check whether
we have the same version of the
Microsoft ODBC drivers.
I think you're looking for Control Panel | Administrative Tools | Data Sources (ODBC). Click the "Drivers" tab.
I think either Access 2000 or Office 2000 shipped with a desktop edition of SQL Server called "MSDE". Might be worth installing that for testing. (Or production, for that matter.)