We Keep Coding

Arbitrary selects failing using cx_Oracle with instant client

I am trying to select data from an ORACLE 12c database using cx_Oracle, but I am getting the exception: "cx_Oracle.OperationalError: ORA-03113: end-of-file on communication channel".
My query behaves fine using Pycharm (jdbc:oracle:thin driver). Using cx_Oracle in python 3.6, however, the query fails unless I reduce the number of IDs in the IN clause from 500 to about 250. The Cursor.fetchall() function is what throws the exception. I do not have privileged access to the database in order to check things like locks or load, but could these be the cause of the issue? According to our DBA, there is nothing wrong on the Oracle db server, and since the query works fine otherwise, I am inclined to believe it. I have messed with the client-side sqlnet.ora as well, which has allowed exceptions to eventually be thrown instead of hanging forever, but I still cannot fetch the data.
def select(self, query, *args):
cur = self.dbh.cursor()
cur.prepare(query)
try:
cur.execute(None, args)
return cur.fetchall()
# my attempt to handle the issue
except (cx_Oracle.OperationalError, cx_Oracle.DatabaseError) as e:
# cx_Oracle.OperationalError: ORA-03113: end-of-file on communication channel
self.logger.error('Oracle Error: {}'.format(traceback.format_exc()))
raise e
The code calls select like this. For brevity, I've omitted the full string IDs
ids = ['1', '2', '3', ...]
query = """\
select * from my_table where id in(:0,:1,:2,:3,:4, ...)
"""
self.select(query, *ids)
The query fails without the placeholders (with the IDs placed directly in the query) as well.
I expect to be able to run any select query using an IN clause with up to 1000 IDs without receiving the ORA-03113 Exception.
Edit:
I installed oracle-instantclient18.5-basic-18.5.0.0.0-3.x86_64.rpm* on ubuntu 18.04.2, have cx_Oracle version 7.1.2, and I am connecting to Oracle 12.1.0.2.0.
The query is on the underlying tables of BMC Software's ARS. I will start working to try to replicate the problem with a local table structure, but it is a mess and will take some time. If I am able to create a local copy of the tables, I'm not sure I'd be able to replicate the issue, as identical queries with different IDs work fine. That makes it seem data driven, however, after I reduced the query to 250 IDs, I swapped the 250 from the first half to the second half, and got the same success result, so it doesn't seem to be just one bad row.
Is there more helpful logging I can enable on the client side to get more information?
Edit2: I should also add that the issue does not just occur with one query. I've seen the same issue with select queries to completely different tables.
Edit3: I just found out that by commenting out some of the columns that I'm selecting also can make the query work. columns like this:
to_char(to_date('1970-01-01','YYYY-MM-DD') + numtodsinterval(EventStart,'SECOND'),'YYYY-MM-DD HH24:MI:SS')
This may indicate that some kind of timeout is being reached which may or may not be configured in my sqlnet.ora:
DISABLE_OOB=on
SQLNET.RECV_TIMEOUT=60
SQLNET.SEND_TIMEOUT=60
TCP.CONNECT_TIMEOUT=300
SQLNET.OUTBOUND_CONNECT_TIMEOUT=300
ENABLE=BROKEN
TRACE_LEVEL_CLIENT=ADMIN
TRACE_FILE_CLIENT=sqlnet
Edit 4: I've tried some more things.
I installed the same version of instant client, except on a windows 7 machine, and ran the same query against the same db instance. The query succeeded.
I also narrowed down that for this particular query, it will accept 499 IDs, but fails with 500. it doesn't matter which ID I comment out from the query.
I also tried tricking the query into thinking there were fewer IDs by using a sub-select instead:
IN(
select regexp_substr(:0,'[^,]+', 1, level) from dual connect by regexp_substr(:0, '[^,]+', 1, level) is not null
)
I got the error "cx_Oracle.DatabaseError: ORA-01460: unimplemented or unreasonable conversion requested", after which I realized made sense because Oracle will only allow a string to be up to 4000 bytes long.

I think I finally found a way to get everything functioning. I finally came across this link:
https://ardentperf.com/2010/09/08/mysterious-oracle-net-errors/
It turns out that this solved my problem. I am still having trouble getting cx_Oracle to honor a connection string of the same format as the tnsnames.ora file, but I changed my code to refer to the tnsnames.ora for now as follows:
connection_info = {
'user': self.config.get(self.db, 'user'),
'pass': self.config.get(self.db, 'password')
}
connection_string = '{user}/{pass}#TEST'\
.format(**connection_info)
connection = cx_Oracle.connect(connection_string)
where my tnsnames.ora contains the following:
TEST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(Host = myhost.com)(Port = 1521))
)
(SDU=1024)
(CONNECT_DATA =
(SID=mysid)
)
)
The key here is the SDU=1024, which inexplicably fixes this issue.
https://docs.oracle.com/cd/B28359_01/network.111/b28317/sqlnet.htm#NETRF184
Documentation from the above link indicates that the default for SDU is 8192 bytes (8 KB) and my understanding is that there is supposed to be auto-negotiation of this value. This does not appear to be the case, and I don't know what past defaults have been.

sql INSERT in python (postgres, cursor, execute)

I had no problem with SELECTing data in python from postgres database using cursor/execute. Just changed the sql to INSERT a row but nothing is inserted to DB. Can anyone let me know what should be modified? A little confused because everything is the same except for the sql statement.
<!-- language: python -->
#app.route("/addcontact")
def addcontact():
# this connection/cursor setting showed no problem so far
conn = pg.connect(conn_str)
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
sql = f"INSERT INTO jna (sid, phone, email) VALUES ('123','123','123')"
cur.execute(sql)
return redirect("/contacts")

first look at your table setup and make sure your variables are named right in the right order, format and all that, if your not logging into the specific database on the sql server it won't know where the table is, you might need to send something like 'USE databasename' before you do your insert statement so your computer is in the right place in the server.
I might not be up to date with the language but is that 'f' supposed to be right before the quotes? if thats in ur code that'd probably throw an error unless it has a use im not aware of or its not relevant to the problem.

You have to commit your transaction by adding the line below after execute(sql)
conn.commit()
Ref: Using INSERT with a PostgreSQL Database using Python

python to MySQL within pythonAnywhere

I'm trying to enter data from python 3.4 into a MySQL database, with both of these entities being within pythonAnywhere. In other words, I'm writing a python 3.4 program in pythonAnywhere and need to connect to a MySQL db also within pythonAnywhere. I've checked other answers here and haven't quite figured it out. I've tried the ( -u -h -p) syntax mentioned by a few posts, but I'm not sure if that is just for gaining access from outside of pythonAnywhere.
Any help would be appreciated.
++++++++++++++++++++
Actually I figured it out (with kudos to Tony Darnell at the Scripting MySQL website, from whom I plagiarized most of this:
import MySQLdb
db=MySQLdb.connect(
host='Your_User_Name.mysql.pythonanywhere-services.com',
user='Your_User_Name',
passwd='Your_pythonAnywhere_password',
db='Your_User_Name$Your_Data_Base')
everything above starting with 'Your' refers to you personal account info with pythonanywhere
everything else gets listed exactly as shown. Watch that $ that follows your user name as part of the database name (db = etc.)
cursor = db.cursor ()
execute the SQL query using execute() method.
cursor.execute ("Enter any MySQL query here. use the quotes. no semi-colon")
fetch a single row from query using fetchone() method.
row = cursor.fetchone ()
print(row)
fetch all the rest of the query using fetchall() method
data = cursor.fetchall()
print(data)
close the cursor object
cursor.close ()
close the connection
db.close ()

python to call sql server procedure (2.7)

Never use python before, here want to using procedure already written in SQL server Script, then write a python application connect to database (already connected) rather than JAVA, because this is much easier.
Let user to give an input as the #Departments, then calculate the average of that Department's average salary.
My sql server procedure code:
CREATE PROC aaatest # Departments varchar(40)
AS
BEGIN
SELECT AVG(P.Salary)
FROM Company P
WHERE P.Department = #Departments
END
please write the a python application to get the input then pass to the #Departments. (conn = pymssql ... is already done!)

I guess this should give you some hint of how to script your requirement:
import pyodbc
cnxn=pyodbc.connect(r'Driver={SQL Server};Server=<servername>;Trusted_Connection;user=<username>;password=<password>',autocommit = True)
cursor=cnxn.cursor()
department=raw_input('Enter department:')
query='EXEC [master].[dbo].[aaatest] '+department+''
cursor.execute(query)
cnxn.close()

pymssql ( python module ) unable to use temporary tables

This isn't a question, so much as a pre-emptive answer. (I have gotten lots of help from this website & wanted to give back.)
I was struggling with a large bit of SQL query that was failing when I tried to run it via python using pymssql, but would run fine when directly through MS SQL. (E.g., in my case, I was using MS SQL Server Management Studio to run it outside of python.)
Then I finally discovered the problem: pymssql cannot handle temporary tables. At least not my version, which is still 1.0.1.
As proof, here is a snippet of my code, slightly altered to protect any IP issues:
conn = pymssql.connect(host=sqlServer, user=sqlID, password=sqlPwd, \
database=sqlDB)
cur = conn.cursor()
cur.execute(testQuery)
The above code FAILS (returns no data, to be specific, and spits the error "pymssql.OperationalError: No data available." if you call cur.fetchone() ) if I call it with testQuery defined as below:
testQuery = """
CREATE TABLE #TEST (
[sample_id] varchar (256)
,[blah] varchar (256) )
INSERT INTO #TEST
SELECT DISTINCT
[sample_id]
,[blah]
FROM [myTableOI]
WHERE [Shipment Type] in ('test')
SELECT * FROM #TEST
"""
However, it works fine if testQuery is defined as below.
testQuery = """
SELECT DISTINCT
[sample_id]
,[blah]
FROM [myTableOI]
WHERE [Shipment Type] in ('test')
"""
I did a Google search as well as a search within Stack Overflow, and couldn't find any information regarding the particular issue. I also looked under the pymssql documentation and FAQ, found at http://code.google.com/p/pymssql/wiki/FAQ, and did not see anything mentioning that temporary tables are not allowed. So I thought I'd add this "question".

Update: July 2016
The previously-accepted answer is no longer valid. The second "will NOT work" example does indeed work with pymssql 2.1.1 under Python 2.7.11 (once conn.autocommit(1) is replaced with conn.autocommit(True) to avoid "TypeError: Cannot convert int to bool").

For those who run across this question and might have similar problems, I thought I'd pass on what I'd learned since the original post. It turns out that you CAN use temporary tables in pymssql, but you have to be very careful in how you handle commits.
I'll first explain by example. The following code WILL work:
testQuery = """
CREATE TABLE #TEST (
[name] varchar(256)
,[age] int )
INSERT INTO #TEST
values ('Mike', 12)
,('someone else', 904)
"""
conn = pymssql.connect(host=sqlServer, user=sqlID, password=sqlPwd, \
database=sqlDB) ## obviously setting up proper variables here...
conn.autocommit(1)
cur = conn.cursor()
cur.execute(testQuery)
cur.execute("SELECT * FROM #TEST")
tmp = cur.fetchone()
tmp
This will then return the first item (a subsequent fetch will return the other):
('Mike', 12)
But the following will NOT work
testQuery = """
CREATE TABLE #TEST (
[name] varchar(256)
,[age] int )
INSERT INTO #TEST
values ('Mike', 12)
,('someone else', 904)
SELECT * FROM #TEST
"""
conn = pymssql.connect(host=sqlServer, user=sqlID, password=sqlPwd, \
database=sqlDB) ## obviously setting up proper variables here...
conn.autocommit(1)
cur = conn.cursor()
cur.execute(testQuery)
tmp = cur.fetchone()
tmp
This will fail saying "pymssql.OperationalError: No data available." The reason, as best I can tell, is that whether you have autocommit on or not, and whether you specifically make a commit yourself or not, all tables must explicitly be created AND COMMITTED before trying to read from them.
In the first case, you'll notice that there are two "cur.execute(...)" calls. The first one creates the temporary table. Upon finishing the "cur.execute()", since autocommit is turned on, the SQL script is committed, the temporary table is made. Then another cur.execute() is called to read from that table. In the second case, I attempt to create & read from the table "simultaneously" (at least in the mind of pymssql... it works fine in MS SQL Server Management Studio). Since the table has not previously been made & committed, I cannot query into it.
Wow... that was a hassle to discover, and it will be a hassle to adjust my code (developed on MS SQL Server Management Studio at first) so that it will work within a script. Oh well...