This isn't a question, so much as a pre-emptive answer. (I have gotten lots of help from this website & wanted to give back.)
I was struggling with a large bit of SQL query that was failing when I tried to run it via python using pymssql, but would run fine when directly through MS SQL. (E.g., in my case, I was using MS SQL Server Management Studio to run it outside of python.)
Then I finally discovered the problem: pymssql cannot handle temporary tables. At least not my version, which is still 1.0.1.
As proof, here is a snippet of my code, slightly altered to protect any IP issues:
conn = pymssql.connect(host=sqlServer, user=sqlID, password=sqlPwd, \
database=sqlDB)
cur = conn.cursor()
cur.execute(testQuery)
The above code FAILS (returns no data, to be specific, and spits the error "pymssql.OperationalError: No data available." if you call cur.fetchone() ) if I call it with testQuery defined as below:
testQuery = """
CREATE TABLE #TEST (
[sample_id] varchar (256)
,[blah] varchar (256) )
INSERT INTO #TEST
SELECT DISTINCT
[sample_id]
,[blah]
FROM [myTableOI]
WHERE [Shipment Type] in ('test')
SELECT * FROM #TEST
"""
However, it works fine if testQuery is defined as below.
testQuery = """
SELECT DISTINCT
[sample_id]
,[blah]
FROM [myTableOI]
WHERE [Shipment Type] in ('test')
"""
I did a Google search as well as a search within Stack Overflow, and couldn't find any information regarding the particular issue. I also looked under the pymssql documentation and FAQ, found at http://code.google.com/p/pymssql/wiki/FAQ, and did not see anything mentioning that temporary tables are not allowed. So I thought I'd add this "question".
Update: July 2016
The previously-accepted answer is no longer valid. The second "will NOT work" example does indeed work with pymssql 2.1.1 under Python 2.7.11 (once conn.autocommit(1) is replaced with conn.autocommit(True) to avoid "TypeError: Cannot convert int to bool").
For those who run across this question and might have similar problems, I thought I'd pass on what I'd learned since the original post. It turns out that you CAN use temporary tables in pymssql, but you have to be very careful in how you handle commits.
I'll first explain by example. The following code WILL work:
testQuery = """
CREATE TABLE #TEST (
[name] varchar(256)
,[age] int )
INSERT INTO #TEST
values ('Mike', 12)
,('someone else', 904)
"""
conn = pymssql.connect(host=sqlServer, user=sqlID, password=sqlPwd, \
database=sqlDB) ## obviously setting up proper variables here...
conn.autocommit(1)
cur = conn.cursor()
cur.execute(testQuery)
cur.execute("SELECT * FROM #TEST")
tmp = cur.fetchone()
tmp
This will then return the first item (a subsequent fetch will return the other):
('Mike', 12)
But the following will NOT work
testQuery = """
CREATE TABLE #TEST (
[name] varchar(256)
,[age] int )
INSERT INTO #TEST
values ('Mike', 12)
,('someone else', 904)
SELECT * FROM #TEST
"""
conn = pymssql.connect(host=sqlServer, user=sqlID, password=sqlPwd, \
database=sqlDB) ## obviously setting up proper variables here...
conn.autocommit(1)
cur = conn.cursor()
cur.execute(testQuery)
tmp = cur.fetchone()
tmp
This will fail saying "pymssql.OperationalError: No data available." The reason, as best I can tell, is that whether you have autocommit on or not, and whether you specifically make a commit yourself or not, all tables must explicitly be created AND COMMITTED before trying to read from them.
In the first case, you'll notice that there are two "cur.execute(...)" calls. The first one creates the temporary table. Upon finishing the "cur.execute()", since autocommit is turned on, the SQL script is committed, the temporary table is made. Then another cur.execute() is called to read from that table. In the second case, I attempt to create & read from the table "simultaneously" (at least in the mind of pymssql... it works fine in MS SQL Server Management Studio). Since the table has not previously been made & committed, I cannot query into it.
Wow... that was a hassle to discover, and it will be a hassle to adjust my code (developed on MS SQL Server Management Studio at first) so that it will work within a script. Oh well...
Related
This question already has answers here:
How to use variables in SQL statement in Python?
(5 answers)
Closed 2 months ago.
def update_inv_quant():
new_quant = int(input("Enter the updated quantity in stock: "))
Hello! I'm wondering how to insert a user variable into an sql statement so that a record is updated to said variable. Also, it'd be really helpful if you could also help me figure out how to print records of the database into the actual python console. Thank you!
I tried doing soemthing like ("INSERT INTO Inv(ItemName) Value {user_iname)") but i'm not surprised it didnt work
It would have been more helpful if you specified an actual database.
First method (Bad)
The usual way (which is highly discouraged as Graybeard said in the comments) is using python's f-string. You can google what it is and how to use it more in-depth.
but basically, say you have two variables user_id = 1 and user_name = 'fish', f-string turns something like f"INSERT INTO mytable(id, name) values({user_id},'{user_name}')" into the string INSERT INTO mytable(id,name) values(1,'fish').
As we mentioned before, this causes something called SQL injection. There are many good youtube videos that demonstrate what that is and why it's dangerous.
Second method
The second method is dependent on what database you are using. For example, in Psycopg2 (Driver for PostgreSQL database), the cursor.execute method uses the following syntax to pass variables cur.execute('SELECT id FROM users WHERE cookie_id = %s',(cookieid,)), notice that the variables are passed in a tuple as a second argument.
All databases use similar methods, with minor differences. For example, I believe SQLite3 uses ? instead of psycopg2's %s. That's why I said that specifying the actual database would have been more helpful.
Fetching records
I am most familiar with PostgreSQL and psycopg2, so you will have to read the docs of your database of choice.
To fetch records, you send the query with cursor.execute() like we said before, and then call cursor.fetchone() which returns a single row, or cursor.fetchall() which returns all rows in an iterable that you can directly print.
Execute didn't update the database?
Statements executing from drivers are transactional, which is a whole topic by itself that I am sure will find people on the internet who can explain it better than I can. To keep things short, for the statement to physically change the database, you call connection.commit() after cursor.execute()
So finally to answer both of your questions, read the documentation of the database's driver and look for the execute method.
This is what I do (which is for sqlite3 and would be similar for other SQL type databases):
Assuming that you have connected to the database and the table exists (otherwise you need to create the table). For the purpose of the example, i have used a table called trades.
new_quant = 1000
# insert one record (row)
command = f"""INSERT INTO trades VALUES (
'some_ticker', {new_quant}, other_values, ...
) """
cur.execute(command)
con.commit()
print('trade inserted !!')
You can then wrap the above into your function accordingly.
I'm trying to pass variables into multiple Teradata SQL files. I have several scripts doing this perfectly fine when I don't have a "create table" in the file. When I run the files with the variables hard coded, they run just fine but when I use an open/replace function I get the error "only an et or null statement is legal after a ddl statement".
This tells me that the "Open().Replace()" function is what's causing this. It seems like this is causing python to run the SQL scripts in parallel instead of serial but I don't know how to account for this. I tried to use the open/replace outside of my execute cursor but then it errors due to the variables not being changed. I have some code samples below that are simplified for this question. This is my first time posting a question so if there's a better way for me to do this please feel free to comment so I can make the edits.
test.sql without variable
create multiset volatile table sdates as (Sel caldt, datesk
from dimdates
where day = 'Mon'
and mth = 'Feb');
ins into vartest (Sel caldt, site, sum(sales)
from fctsales f
join sdates s
on f.datesk = s.datesk);
python code that works
query = cur.execute(file = 'test.sql')
test.sql with variables
create multiset volatile table sdates as (Sel caldt, datesk
from dimdates
where day = 'bbbb'
and mth = 'oooo');
ins into vartest (Sel caldt, site, sum(sales)
from fctsales f
join sdates s
on f.datesk = s.datesk);
python code that fails
bbbb = 'Mon'
oooo = 'Feb'
query = cur.execute(open(file = 'test.sql').read().replace('bbbb',bbbb).replace('oooo',oooo))
DatabaseError: (3932, '[25000] [Teradata][ODBC Teradata Driver][Teradata Database](-3932)Only an ET or null statement is legal after a DDL Statement.')
Edit 1: Added exact error code
1 step: Create a temporary table with pyodbc into sql server for objects
2 step: Select objects from temporary table and load it into pandas dataframe
3 step: print dataframe
for creating a temporary table i work with pyodbc cursor as it trohws errors with pandas.read_sql command. wheras it trohws an error if i try to convert the cursor into a pandas dataframe. even with the special line for handling tuples into dataframes.
my program to connect, create, read and print which works as long as the query stays simple as it is now. (my actual approach has a few hundred lines of sql query statement)
import codecs
import os
import io
import pandas as pd
import pyodbc as po
server = 'sql_server'
database = 'sql_database'
connection = po.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';Trusted_Connection=yes;')
cursor = connection.cursor()
query1 = """
CREATE TABLE #ttobject (object_nr varchar(6), change_date datetime)
INSERT INTO #ttobject (object_nr)
VALUES
('112211'),
('113311'),
('114411');
"""
query2 = """
SELECT *
FROM #ttobject
Drop table if exists #ttobject
"""
cursor.execute(query1)
df = pd.read_sql_query(query2, connection)
print(df)
Because of the lenght of the actually query i save you the trouble but instead post here the error code:
('HY000', '[HY000] [Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt (0) (SQLExecDirectW)')
This error gets thrown at query2 which is a multiple select statement with some joins and pivote functions
When I'm trying to put everything into one cursor i got issues with converting it from cursor to DataFrame (tried several methodes, maybe someone knows one which isn't on SO already or has a special title so i couldn't find it)
same problem if I'm trying to only use pd.read_sql then the creation of the temporary table is not working
I don't know where to go on from here.
Please let me know if i can assist you with further details which i may overwatched in accordance to my lostlyness :S
23.5.19 Further investigating:
According to Gord i tried to add autocommit to true which will work
for simple sql statements but not for my really long and
timeconsuming one.
Secondly i tried to add
"cursor.execute('SET NOCOUNT ON; EXEC schema.proc #muted = 1')
At the moment i guess that the first query takes longer so python already starting with the second and therefore the connection is
blocked. Or that the first query is returing some feedback so python
thinks it is finished before it actually is.
Added a time.sleep(100) after ececution of first query but still getting the hstmt is busy error. Wondering why this is becaus it should have had enough time to process the first
Funfact: The query is running smoothly as long as I'm not trying to output any result from it
I am trying to select data from an ORACLE 12c database using cx_Oracle, but I am getting the exception: "cx_Oracle.OperationalError: ORA-03113: end-of-file on communication channel".
My query behaves fine using Pycharm (jdbc:oracle:thin driver). Using cx_Oracle in python 3.6, however, the query fails unless I reduce the number of IDs in the IN clause from 500 to about 250. The Cursor.fetchall() function is what throws the exception. I do not have privileged access to the database in order to check things like locks or load, but could these be the cause of the issue? According to our DBA, there is nothing wrong on the Oracle db server, and since the query works fine otherwise, I am inclined to believe it. I have messed with the client-side sqlnet.ora as well, which has allowed exceptions to eventually be thrown instead of hanging forever, but I still cannot fetch the data.
def select(self, query, *args):
cur = self.dbh.cursor()
cur.prepare(query)
try:
cur.execute(None, args)
return cur.fetchall()
# my attempt to handle the issue
except (cx_Oracle.OperationalError, cx_Oracle.DatabaseError) as e:
# cx_Oracle.OperationalError: ORA-03113: end-of-file on communication channel
self.logger.error('Oracle Error: {}'.format(traceback.format_exc()))
raise e
The code calls select like this. For brevity, I've omitted the full string IDs
ids = ['1', '2', '3', ...]
query = """\
select * from my_table where id in(:0,:1,:2,:3,:4, ...)
"""
self.select(query, *ids)
The query fails without the placeholders (with the IDs placed directly in the query) as well.
I expect to be able to run any select query using an IN clause with up to 1000 IDs without receiving the ORA-03113 Exception.
Edit:
I installed oracle-instantclient18.5-basic-18.5.0.0.0-3.x86_64.rpm* on ubuntu 18.04.2, have cx_Oracle version 7.1.2, and I am connecting to Oracle 12.1.0.2.0.
The query is on the underlying tables of BMC Software's ARS. I will start working to try to replicate the problem with a local table structure, but it is a mess and will take some time. If I am able to create a local copy of the tables, I'm not sure I'd be able to replicate the issue, as identical queries with different IDs work fine. That makes it seem data driven, however, after I reduced the query to 250 IDs, I swapped the 250 from the first half to the second half, and got the same success result, so it doesn't seem to be just one bad row.
Is there more helpful logging I can enable on the client side to get more information?
Edit2: I should also add that the issue does not just occur with one query. I've seen the same issue with select queries to completely different tables.
Edit3: I just found out that by commenting out some of the columns that I'm selecting also can make the query work. columns like this:
to_char(to_date('1970-01-01','YYYY-MM-DD') + numtodsinterval(EventStart,'SECOND'),'YYYY-MM-DD HH24:MI:SS')
This may indicate that some kind of timeout is being reached which may or may not be configured in my sqlnet.ora:
DISABLE_OOB=on
SQLNET.RECV_TIMEOUT=60
SQLNET.SEND_TIMEOUT=60
TCP.CONNECT_TIMEOUT=300
SQLNET.OUTBOUND_CONNECT_TIMEOUT=300
ENABLE=BROKEN
TRACE_LEVEL_CLIENT=ADMIN
TRACE_FILE_CLIENT=sqlnet
Edit 4: I've tried some more things.
I installed the same version of instant client, except on a windows 7 machine, and ran the same query against the same db instance. The query succeeded.
I also narrowed down that for this particular query, it will accept 499 IDs, but fails with 500. it doesn't matter which ID I comment out from the query.
I also tried tricking the query into thinking there were fewer IDs by using a sub-select instead:
IN(
select regexp_substr(:0,'[^,]+', 1, level) from dual connect by regexp_substr(:0, '[^,]+', 1, level) is not null
)
I got the error "cx_Oracle.DatabaseError: ORA-01460: unimplemented or unreasonable conversion requested", after which I realized made sense because Oracle will only allow a string to be up to 4000 bytes long.
I think I finally found a way to get everything functioning. I finally came across this link:
https://ardentperf.com/2010/09/08/mysterious-oracle-net-errors/
It turns out that this solved my problem. I am still having trouble getting cx_Oracle to honor a connection string of the same format as the tnsnames.ora file, but I changed my code to refer to the tnsnames.ora for now as follows:
connection_info = {
'user': self.config.get(self.db, 'user'),
'pass': self.config.get(self.db, 'password')
}
connection_string = '{user}/{pass}#TEST'\
.format(**connection_info)
connection = cx_Oracle.connect(connection_string)
where my tnsnames.ora contains the following:
TEST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(Host = myhost.com)(Port = 1521))
)
(SDU=1024)
(CONNECT_DATA =
(SID=mysid)
)
)
The key here is the SDU=1024, which inexplicably fixes this issue.
https://docs.oracle.com/cd/B28359_01/network.111/b28317/sqlnet.htm#NETRF184
Documentation from the above link indicates that the default for SDU is 8192 bytes (8 KB) and my understanding is that there is supposed to be auto-negotiation of this value. This does not appear to be the case, and I don't know what past defaults have been.
I had no problem with SELECTing data in python from postgres database using cursor/execute. Just changed the sql to INSERT a row but nothing is inserted to DB. Can anyone let me know what should be modified? A little confused because everything is the same except for the sql statement.
<!-- language: python -->
#app.route("/addcontact")
def addcontact():
# this connection/cursor setting showed no problem so far
conn = pg.connect(conn_str)
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
sql = f"INSERT INTO jna (sid, phone, email) VALUES ('123','123','123')"
cur.execute(sql)
return redirect("/contacts")
first look at your table setup and make sure your variables are named right in the right order, format and all that, if your not logging into the specific database on the sql server it won't know where the table is, you might need to send something like 'USE databasename' before you do your insert statement so your computer is in the right place in the server.
I might not be up to date with the language but is that 'f' supposed to be right before the quotes? if thats in ur code that'd probably throw an error unless it has a use im not aware of or its not relevant to the problem.
You have to commit your transaction by adding the line below after execute(sql)
conn.commit()
Ref: Using INSERT with a PostgreSQL Database using Python