We Keep Coding

Inserting Query Results into Singlestore Table with Python and Sqlalchemy

I have parameterized queries with f strings such that the queries will select some data from a series of tables and joins, and I want to insert the resulting set of data into another pre-created table (tables been designed to house these results).
Python executes the code but the query results never show up in my table.
Assuming target_table is already created in singlestore database:
qry_load = 'insert into target_table select * from some_tables'
conn = engine.connect()
trans = conn.begin()
try:
conn.execute(qry_load)
trans.commit()
except:
trans.rollback()
raise
The code executes and acts as if all is ok, but the data never shows up in the target table.
How do I see what singlestore is passing back to better debug what is happening within the database?

Just replace begin() with cursor() function:
conn = engine.connect()
trans = conn.cursor()
If not resolved
1- Verify structure of source and destination tables if they are same or not.
2- remove try ,except and rollback() block so you can know the actual error.
Ex.
qry_load = 'insert into target_table select * from some_tables'
conn = engine.connect()
trans = conn.cursor()
conn.execute(qry_load)
trans.commit()

Can python cursor.execute accept multiple queries in one go?

Can the cursor.execute call below execute multiple SQL queries in one go?
cursor.execute("use testdb;CREATE USER MyLogin")
I don't have python setup yet but want to know if above form is supported by cursor.execute?
import pyodbc
# Some other example server values are
# server = 'localhost\sqlexpress' # for a named instance
# server = 'myserver,port' # to specify an alternate port
server = 'tcp:myserver.database.windows.net'
database = 'mydb'
username = 'myusername'
password = 'mypassword'
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
#Sample select query
cursor.execute("SELECT ##version;")
row = cursor.fetchone()
while row:
print(row[0])
row = cursor.fetchone()

Multiple SQL statements in a single string is often referred to as an "anonymous code block".
There is nothing in pyodbc (or pypyodbc) to prevent you from passing a string containing an anonymous code block to the Cursor.execute() method. They simply pass the string to the ODBC Driver Manager (DM) which in turn passes it to the ODBC Driver.
However, not all ODBC drivers accept anonymous code blocks by default. Some databases default to allowing only a single SQL statement per .execute() to protect us from SQL injection issues.
For example, MySQL/Connector ODBC defaults MULTI_STATEMENTS to 0 (off) so if you want to run an anonymous code block you will have to include MULTI_STATEMENTS=1 in your connection string.
Note also that changing the current database by including a USE … statement in an anonymous code block can sometimes cause problems because the database context changes in the middle of a transaction. It is often better to execute a USE … statement by itself and then continue executing other SQL statements.

Yes, it is possible.
operation = 'SELECT 1; INSERT INTO t1 VALUES (); SELECT 2'
for result in cursor.execute(operation, multi=True):
But it is not a comprehensive solution. For example, in queries with two selections, you have problems.
Consider that two types of answers must be fetch all in the cursor!
So the best solution is to break the query to sub queries and do your work step by step.
for example :
s = "USE some_db; SELECT * FROM some_table;"
s = filter(None, s.split(';'))
for i in s:
cur.execute(i.strip() + ';')

in the pyodbc documentation should give you the example your looking for. more over in the GitHub wiki: https://github.com/mkleehammer/pyodbc/wiki/Objects#cursors
you can see an example here:
cnxn = pyodbc.connect(...)
cursor = cnxn.cursor()
cursor.execute("""
select user_id, last_logon
from users
where last_logon > ?
and user_type <> 'admin'
""", twoweeks)
rows = cursor.fetchall()
for row in rows:
print('user %s logged on at %s' % (row.user_id, row.last_logon))
from this example and exploring the code, I would say your next step is testing a multi cursor.execute("<your_sql_Querie>").
if this test works, maybe try and create a CLASS then create instances of that class for each query you want to run.
This would be the basic evolution of a developers effort of reproducing documentation...hope this helps you :)

Yes, you can results for multiple queries by using the nextset() method...
query = "select * from Table1; select * from Table2"
cursor = connection.cursor()
cursor.execute(query)
table1 = cursor.fetchall()
cursor.nextset()
table2 = cursor.fetchall()
The code explains it... cursors return result "sets", which you can move between using the nextset() method.

When does sqlite3 roll back transactions in python, and when doesn't it?

I am creating a little workshop to teach how to use python and SQL and came across this oddity. I wanted to show how to use the with statement to create a transaction with sqlite:
import sqlite3
filename = 'data/transaction.db'
print("_________________________")
print("Create Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
sqls = [
'DROP TABLE IF EXISTS test',
'CREATE TABLE test (i integer)',
'INSERT INTO "test" VALUES(99)',
'SELECT * FROM test']
for sql in sqls:
cursor.execute(sql)
print(cursor.fetchall())
print("_________________________")
print("Create Error with 'with'")
try:
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
sqls = [
'update test set i = 1',
'SELECT * FROM test',
'fnord', # <-- trigger error
'update test set i = 0',]
for sql in sqls:
cursor.execute(sql)
print(cursor.fetchall())
except sqlite3.OperationalError as err:
print(err)
# near "fnord": syntax error
print("_________________________")
print("Show Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
cursor.execute('SELECT * FROM test')
for row in cursor:
print(row)
# (99,)
This works exactly as expected. However to prove that without the with block the executions would be done halfway I tried the following:
print("_________________________")
print("Create Error without 'with'")
conn = sqlite3.connect(filename)
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
cursor.execute( 'UPDATE test SET i = 1 WHERE i = 99')
print(cursor.fetchall())
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
cursor.execute( 'update test set i = 0')
print(cursor.fetchall())
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
conn.close()
print("_________________________")
print("Show Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
cursor.execute('SELECT * FROM test')
for row in cursor:
print(row)
# (99,)`
The whole output is:
_________________________
Create Table
[]
[]
[]
[(99,)]
_________________________
Create Error with 'with'
[]
[(1,)]
near "fnord": syntax error
_________________________
Show Table
(99,)
_________________________
Create Error without 'with'
[(99,)]
[]
[(1,)]
[]
[(0,)]
_________________________
Show Table
(99,) # Why is this not (0,)???
I am very confused as to why the last Block shows a 99 again. Eventually the plan is to add a try,except block with an exception, such that the SQL code mimics the first block - however I am confused without this already :).
Thanks for clarifying

From the python sqlite3 API doc:
The underlying sqlite3 library operates in autocommit mode by default,
but the Python sqlite3 module by default does not.
autocommit mode means that statements that modify the database take
effect immediately. A BEGIN or SAVEPOINT statement disables autocommit
mode, and a COMMIT, a ROLLBACK, or a RELEASE that ends the outermost
transaction, turns autocommit mode back on.
The Python sqlite3 module by default issues a BEGIN statement
implicitly before a Data Modification Language (DML) statement (i.e.
INSERT/UPDATE/DELETE/REPLACE).
Python will rollback transactions if the connection is closed without a commit (or an explicit ROLLBACK is issued). No transactions are committed in this program.
FYI a new connection is created in the "Create error without 'with'" block, but no new cursor is instantiated.

The Python with statement works with context managers. Whereas some context managers will release resources and possibly close an object, it seems that at least with the sqlite3.connection object, it merely commits or rolls back transactions but does not close the connection. This can be confirmed for the DB-API 2.0 interface:
Connection objects can be used as context managers that automatically commit or rollback transactions. In the event of an exception, the transaction is rolled back; otherwise, the transaction is committed:
...
# Successful, con.commit() is called automatically afterwards
...
# con.rollback() is called after the with block finishes with an exception, the
# exception is still raised and must be caught
...
# Connection object used as context manager only commits or rollbacks transactions,
# so the connection object should be closed manually
For the non-with statement block, you are never committing the transactions. When the connection is closed, all changes are automatically rolled back.
You need to call
conn.commit();
See Why the need to commit explicitly when doing an UPDATE? for more details.
As a side note, the section of your code titled "Create Error without 'with'" does not actually cause an error/exception.

Empty table in MySQL even though Python can insert data into table

I'm new to mySQL and Python.
I have code to insert data from Python into mySQL,
conn = MySQLdb.connect(host="localhost", user="root", passwd="kokoblack", db="mydb")
for i in range(0,len(allnames)):
try:
query = "INSERT INTO resumes (applicant, jobtitle, lastworkdate, lastupdate, url) values ("
query = query + "'"+allnames[i]+"'," +"'"+alltitles[i]+"',"+ "'"+alldates[i]+"'," + "'"+allupdates[i]+"'," + "'"+alllinks[i]+"')"
x = conn.cursor()
x.execute(query)
row = x.fetchall()
except:
print "error"
It seems to be working fine, because "error" never appears. Instead, many rows of "1L" appear in my Python shell. However, when I go to MySQL, the "resumes" table in "mydb" remains completely empty.
I have no idea what could be wrong, could it be that I am not connected to MySQL's server properly when I'm viewing the table in MySQL? Help please.
(I only use import MySQLdb, is that enough?)

use commit to commit the changes that you have done
MySQLdb has autocommit off by default, which may be confusing at first
You could do commit like this
conn.commit()
or
conn.autocommit(True) Right after the connection is created with the DB

Error Updating Mysql database via python

I am trying to add all words of a text file into a column such that one row has one word. my code is as :
import MySQLdb
conn = MySQLdb.connect (host = "localhost",user = "root", db = "pcorpora")
c = conn.cursor()
file = open('C:\Users\Admin\Desktop\english.txt', 'r')
words = list(file.read())
i=0
for value in words:
c.execute("""INSERT INTO tenglish (`english words`) VALUES (%s)""" % (words[i]) i=i+1)`
The code run without error but table is still empty.

You should use commit
c.execute("""INSERT INTO tenglish (`english words`) VALUES (%s)""" % (value))
con.commit()
This method sends a COMMIT statement to the MySQL server, committing
the current transaction. Since by default Connector/Python does not
autocommit, it is important to call this method after every
transaction that modifies data for tables that use transactional
storage engines.