We Keep Coding

executing a sql query using python

I'm trying to create a small python app to extract data from specific table of database.
The extracted rows have to be between CREATION_DATETIME specified by user.
Heres the code:
startdate = input("Prosze podac poczatek przedzialu czasowego (format RRRR-MM-DD GG:MM:SS): ")
enddate = input("Prosze podac koniec przedzialu czasowego (format RRRR-MM-DD GG:MM:SS): ")
query = "SELECT * FROM BRDB.RFX_IKW_MODIFY_EXEC_ORDER_CANCEL_LOG WHERE CREATION_DATETIME between '%s' and '%s' ORDER BY CREATION_DATETIME DESC;"
tuple1 = (startdate, enddate)
cursor.execute(*query, (tuple1,))
records = cursor.fetchall()
print("Total number of rows in table: ", cursor.rowcount)
print(records)
I'm not much of developer and I'm stuck at error "TypeError: CMySQLCursorPrepared.execute() takes from 2 to 4 positional arguments but 104 were given" in various counts, depends on how I try to modify the code.
Could you guys help me out in specyfing that query correctly?
Thank you in advance.
Tried various tutorial about parametrized query but with no luck.

You're starring the query, making it an iterable of the characters making up the string, which probably isn't what you meant (i.e., you should emove the * operator). In addition, tuple1 is already a tuple, you shouldn't enclose it inside another tuple:
cursor.execute(query, tuple1)
# Remove the *-^
# Use tuple1 directly-^

here is the full code
import mysql.connector
from mysql.connector import Error
try:
print("Laczenie z baza danych....")
connection = mysql.connector.connect(host='',
port='',
database='',
user='',
password='')
if connection.is_connected():
db_Info = connection.get_server_info()
print("Wersja servera MySQL:", db_Info)
cursor = connection.cursor(prepared=True)
cursor.execute("select database();")
record = cursor.fetchone()
print("Pomyslnie polaczono z baza danych: ", record)
except Error as e:
print("Blad polaczenia!", e)
quit()
try:
startdate = input("Prosze podac poczatek przedzialu czasowego (format RRRR-MM-DD GG:MM:SS): ")
enddate = input("Prosze podac koniec przedzialu czasowego (format RRRR-MM-DD GG:MM:SS): ")
query = "SELECT * FROM BRDB.RFX_IKW_MODIFY_EXEC_ORDER_CANCEL_LOG WHERE CREATION_DATETIME between '%s' and '%s' ORDER BY CREATION_DATETIME DESC;"
tuple1 = (startdate, enddate,)
cursor.execute(query, tuple1)
records = cursor.fetchall()
print("Fetching each row using column name")
for row in records:
message_id = row["MESSAGE_ID"]
executable_order_id = row["EXECUTABLE_ORDER_ID"]
creation_datetime = row["CREATION_DATETIME"]
message_type = row["MESSAGE_TYPE"]
message_status = row["MESSAGE_STATUS"]
print(message_id, executable_order_id, creation_datetime, message_status)
except mysql.connector.Error as e:
print("Error reading data from MySQL table", e)
finally:
if connection.is_connected():
cursor.close()
connection.close()
print("MySQL connection is closed")

Python psycopg2 returning False when pgadmin returns True

I have a python function in which I want to check if a PostgreSQL table exists or not (True, False)
it does not return True... even when I am logged into the same DB and checking in PGAdmin4.. and getting True.
Am I missing a commit? I tried adding a commit() to no effect.
def __exists_table(self, table_name):
cursor = self.__get_a_cursor()
try:
string_to_execute = "SELECT EXISTS(SELECT 1 FROM pg_catalog.pg_tables WHERE schemaname = 'public' AND tablename = '" + table_name + "');"
cursor.execute(string_to_execute)
query_results = cursor.fetchall()
if len(query_results) > 1:
print("__exists_data got back multiple results, using the first")
query_results = query_results[0][0]
return query_results
except Exception as err:
print("Exception on __exists_table: " + str(err))
raise err
finally:
cursor.close()

Your code appears to work as written.
I have a database that contains a single table, table1:
$ psql -h localhost
psql (11.6, server 12.1 (Debian 12.1-1.pgdg100+1))
Type "help" for help.
lars=> \d
List of relations
Schema | Name | Type | Owner
--------+--------+-------+-------
public | table1 | table | lars
(1 row)
If I wrap your code up in a runnable script, like this:
import psycopg2
class DBTest:
def __init__(self):
self.db = psycopg2.connect('host=localhost dbname=lars password=secret')
def __get_a_cursor(self):
return self.db.cursor()
def __exists_table(self, table_name):
cursor = self.__get_a_cursor()
try:
string_to_execute = "SELECT EXISTS(SELECT 1 FROM pg_catalog.pg_tables WHERE schemaname = 'public' AND tablename = '" + table_name + "');"
cursor.execute(string_to_execute)
query_results = cursor.fetchall()
if len(query_results) > 1:
print("__exists_data got back multiple results, using the first")
query_results = query_results[0][0]
return query_results
except Exception as err:
print("Exception on __exists_table: " + str(err))
raise err
finally:
cursor.close()
def test_exists_table(self, table_name):
return self.__exists_table(table_name)
db = DBTest()
for table_name in ['table1', 'table2']:
if db.test_exists_table(table_name):
print(f'{table_name} exists')
else:
print(f'{table_name} does not exist')
Running it produces the output I would expect:
table1 exists
table2 does not exist
Having said that, I would make the follow change to your code. First, rather than creating your query string like this:
string_to_execute = """SELECT EXISTS(
SELECT 1 FROM pg_catalog.pg_tables
WHERE schemaname = 'public'
AND tablename = '""" + table_name + "');"
cursor.execute(string_to_execute)
I would let your database driver take care of parameter substitution for you:
string_to_execute = """SELECT EXISTS(
SELECT 1 FROM pg_catalog.pg_tables
WHERE schemaname = 'public'
AND tablename = %s
)"""
cursor.execute(string_to_execute, (table_name,))
This is easier to read and safer, since it will properly quote any special character in the parameter.

optimize python code for quicker response

This code works, but is very slow. And I will like to use sqlalchemy module because the rest of the script uses that instead of mysql. Is there any advantage of using sqlalchemy or should I continue with this ...
for emp_id in mylist:
try:
connection = mysql.connector.connect(host='x.x.x.x', port='3306', database='xxx', user='root', password='xxx')
cursor = connection.cursor(prepared=True)
sql_fetch_blob_query = """SELECT col1, col2, Photo from tbl where ProfileID = %s"""
cursor.execute(sql_fetch_blob_query, (emp_id, ))
record = cursor.fetchall()
for row in record:
image = row[2]
file_name = 'myimages4'+'/'+str(row[0])+ '_' + str(row[1]) + '/' + 'simage' + str(emp_id) + '.jpg'
write_file(image, file_name)
except mysql.connector.Error as error :
connection.rollback()
print("Failed to read BLOB data from MySQL table {}".format(error))
finally:
if(connection.is_connected()):
cursor.close()
connection.close()

Do you really need to set up new mysql connection and obtain cursor on each iteration? If no, opening it once at the beginning will really speed up your code.
connection = mysql.connector.connect(host='x.x.x.x', port='3306', database='xxx', user='root', password='xxx', charset="utf8")
cursor = connection.cursor(prepared=True)
for emp_id in mylist:
try:
sql_fetch_blob_query = """SELECT col1, col2, Photo from tbl where ProfileID = %s"""
cursor.execute(sql_fetch_blob_query, (emp_id, ))
record = cursor.fetchall()
for row in record:
image = row[2]
file_name = 'myimages4'+'/'+str(row[0])+ '_' + str(row[1]) + '/' + 'simage' + str(emp_id) + '.jpg'
write_file(image, file_name)
except mysql.connector.Error as error :
connection.rollback()
print("Failed to read BLOB data from MySQL table {}".format(error))
finally:
# ouch ...
if(connection.is_connected()):
cursor.close()
connection.close()
UPD:
Actually you don't even need to make N queries to database, because all data can be obtained in one query with WHERE ProfileID IN (.., ..) SQL statement. Take a look this small code, which solves a pretty much identical task:
transaction_ids = [c['transaction_id'] for c in checkouts]
format_strings = ','.join(['%s'] * len(transaction_ids))
dm_cursor.execute("SELECT ac_transaction_id, status FROM transactions_mapping WHERE ac_transaction_id IN (%s)" % format_strings, tuple(transaction_ids))
payments = dm_cursor.fetchall()
Please use it to solve your problem.

Python sqlite loop thought all tables in database

I have 700 tables in a test.db file, and was wondering how do I loop through all these tables and return the table name if columnA value is -?
connection.execute('SELECT * FROM "all_tables" WHERE "columnA" = "-"')
How do I put all 700 tables in all_tables?

To continue on a theme:
import sqlite3
try:
conn = sqlite3.connect('/home/rolf/my.db')
except sqlite3.Error as e:
print('Db Not found', str(e))
db_list = []
mycursor = conn.cursor()
for db_name in mycursor.execute("SELECT name FROM sqlite_master WHERE type = 'table'"):
db_list.append(db_name)
for x in db_list:
print "Searching",x[0]
try:
mycursor.execute('SELECT * FROM '+x[0]+' WHERE columnA" = "-"')
stats = mycursor.fetchall()
for stat in stats:
print stat, "found in ", x
except sqlite3.Error as e:
continue
conn.close()

SQLite
get all tables name:
SELECT name FROM sqlite_master WHERE type='table' ORDER BY name;
Cycle
for table in tables:
...
connection.execute('SELECT * FROM "table1" WHERE "columnA" = "-"')
or one SQL request UNION
sql = []
for table in tables
sql.append('(SELECT * FROM "' + table + '" WHERE "columnA" = "-";)')
' UNION '.join(sql)

You could query the sqlite_master to get all the table names within your database: SELECT name FROM sqlite_master WHERE type = 'table'
sqlite_master can be thought of as a table that contains information about your databases (metadata).
A quick but most likely inefficient way (because it will be running 700 queries with 700 separate resultsets) to get the list of table names, loop through those tables and return data where columnA = "-":
for row in connection.execute('SELECT name FROM sqlite_master WHERE type = "table" ORDER BY name').fetchall()
for result in connection.execute('SELECT * FROM ' + row[1] + ' WHERE "columnA" = "-"').fetchall()
# do something with results
Note: Above code is untested but gives you an idea on how to approach this.

Using placeholders in a database query doesn't work

I am querying a mysql database version 5.6.13 using python 2.7.
This works:
whichCustomer = str(1934)
qry = ("SELECT * FROM customers WHERE customerid = " + whichCustomer)
cursor.execute(qry)
The query also works:
qry = ("SELECT * FROM customers WHERE customerid = 1934")
cursor.execute(qry)
BUT, when I try to use string substitution the query fails:
whichCustomer = 1934
qry = ("SELECT * FROM customers WHERE customerid = %d")
cursor.execute(qry, (whichCustomer))
Is there something I am missing. The full try/execute code follows:
try:
import mysql.connector
print 'Module mysql initialized'
print 'Attempting connection to cheer database'
cnx = mysql.connector.connect(user='notsure',
password='notsure',
host='localhost',
database='notreal')
cursor = cnx.cursor()
whichCustomer = str(1934)
qry = ("SELECT * FROM customers WHERE customerid = " + whichCustomer)
cursor.execute(qry)
recx = cursor.fetchone()
print recx[1]
cnx.close()
print 'Successful connection to notreal database'
except:
print 'Error initialzing mysql databsasr'

You need to use %s for SQL parameters, and the second argument must be a sequence, like a tuple:
whichCustomer = 1934
qry = ("SELECT * FROM customers WHERE customerid = %s")
cursor.execute(qry, (whichCustomer,))
Note the comma in the second parameter; without a comma, that parameter is not a tuple and just the 1934 integer value is passed in instead.
Although both Python string interpolation placeholders and SQL parameters use closely related syntax, they are not the same thing. As such, SQL parameters for positional values are always expressed as %s regardless of the type.