We Keep Coding

Psycopg2 : Insert multiple values if not exists in the table

I need to insert multiple values into a table after checking if it doesn't exist using psycopg2.
The query am using:
WITH data(name,proj_id) as (
VALUES ('hello',123),('hey',123)
)
INSERT INTO keywords(name,proj_id)
SELECT d.name,d.proj_id FROM data d
WHERE NOT EXISTS (SELECT 1 FROM keywords u2 WHERE
u2.name=d.name AND u2.proj_id=d.proj_id)
But how to format or add the values section from tuple to ('hello',123),('hey',123) in query.

As suggested in the comment, assuming that your connection is already established as conn one of the ways would be:
from typing import Iterator, Dict, Any
def insert_execute_values_iterator(connection, keywords: Iterator[Dict[str, Any]], page_size: int = 1000) -> None:
with connection.cursor() as cursor:
psycopg2.extras.execute_values(
cursor,
""" WITH data(name,proj_id) as (VALUES %s)
INSERT INTO keywords(name,proj_id)
SELECT d.name,d.proj_id FROM data d
WHERE NOT EXISTS (SELECT 1 FROM keywords u2 WHERE
u2.name=d.name AND u2.proj_id=d.proj_id);""",
(( keyword['name'],
keyword['proj_id'] ) for keyword in keywords),
page_size=page_size)
insert_execute_values_iterator(conn,{'hello':123,'hey':123})

insert_query = """WITH data(name, proj_id) as (
VALUES (%s,%s)
)
INSERT INTO keywords(name, proj_id)
SELECT d.name,d.proj_id FROM data d
WHERE NOT EXISTS (
SELECT 1 FROM keywords u2
WHERE u2.name = d.name AND u2.proj_id = d.proj_id)"""
tuple_values = (('hello',123),('hey',123))
psycopg2.extras.execute_batch(cursor,insert_query,tuple_values)

Not able to fetch records from DB through python by using Parameterized Queries

The above function has parameters endTime, startTime, list1 and column_filter to it and I am trying to read a query by making the WHERE clause conditions parameterized.
endT = endTime
startT = startTime
myList = ",".join("'" + str(i) + "'" for i in list1)
queryArgs = {'db': devDB,
'schema': dbo,
'table': table_xyz,
'columns': ','.join(column_filter)}
query = '''
WITH TIME_SERIES AS
(SELECT ROW_NUMBER() OVER (PARTITION BY LocId ORDER BY Created_Time DESC) RANK, {columns}
from {schema}.{table}
WHERE s_no in ? AND
StartTime >= ? AND
EndTime <= ? )
SELECT {columns} FROM TIME_SERIES WHERE RANK = 1
'''.format(**queryArgs)
args = (myList, startT, endT)
return self.read(query, args)
The below is my read which connects to the DB to fetch records and a condition is also added to check if its parameterized or not.
def read(self, query, parameterValues = None):
cursor = self.connect(cursor=True)
if parameterValues is not None:
rows = cursor.execute(query, parameterValues)
else:
rows = cursor.execute(query)
df = pd.DataFrame.from_records(rows.fetchall())
if len(df.columns) > 0:
df.columns = [x[0] for x in cursor.description]
cursor.close()
return df
The query args are getting picked up but not the parameterized values. In my case, it is going inside the read method with parameter values of (myList, startT ,endT) as a tuple. The query in WHERE clause remains unchanged (parameters not able to replace ? ), and as a result I am not able to fetch any records. Can you specify where I might be going wrong?

MySQL - select table name if it contains record for list of tables

I am interested in finding the most efficient manner to query the following:
For a list of table names, return the table name if it contains at least one record that meet the conditions
Essentially, something similar to the following Python code in a single query:
dfs = [pd.read_sql('SELECT name FROM {} WHERE a=1 AND b=2'.format(table), engine) for table in tables]
tables = [table for table, df in zip(tables, dfs) if not df.empty]
Is this possible in MySQL?

Assuming you trust the table names in tables not to contain any surprises leading to SQL injection, you could device something like:
from sqlalchemy import text
selects = [f'SELECT :table_{i} FROM {table} WHERE a = 1 AND b = 2'
for i, table in enumerate(tables)]
stmt = ' UNION '.join(selects)
stmt = text(stmt)
results = engine.execute(
stmt, {f'table_{i}': table for i, table in enumerate(tables)})
or you could use SQLAlchemy constructs to build the same query safely:
from sqlalchemy import table, column, union, and_, select, Integer, literal
tbls = [table(name,
column('a', Integer),
column('b', Integer)) for name in tables]
stmt = union(*[select([literal(name).label('name')]).
select_from(tbl).
where(and_(tbl.c.a == 1, tbl.c.b == 2))
for tbl, name in zip(tbls, tables)])
results = engine.execute(stmt)

You can use a UNION of queries that search each table.
(SELECT 'table1' AS table_name
FROM table1
WHERE a = 1 AND b = 2
LIMIT 1)
UNION
(SELECT 'table2' AS table_name
FROM table2
WHERE a = 1 AND b = 2
LIMIT 1)
UNION
(SELECT 'table3' AS table_name
FROM table3
WHERE a = 1 AND b = 2
LIMIT 1)
...

Using (Select * from a table ) to insert data into a table

I have bottom MySql query (sql1).
sq1 = 'select course_id, creator_id, max(course_num) + 1, recordid
' from Courses where recordid in' \
' (' + ','.join(map(str, RecordMatch1)) + ') group by recordid'
cursor.execute(sql1)
BTW, RecordMatch1 is an object that has matching data from other previous queries.
I am trying to see if this is possible; (select * from sql1) portion.
sql2 = ' insert into Courses (course_id, creator_id, course_num, record_id) '\
' Values ( select * from sql1)'
cursor.execute(sql2)
Or do I have to express everything rather than using (Select * )?
What is best practice?

You can do this, but you should specify columns in case of schema changes.

Just need to confirm you are trying to run a select query and insert its output to a insert query. If that is the case this appears to be good.

yes, you can but you should do something like
sql = "SELECT course_id, creator_id, course_num, record_id FROM Courses"
all = cursor.fetchall()
for i in range(len(all))
sql1 = "INSERT INTO Courses (course_id, creator_id, course_num, record_id) VALUES (%s, %s, %s, %s)"
cursor.execute(sql1, (all[i]['Key'], all[i]['Key2'], all[i]['Key3'], all[i]['Key3']))
you can change the select like you want, remember that return a dictionary so take care about the keys, add print(all) to see what happen with the select and see the keys of each column

How can I make a nested query more efficient?

Lets say I'm having 4 tables 'A(id, type, protocol), B(id, A_id, info), C(id, B_id, details) and D(id, C_id, port_info). Table A and Table B are connected via foreign key id from Table A and A_id from Table B. Similarly, Table B and TableC are connected via foreign key id from TableB and B_id from Table C, and in the same way , Table C and Table D are also connected.
Now, I want to get port_info from Table D of all the protocols from Table A.
I know one method whose time complexity is O(n^4), which I'm using currently. The method is as follow :
db = MySQLdb.connect(host="localhost", user="root", passwd="", db="mydb")
cur = db.cursor()
cur.execute("SELECT * FROM A")
A_results = cur.fetchall()
for A_row in A_results :
id = A_row[0]
cur.execute("SELECT * FROM B WHERE A_id = %d " % (id ))
B_results = cur.fetchall()
for B_row in B_results :
id = B_row[0]
cur.execute("SELECT * FROM C WHERE B_id = %d " % (id ))
c_results = cur.fetchall()
for C_row in C_results :
id = C_row[0]
cur.execute("SELECT * FROM D WHERE C_id = %d " % (id ))
D_results = cur.fetchall()
for D_row in D_results :
print "Port = " + str(port)
But this method takes O(n^4), so is there any efficient way in terms of time complexity , that can solve this problem.
Your suggestions are highly appreciated.

Execute it in a single JOIN query and let MySQL do the necessary optimizations while handling large data sets (which, after all, is what the database is best at), providing your application with a single result set. The query looks like this:
SELECT A.protocol, D.port_info
FROM A JOIN B ON A.id = B.A_id
JOIN C ON B.id = C.B_id
JOIN D ON C.id = D.C_id
ORDER BY protocol
...and then use your cursor to go through that single resultset.