We Keep Coding

python sqlite - deleting selected records

I'm trying to use sqlite3 in python to delete a selection of rows from a table. My attempt fails, but I can't work out why.
The sql query works ok, but I can't implement it within the python code.
I have a set of records that are moved from current_table to archive_table after a period of time.
I'd like to clean up the current_table by removing those rows that are in the archive_table (matched on id).
Intended SQL query:
DELETE FROM current_table WHERE id IN ( SELECT id FROM archive_table);
Attempted python code:
import sqlite3
def clean_up(current_table, archive_table):
db = sqlite3.connect(sqlite_db)
cursor = db.cursor()
sql_query_delete = '''DELETE FROM %s WHERE id IN ( SELECT id FROM %s);''' % (current_table, archive_table)
try:
cursor.execute(sql_query_delete)
db.commit()
db.close()
except:
print("error deleting")

Now working. The database file was locked by another process. Removing the pointless try/except led me to the detailed error message.

Python/MySQL: Using %s in a pythonic manner instead of a mysql.connector manner in an INSERT statement

I am trying to write a script that reads a tab delimited text file and first creates a mysql table and then inserts the data into that table.
Problem: I'm stuck on writing the INSERT query because %s placeholder serves a new purpose with the mysql.connector API. Here is my code:
def insertmanyquery(tabletitle, headers, values):
'''connects to a mysql database and inserts a list of tab delimited rows into a table'''
cnxn = connect(all the connection parameters)
cursor = cnxn.cursor()
numofvalues = r"%s, " * len(headers.split(','))
numofvalues = numofvalues[:-2]
query = "INSERT INTO %s (%s) VALUES (%s)" % (tabletitle, headers, numofvalues)
cursor.executemany(query,values)
cnxn.commit()
cursor.close()
cnxn.close()
This would hopefully allow the insert query to adapt to however many columns are present in the table.
If I call the function as follows:
tabletitle = 'Bikes'
headers = 'BikeBrand, BikeName, Purpose, Price, YearPurchased'
values = ['Norco', 'Range', 'Enduro', 8,000.00, 2018]
insertmanyquery(tabletitle, headers, values)
I get the following error: mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
If I just print the query instead of executing it, it looks fine:
INSERT INTO Bikes (BikeBrand, BikeName, Purpose, Price, YearPurchased) VALUES (%s, %s, %s, %s, %s)
I believe I am getting this error because the third %s in my INSERT query is being interpreted as a part of the connector INSERT syntax instead of first being interpreted in a pythonic manner and then being interpreted as connector syntax.
I am very new to coding so maybe I'm approaching this all wrong, regardless, I'd like to hear potential solutions to this problem or better ways to code this.
Thank you for your time
UPDATE:
I have tried making the query query = "INSERT INTO %s (%s)" % (tabletitle, headers) + " VALUES (" + numofvalues +");" and I still get the same error! so it doesn't have to do with using the placeholder.. >.<

Python PYDOBC Insert Into SQL Server DB with Parameters

I am currently trying to use pyodbc to insert data from a .csv into an Azure SQL Server database. I found a majority of this syntax on Stack Overflow, however for some reason I keep getting one of two different errors.
1) Whenever I use the following code, I get an error that states 'The SQL contains 0 parameter markers, but 7 parameters were supplied'.
import pyodbc
import csv
cnxn = pyodbc.connect('driver', user='username', password='password', database='database')
cnxn.autocommit = True
cursor = cnxn.cursor()
csvfile = open('CSV File')
csv_data = csv.reader(csvfile)
SQL="insert into table([Col1],[Col2],[Col3],[Col4],[Col5],[Col6],[Col7]) values ('?','?','?','?','?','?','?')"
for row in csv_data:
cursor.execute(SQL, row)
time.sleep(1)
cnxn.commit()
cnxn.close()
2) In order to get rid of that error, I am defining the parameter markers by adding '=?' to each of the columns in the insert statement (see code below), however this then gives the following error: ProgrammingError: ('42000'"[42000] [Microsoft] [ODBC SQL Server Driver][SQL Server] Incorrect syntax near '=').
import pyodbc
import csv
cnxn = pyodbc.connect('driver', user='username', password='password', database='database')
cnxn.autocommit = True
cursor = cnxn.cursor()
csvfile = open('CSV File')
csv_data = csv.reader(csvfile)
SQL="insert into table([Col1]=?,[Col2]=?,[Col3]=?,[Col4]=?,[Col5]=?,[Col6]=?,[Col7]=?) values ('?','?','?','?','?','?','?')"
for row in csv_data:
cursor.execute(SQL, row)
time.sleep(1)
cnxn.commit()
cnxn.close()
This is the main error I am haveing trouble with, I have searched all over Stack Overflow and can't seem to find a solution. I know this error is probably very trivial, however I am new to Python and would greatly appreciate any advice or help.

Since SQL server can import your entire CSV file with a single statement this is a reinvention of the wheel.
BULK INSERT my_table FROM 'CSV_FILE'
WITH ( FIELDTERMINATOR=',', ROWTERMINATOR='\n');
If you want to persist with using python, just execute the above query with pyodbc!
If you would still prefer to execute thousands of statements instead of just one
SQL="insert into table([Col1],[Col2],[Col3],[Col4],[Col5],[Col6],[Col7]) values (?,?,?,?,?,?,?)"
note that the ' sorrounding the ? shouldn't be there.

# creating column list for insertion
colsInsert = "["+"],[".join([str(i) for i in mydata.columns.tolist()]) +']'
# Insert DataFrame recrds one by one.
for i,row in mydata.iterrows():
sql = "INSERT INTO Test (" +colsInsert + ") VALUES (" + "%?,"*(len(row)-1) + "%?)"
cursor.execute(sql, tuple(row))
# cursor.execute(sql, tuple(row))
# the connection is not autocommitted by default, so we must commit to save our changes
c.commit()

import csv file into Mysql Database using python

This is my code:
#!/usr/bin/python
import MySQLdb
import csv
db = MySQLdb.connect(host="host", # The Host
user="username", # username
passwd="pwd", # password
db="databasename") # name of the data base
sqlLoadData = 'LOAD DATA LOCAL INFILE "csv?_file_name.csv" INTO TABLE tablename '
sqlLoadData += 'FIELDS TERMINATED BY "," LINES TERMINATED BY "\n"'
sqlLoadData += 'IGNORE 1 LINES'
sqlLoadData += 'ENCLOSED BY '"' ESCAPED BY "\\" '
try:
curs = db.cursor()
curs.execute(sqlLoadData)
resultSet = curs.fetchall()
except StandardError, e:
print e
db.rollback()
db.close()
I recieve the error Message : You have an error in your SQL Syntax; chekc the manual that correcpond to your Mysql Server.
When I remove the part sqlLoadData += 'ENCLOSED BY '"' ESCAPED BY "\\" ' everything work perfect. I used the last part just to remove the quote from the values.
I also tried:
cursor = mydb.cursor()
reader = csv.reader(open('Cumulative.csv', 'rb'))
reader.next() for row in reader[1:]:
cursor.execute('INSERT INTO Cumulative (C1, C2, C3, C4, C5, C6) VALUES(%s, %s, %s, %s, %s, %s)', row)
cursor.commit()
close the connection to the database.
cursor.close()
I want just to remove the quote so the integer field will support the data. so with quote "1" will be considered as a String instead of integer
Can Anyone please help me to understand this?
Thanks!

looks like you forgot to terminate the preceding line with a space or newline character. Thi sis causing a syntax error when the parser tries to understand LINESENCLOSED which obviously isn't a keyword.
sqlLoadData += 'IGNORE 1 LINES \n'
sqlLoadData += ''ENCLOSED BY '"' ESCAPED BY "\" ''
As a rule of thumb: when you're debugging, and you're able to fix you're code by removing a line, don't rule out the line immediately above
EDIT: Modified the quotes around the second line. I think it was breaking in the "enclosed by" statement.

After 2 days worth of research I found the answer:
!/usr/bin/python
import MySQLdb
import csv
db = MySQLdb.connect(host="host", # The Host
user="username", # username
passwd="pwd", # password
db="databasename") # name of the data base
cursor = connection.cursor()
Query = """ LOAD DATA LOCAL INFILE 'usrl to csv file' INTO TABLE
table_nameFIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED
BY '"' Lines terminated by '\n' IGNORE 1 LINES """
cursor.execute(Query)
connection.commit()
cursor.close()
hope it will help somebody out there.

After days and hours of searching the internet and running into all sort of errors and warnings, this worked perfectly. I hope this saves someone some time
import MySQLdb
import os
import string
db = MySQLdb.connect (host="host",
user="user",
passwd="pwd",
db="database_name",
local_infile = 1) #Grants permission to write to db from an input file. Without this you get sql Error: (1148, 'The used command is not allowed with this MySQL version')
print "\nConnection to DB established\n"
#The statement 'IGNORE 1 LINES' below makes the Python script ignore first line on csv file
#You can execute the sql below on the mysql bash to test if it works
sqlLoadData = """load data local infile 'file.csv' into table table_name FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES;"""
try:
curs = db.cursor()
curs.execute(sqlLoadData)
db.commit()
print "SQL execution complete"
resultSet = curs.fetchall()
except StandardError, e:
print "Error incurred: ", e
db.rollback()
db.close()
print "Data loading complete.\n"
Thanks, I hope this helps :)

Graceful Primary Key Error handling in Python/psycopg2

Using Python 2.7 and
In [150]: psycopg2.version
Out[150]: '2.4.2 (dt dec pq3 ext)'
I have a simple python scripts that processing transactions and writes data to a database. Occasionally there is an insert that violates my primary key. This is fine, i just want it to ignore that record and continue on it merry way. The problem I am having is that psycopg2 primary key error is aborting the entire transaction block and all inserts after the error fail. Here is an example error
ERROR: duplicate key value violates unique constraint "encounter_id_pkey"
DETAIL: Key (encounter_id)=(9012235) already exists.
This is on the next insert. not a violation.
Inserting: 0163168~9024065
ERROR: current transaction is aborted, commands ignored until end of transaction block
The Second error repeats itself for every insert. Here is a simplified loop. I am looping through a pandas data frame, but it could be any loop.
conn = psycopg2.connect("dbname='XXXX' user='XXXXX' host='XXXX' password='XXXXX'")
cur = conn.cursor()
for i, val in df2.iteritems():
try:
cur = conn.cursor()
cur.execute("""insert into encounter_id_table (
encounter_id,current_date )
values
(%(create_date)s, %(encounter_id)s ) ;""",
'encounter_id':i.split('~')[1],
'create_date': datetime.date.today() })
cur.commit()
cur.close()
except Exception , e:
print 'ERROR:', e[0]
cur.close()
conn.close()
Again the basic idea is to gracefully handle the Error. In the dictum of Admiral Nelson of the Royal Navy: "Damn the maneuvers go straight at them". Or in our case damn the Errors go straight at them." I thought by opening a cursor on every insert that I would be resetting the transaction block. I do not want to have to reset the connection just because of a primary key error. Is there something i am just missing?
Thanks before hand for your time.
John

You should rollback transaction on error.
I've added one more try..except..else construction in the code bellow to show the exact place where exception will occur.
try:
cur = conn.cursor()
try:
cur.execute("""insert into encounter_id_table (
encounter_id,current_date )
values
(%(create_date)s, %(encounter_id)s ) ;""",
'encounter_id':i.split('~')[1],
'create_date': datetime.date.today() })
except psycopg2.IntegrityError:
conn.rollback()
else:
conn.commit()
cur.close()
except Exception , e:
print 'ERROR:', e[0]

First of all: CURRENT_DATE is a reserved word in every SQL standard as well as in PostgreSQL. You cannot use it as identifier without double-quoting it. I would strongly advice not to use it at all. I renamed the column to curdate in my example
Next, I am no expert in python syntax, but you seem to have reversed the order of your insert-columns:
(%(create_date)s, %(encounter_id)s )
Should be:
( %(encounter_id)s, %(create_date)s)
To your main question: you can avoid the problem altogether by checking if the key is already in the table before using it in the insert command:
INSERT INTO encounter_id_table (encounter_id, curdate)
SELECT 1234, now()::date
WHERE NOT EXISTS (SELECT * FROM encounter_id_table t
WHERE t.encounter_id = 1234);
In Python syntax, that should be:
cur.execute("""INSERT INTO encounter_id_table (encounter_id, curdate)
SELECT %(encounter_id)s, %(create_date)s,
WHERE NOT EXISTS (
SELECT * FROM encounter_id_table t
WHERE t.encounter_id = %(encounter_id)s);""",
{'encounter_id':i.split('~')[1],
'create_date': datetime.date.today()})