Using Python 2.7 and
In [150]: psycopg2.version
Out[150]: '2.4.2 (dt dec pq3 ext)'
I have a simple python scripts that processing transactions and writes data to a database. Occasionally there is an insert that violates my primary key. This is fine, i just want it to ignore that record and continue on it merry way. The problem I am having is that psycopg2 primary key error is aborting the entire transaction block and all inserts after the error fail. Here is an example error
ERROR: duplicate key value violates unique constraint "encounter_id_pkey"
DETAIL: Key (encounter_id)=(9012235) already exists.
This is on the next insert. not a violation.
Inserting: 0163168~9024065
ERROR: current transaction is aborted, commands ignored until end of transaction block
The Second error repeats itself for every insert. Here is a simplified loop. I am looping through a pandas data frame, but it could be any loop.
conn = psycopg2.connect("dbname='XXXX' user='XXXXX' host='XXXX' password='XXXXX'")
cur = conn.cursor()
for i, val in df2.iteritems():
try:
cur = conn.cursor()
cur.execute("""insert into encounter_id_table (
encounter_id,current_date )
values
(%(create_date)s, %(encounter_id)s ) ;""",
'encounter_id':i.split('~')[1],
'create_date': datetime.date.today() })
cur.commit()
cur.close()
except Exception , e:
print 'ERROR:', e[0]
cur.close()
conn.close()
Again the basic idea is to gracefully handle the Error. In the dictum of Admiral Nelson of the Royal Navy: "Damn the maneuvers go straight at them". Or in our case damn the Errors go straight at them." I thought by opening a cursor on every insert that I would be resetting the transaction block. I do not want to have to reset the connection just because of a primary key error. Is there something i am just missing?
Thanks before hand for your time.
John
You should rollback transaction on error.
I've added one more try..except..else construction in the code bellow to show the exact place where exception will occur.
try:
cur = conn.cursor()
try:
cur.execute("""insert into encounter_id_table (
encounter_id,current_date )
values
(%(create_date)s, %(encounter_id)s ) ;""",
'encounter_id':i.split('~')[1],
'create_date': datetime.date.today() })
except psycopg2.IntegrityError:
conn.rollback()
else:
conn.commit()
cur.close()
except Exception , e:
print 'ERROR:', e[0]
First of all: CURRENT_DATE is a reserved word in every SQL standard as well as in PostgreSQL. You cannot use it as identifier without double-quoting it. I would strongly advice not to use it at all. I renamed the column to curdate in my example
Next, I am no expert in python syntax, but you seem to have reversed the order of your insert-columns:
(%(create_date)s, %(encounter_id)s )
Should be:
( %(encounter_id)s, %(create_date)s)
To your main question: you can avoid the problem altogether by checking if the key is already in the table before using it in the insert command:
INSERT INTO encounter_id_table (encounter_id, curdate)
SELECT 1234, now()::date
WHERE NOT EXISTS (SELECT * FROM encounter_id_table t
WHERE t.encounter_id = 1234);
In Python syntax, that should be:
cur.execute("""INSERT INTO encounter_id_table (encounter_id, curdate)
SELECT %(encounter_id)s, %(create_date)s,
WHERE NOT EXISTS (
SELECT * FROM encounter_id_table t
WHERE t.encounter_id = %(encounter_id)s);""",
{'encounter_id':i.split('~')[1],
'create_date': datetime.date.today()})
Related
I got some python code (psycopg2) with which should insert data into a database:
def debug(self):
try:
self.connection.execute(
"SELECT test();")
res = self.connection.fetchall()
print(res)
except Exception as e:
print(e)
return
The test() function in pgsql is this:
CREATE OR REPLACE FUNCTION test(
) RETURNS setof varchar
AS $Body$
BEGIN
INSERT INTO Linie(name) VALUES('3');
RETURN QUERY(SELECT * FROM linie);
END;
$Body$ LANGUAGE plpgsql;
When i change the "name" value and execute the query in pgAdmin there is a now entry in the database. However when calling the function from python it always overrides the value.
The table is defined as follows:
CREATE TABLE Linie(
name varchar,
PRIMARY KEY (name)
);
For example with pgAdmin i can insert 1,2,3,4,5.
With python after running 5 equivalent queries it is just 5.
Calling the test function with nodeJS works fine.
When calling the function once from python then changing the insert value and then calling it from python again, the values are not replaced but inserted.
Also it does not throw any errors and returns the table as it should (except the replaced value).
why is this happening and what can i do against it?
Psycopg2 by default will not commit changes made to the database unless you explicitly call connection.commit() after executing your SQL. You could alter you code like so:
def debug(self):
try:
self.connection.execute(
"SELECT test();")
res = self.connection.fetchall()
self.connection.commit()
print(res)
except Exception as e:
print(e)
return
However, please be careful doing this as I have no information on what exactly self.connection is an instance of, therefore I have assumed it to be of type connection :)
Alternatively, when you setup your connection to the DB, set the property autocommit to True, as documented here. Example:
self.connection = psycopg2.connect(user='foo', password='bar', host='localhost', dbname='mydb')
self.connection.autocommit = True
If you are already using autocommit let me know and I'll have another look at your question.
I'm trying to use sqlite3 in python to delete a selection of rows from a table. My attempt fails, but I can't work out why.
The sql query works ok, but I can't implement it within the python code.
I have a set of records that are moved from current_table to archive_table after a period of time.
I'd like to clean up the current_table by removing those rows that are in the archive_table (matched on id).
Intended SQL query:
DELETE FROM current_table WHERE id IN ( SELECT id FROM archive_table);
Attempted python code:
import sqlite3
def clean_up(current_table, archive_table):
db = sqlite3.connect(sqlite_db)
cursor = db.cursor()
sql_query_delete = '''DELETE FROM %s WHERE id IN ( SELECT id FROM %s);''' % (current_table, archive_table)
try:
cursor.execute(sql_query_delete)
db.commit()
db.close()
except:
print("error deleting")
Now working. The database file was locked by another process. Removing the pointless try/except led me to the detailed error message.
I was wondering if anyone could help me figure out why the data that my python script has gathered is not being input into the mysql database properly
Here is my code
/*** index is a dictionary that has the struture:
index = {links:[words], links: [words],.......} ***/
#There are multiple items in words
for k, v in index.iteritems():
links = str(k)
words = str(v) [1:-1]
import MySQLdb
db = MySQLdb.connect("localhost","root","","my_db" )
cursor = db.cursor()
try:
cursor.execute(('insert into SITE(url, keywords) values("%s", "%s")' % \
(links,words)))
db.commit()
except:
db.rollback()
db.close()
Each row in my table should have two feilds: url and keywords
Instead of insterting 6 rows it only inserts two.
Please help?!
Perhaps there is a problem, because you open a new connection for every item. Then you shouldn't format values into a SQL-statement, use parameters instead. Third, you shouldn't ignore exceptions, because then, you cannot figure out, what's going wrong. The representation of a list is nothing, to work with in production code, use join instead
import logging
import MySQLdb
db = MySQLdb.connect("localhost","root","","my_db" )
for links, words in index.iteritems():
cursor = db.cursor()
try:
cursor.execute('insert into SITE(url, keywords) values(%s, %s)', (links, ','.join(words)))
db.commit()
except Exception:
logging.exception('insert went wrong')
db.rollback()
db.close()
I am using raw sql queries for inserting the data into DB. The insertion is working right, Now I want to perform some checks on this insert query e.g. If the query has inserted the data or not
suppose I have insert query like
cursor.execute("some insert query" )
Now I want to know whether cursor.execute has inserted the row then show me some text like success and if it fails to insert for some reason then show me text like error and also if the row is already inserted then show, row already exist.
But I don't know how to perform these checks on cursor.execute.
edit
for i in range(numrows):
row = cursor.fetchone()
if row[6]==1:
arr["user_id"]=row[0]
arr["email"]=row[1]
arr["style_quiz_score"]=row[2]
arr["style_quiz_answer"]=row[3]
arr["date_joined"]=row[4]
arr["is_active"]=row[5]
arr['firstname'] = row[7]
arr["username"]=re.sub(r'[^a-zA-Z0-9]', '_', arr["email"])
elif row[6]==2:
arr['lastname'] = row[7]
cursor1.execute("insert into auth_user(id,username,first_name,last_name,email,password,is_staff,is_active,is_superuser,date_joined,last_login) values(%s,%s,%s,%s,%s,'NULL',0,%s,0,%s,0)",[arr["user_id"],arr["username"],arr['firstname'],arr['lastname'],arr["email"],arr["is_active"],arr["date_joined"]])
when i am executing cursor1.execute outside forloop than it insert the last entry , but if i execute it in inside forloop than it gives error and nothing will be inserted
Assuming you're using Django (you're not specific about it in your question, but you're using the django tag), you need to do transaction.commit_unless_managed() (from django.db import transaction) after issuing the insert query with cursor.execute.
You can check for exceptions when calling commit_unless_managed to see if the insert went well or not:
from django.db import connection, transaction, DatabaseError, IntegrityError
cursor = connection.cursor()
cursor.execute("some insert query" )
try:
transaction.commit_unless_managed()
except DatabaseError, IntegrityError:
print 'error'
else:
print 'success'
I have some code that links to Access and works fine with adodbapi, except for one niggling issue which I cant resolve. Basically I want to create a new table in Access with the Column Headings "Key" and "Value" but it doenst seem to work unless I include the commas which I dont want.
I get the following error:
adodbapi.adodbapi.DatabaseError: (-2147352567, 'Exception occurred.', (0, u'Microsoft JET Database Engine', u'Syntax error in field definition.', None, 5003292, -2147217900), None)
import adodbapi
# create the DSN execution string and point it to the desired Database
database = 'D:\Temp.mdb'
constr = 'Provider=Microsoft.Jet.OLEDB.4.0; Data Source=%s ' % database
conn = adodbapi.connect(constr)
# create a cursor
cur = conn.cursor()
# below doesnt work
cur.execute('CREATE TABLE OtherInfo(Key VARCHAR(100), Value VARCHAR(100))')
# but this does
cur.execute('CREATE TABLE OtherInfo(Key2 VARCHAR(100), Value2 VARCHAR(100))')
# so does this
cur.execute('CREATE TABLE OtherInfo('Key' VARCHAR(100), 'Value' VARCHAR(100))')
# this also fails unless similar to above
cur.execute("INSERT INTO OtherInfo(Key,Value) VALUES('AppName','XXX')")
# close the cursor and connection
conn.commit() # this saves all of the changes made above
cur.close()
conn.close()
How can I make it insert Column headings and Data as {Key, Value} without having to resort to 'Key' etc as the program which uses this table cannot reference other names?
Thanks for any help.
Figured it out, it needs a [wrapper] to work as below:
cur.execute('CREATE TABLE OtherInfo([Key] VARCHAR(100), [Value] VARCHAR(100))')
Thanks to anyone who took the trouble to view.