Ignore duplicate rows in PostgreSQL - python

I'm using python and psycopg2 to insert a dict to my PostgreSQL database.
I have postgres version 13.2
At the moment I only have the code below to execute :
INSERT INTO movie_data(title, description, rating, published, cast_and_crew, age_group, country)
VALUES ('movie name', 'something', 8, 2020, 'an actor', 'pg-13', 'GB')
But because I already have this row in my db I get the 'already exists' error.
I've tried the ON CONFLICT DO NOTHING but I get the error below :
there is no unique or exclusion constraint matching the ON CONFLICT specification
What's the best way to ignore the duplicate rows and keep on inserting?
I'm new to SQL so if anyone could explain and include the whole code for this purpose I would really appreciate it.
UPDATE : below is my CREATE TABLE statement
CREATE TABLE public.movie_data
(
title text COLLATE pg_catalog."default" NOT NULL,
description text COLLATE pg_catalog."default" NOT NULL,
published integer,
cast_and_crew text COLLATE pg_catalog."default",
age_group text COLLATE pg_catalog."default",
country text COLLATE pg_catalog."default",
rating integer,
CONSTRAINT movie_data_pkey PRIMARY KEY (title, description)
)
TABLESPACE pg_default;
ALTER TABLE public.movie_data
OWNER to postgres;

error is clear , you need to have a unique constraint on your table for the columns that needs to be unique
alter table movie_data
add constraint constraintname unique (title, description);
now you can use this constraint :
INSERT INTO movie_data(title, description, rating, published, cast_and_crew, age_group, country)
VALUES ('movie name', 'something', 8, 2020, 'an actor', 'pg-13', 'GB')
on conflict on constraint constraintname do nothing;
as Adrian pointed out , instead of that , if your Primary key is on those column that needs to be unique , you simply could :
INSERT INTO movie_data(title, description, rating, published, cast_and_crew, age_group, country)
VALUES ('movie name', 'something', 8, 2020, 'an actor', 'pg-13', 'GB')
on conflict (title, description) do nothing;

Why would you like to have a primary key with duplicated rows? The main purpose of primary keys is to uniqueliy identify rows.
I think you should drop your primary key:
alter table public.movie_data drop constraint movie_data_pkey;

Related

insert row if not exist in database

Hello how can i do to insert uniq rows without duplicate.
cursor.execute("CREATE TABLE IF NOT EXISTS tab1 (id varchar(36) primary key, cap1 VARCHAR(4), cap2 varchar(55), cap3 int(6), Version VARCHAR(4));")
id = uuid.uuid1()
id = str(id)
cursor.execute("INSERT IGNORE INTO tab1 (id, cap1, cap2, cap3, Version) VALUES (%s, %s, %s, %s, %s )", (vals))
I should not insert the third row while is the same as first row.
Hope im clear .
Thank you in advance,
The problem is that uuid() will always give a unique identifier and since id is a primary key, the row is getting inserted with duplicate values except for id column which is different always.
I think this link might answer your question or else, create a unique index on columns that you want to be unique.
Let me know if it helps!!

SQLite with Python "Table has X columns but Y were supplied"

I have a python script that executes some simple SQL.
c.execute("CREATE TABLE IF NOT EXISTS simpletable (id integer PRIMARY KEY, post_body text, post_id text, comment_id text, url text);")
command = "INSERT OR IGNORE INTO simpletable VALUES ('%s', '%s', '%s', '%s')" % (comments[-1].post_body, comments[-1].post_id, comments[-1].comment_id,
comments[-1].url)
c.execute(command)
c.commit()
But when I execute it, I get an error
sqlite3.OperationalError: table simpletable has 5 columns but 4 values were supplied
Why is it not automatically filling in the id key?
In Python 3.6 I did as shown below and data was inserted successfully.
I used None for autoincrementing ID since Null was not found.
conn.execute("INSERT INTO CAMPAIGNS VALUES (?, ?, ?, ?)", (None, campaign_name, campaign_username, campaign_password))
The ID structure is as follows.
ID INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL
If you don't specify the target columns VALUES is expected to provide values for all columns and that you didn't do.
INSERT
OR IGNORE INTO simpletable
(text,
post_id,
comment_id,
text)
VALUES ('%s',
'%s',
'%s',
'%s');
Specifying the target columns is advisable in any case. The query won't break, if, for any reason, the order of the columns in the tables changes.
try to specify the columns names to ensure that the destination of values doesn't depends on order.
ex:
INTO simpletable
(text,
post_id,
comment_id,
text)
And if you wants the id column to be automatically incremented make sure to add Identity property on, or similar auto increment of your dbms.
ex:
CREATE TABLE IF NOT EXISTS simpletable (id integer PRIMARY KEY Identity(1,1),
and remember your script is not prepared to alter the table structure, only creation.
If you wrote code correctly delete your SQL file(name.db) and run your code again some time it solve the problem.
Imagine this is your code:
cursor.execute('''CREATE TABLE IF NOT EXISTS food(name TEXT , price TEXT)''')
cursor.execute('INSERT INTO food VALUES ("burger" , "20")')
connection.commit()
and you see an error like this:
table has 1 column but 2 values were supplied
it happened because for example you create a file with one column and then you modify your file to two column but you don't change the file name so compiler do not over write it because it exist.

Python sqlite - insert if not exists [duplicate]

I have an SQLite database. I am trying to insert values (users_id, lessoninfo_id) in table bookmarks, only if both do not exist before in a row.
INSERT INTO bookmarks(users_id,lessoninfo_id)
VALUES(
(SELECT _id FROM Users WHERE User='"+$('#user_lesson').html()+"'),
(SELECT _id FROM lessoninfo
WHERE Lesson="+lesson_no+" AND cast(starttime AS int)="+Math.floor(result_set.rows.item(markerCount-1).starttime)+")
WHERE NOT EXISTS (
SELECT users_id,lessoninfo_id from bookmarks
WHERE users_id=(SELECT _id FROM Users
WHERE User='"+$('#user_lesson').html()+"') AND lessoninfo_id=(
SELECT _id FROM lessoninfo
WHERE Lesson="+lesson_no+")))
This gives an error saying:
db error near where syntax.
If you never want to have duplicates, you should declare this as a table constraint:
CREATE TABLE bookmarks(
users_id INTEGER,
lessoninfo_id INTEGER,
UNIQUE(users_id, lessoninfo_id)
);
(A primary key over both columns would have the same effect.)
It is then possible to tell the database that you want to silently ignore records that would violate such a constraint:
INSERT OR IGNORE INTO bookmarks(users_id, lessoninfo_id) VALUES(123, 456)
If you have a table called memos that has two columns id and text you should be able to do like this:
INSERT INTO memos(id,text)
SELECT 5, 'text to insert'
WHERE NOT EXISTS(SELECT 1 FROM memos WHERE id = 5 AND text = 'text to insert');
If a record already contains a row where text is equal to 'text to insert' and id is equal to 5, then the insert operation will be ignored.
I don't know if this will work for your particular query, but perhaps it give you a hint on how to proceed.
I would advice that you instead design your table so that no duplicates are allowed as explained in #CLs answer below.
For a unique column, use this:
INSERT OR REPLACE INTO tableName (...) values(...);
For more information, see: sqlite.org/lang_insert
insert into bookmarks (users_id, lessoninfo_id)
select 1, 167
EXCEPT
select user_id, lessoninfo_id
from bookmarks
where user_id=1
and lessoninfo_id=167;
This is the fastest way.
For some other SQL engines, you can use a Dummy table containing 1 record.
e.g:
select 1, 167 from ONE_RECORD_DUMMY_TABLE

Print values from another table SQLite3 w/ Python

I want to create a simple recipe script. So I have a Table with some recipes and one with ingredients. Now I linked all ingredients_id to the the recipe.
Is it possible to print the name of those id's from another table?
import sqlite3
conn = sqlite3.connect('food.db')
c = conn.cursor()
c.execute("""CREATE TABLE IF NOT EXISTS ingredients (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL)""")
c.execute("""CREATE TABLE IF NOT EXISTS recipes (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
quantity REAL,
ingredients_id TEXT,
FOREIGN KEY(ingredients_id) REFERENCES ingredients(id)
)""")
c.execute("INSERT INTO recipes VALUES ('0', 'Pasta with Tomato', '1', '2,3')")
c.execute("INSERT INTO ingredients VALUES ('2', 'Pasta')")
c.execute("INSERT INTO ingredients VALUES ('3', 'Tomato')")
c.execute("SELECT * FROM recipes WHERE id='0'")
print(c.fetchone())
conn.commit()
conn.close()
Your problem is classic many-to-many relationship.
Each product can be ingredient in many recipes, each recipe can have multiple ingredients.
To implement this, you need 3rd table, that holds "membership" per product per recipe. This is common (if not best) practice for m2m problem.
There's a lot of examples in SO, pls look for them

Python sqlite3: INSERT into table WHERE NOT EXISTS, using ? substitution parameter

I'm creating a table of descriptions from a list of not necessarily unique descriptions. I would like the table to contain only distinct descriptions, so while inserting descriptions into the table, I need to check to see if they already exist. My code(simplified) looks something like as follows:
cur.execute(''' CREATE TABLE descriptions
(id INTEGER PRIMARY KEY AUTOINCREMENT, desc TEXT)''')
descripts = ["d1", "d2", "d3", "d4", "d3", "d1", "d5", "d6", "d7", "d2"]
cur.executemany('''
INSERT INTO descriptions(desc)
VALUES (?)
WHERE NOT EXISTS (
SELECT *
FROM descriptions as d
WHERE d.desc=?)
''', zip(descripts, descripts))
The result is OperationalError: near "WHERE": syntax error, and I'm not sure exactly where I'm going wrong.
Just a note: I realize I could solve this using a set() structure in python, but for academic reasons this is not permitted.
Thanks
To replace VALUES by SELECT should work
cursor.executemany('''
INSERT INTO descriptions(desc)
SELECT (?)
WHERE NOT EXISTS (
SELECT *
FROM descriptions as d
WHERE d.desc=?)''',
zip(descripts, descripts))

Categories