I stepped over an problem while implementing a Database into my Python project.
I'm creating a new Table with the following Code:
mycursor = mydb.cursor()
sql = f"CREATE TABLE IF NOT EXISTS _{self.client_id} (tour_date DATE, tour_distance INT, tour_duration INT, tour_elevation_up INT, tour_elevation_down INT, tour_map_image TEXT, tour_name TEXT, tour_sport TEXT, tour_start_point TEXT, tour_type TEXT)"
mycursor.execute(sql)
mydb.commit()
I'm iterating over my Data and want to past it into the Table. But I won't want that if an entry already exists in the table it adds the same data again.
This is my code I currently have to Insert into my Table:
mycursor = mydb.cursor()
sql = f"INSERT INTO _{self.client_id} (tour_date, tour_distance, tour_duration, tour_elevation_up, tour_elevation_down, tour_map_image, tour_name, tour_sport, tour_start_point, tour_type) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s) "
val = (TourDate, TourDistance, TourDuration, TourElevation_up, TourElevation_down, TourMap_image, TourName, TourSport, TourStart_point, TourType)
mycursor.execute(sql, val)
mydb.commit()
So my question is how can I check if a entry already exists in the Table and then avoiding creating a duplicate?
you can "select count() from your_table_name where client_id='current_id'
if count() return int that is greater than 0, you should not insert it into the databse.
First - avoid to use TEXT without it necessary
Second - create table with necessary indexes:
CREATE TABLE IF NOT EXISTS _{self.client_id} (
tour_date DATE,
tour_distance INT,
tour_duration INT,
tour_elevation_up INT,
tour_elevation_down INT,
tour_map_image TEXT,
tour_name VARCHAR(64) PRIMARY KEY,
tour_sport VARCHAR(64),
tour_start_point VARCHAR(64),
tour_type VARCHAR(64)
);
Third - use INSERT IGNORE ... statement for prevent duplicates
Related
I'm trying to insert some rows, but this problema occurs:
TypeError: not all arguments converted during string formatting
sql code:
f"""insert into {table} ({insert}) VALUES ({formating}) ON CONFLICT ({', '.join(key)}) DO UPDATE SET ({insert}) = ({excluded})"""
sql translate:
insert into public.atend (cd_atendimento, cd_ori_ate, cd_paciente, op_type) VALUES (%s, %s, %s, %s) ON CONFLICT (cd_atendimento) DO UPDATE SET (cd_atendimento, cd_ori_ate, cd_paciente, op_type) = (EXCLUDED.cd_atendimento, EXCLUDED.cd_ori_ate, EXCLUDED.cd_paciente, EXCLUDED.op_type)
I have the following problem: I need a dynamic create statement, depending on what attributes my object has.
its following object:
class Table:
columns = []
def __init__(self, name, columns):
self.columns = columns
self.name = name
def columnsNumber(self) -> int:
return self.columns.__len__()
this is what the insert looks like:
sql = "INSERT INTO tableOverview (tableName, columns, datum) VALUES(%s, %s, CURRENT_TIMESTAMP);"
val = (table.name, table.columns.__len__())
await cursor.execute(sql, (val))
for x in table.columns:
sql = "ALTER TABLE %s ADD COLUMN %s VARCHAR(100) UNIQUE " % (table.name,x)
await cursor.execute(sql)
now I don't know, how to prevent a SQL injection.
For the ALTER TABLE statements you can quote the identifier names with backticks as described here.
for x in table.columns:
sql = "ALTER TABLE `%s` ADD COLUMN `%s` VARCHAR(100) UNIQUE " % (table.name,x)
await cursor.execute(sql)
In the insert statement, the code is already correctly using parameter substitution to ensure the inserted values are correctly quoted.
sql = "INSERT INTO tableOverview (tableName, columns, datum) VALUES(%s, %s, CURRENT_TIMESTAMP);"
val = (table.name, table.columns.len())
await cursor.execute(sql, val)
I want to upsert with least effort, for simplicity, i reduce columns, this not work:
sql = '''INSERT INTO temp.tickets
(id, created_at, updated_at, emails, status)
VALUES
(%s, %s, %s, %s, %s)
ON CONFLICT (id)
DO UPDATE SET ( emails, status) values (%s,%s)
'''
cursor = cm.cursor()
## cm is a custom module
cursor.execute(sql, (ticket['id'],
ticket['created_at'],
ticket['updated_at'],
ticket['emails'], ticket['status'], )
This code show Error:
return super(DictCursor, self).execute(query, vars)
IndexError: tuple index out of range
What I need to change in the cursor.execute() to work?
The Bellow code work but I like to use %s instead of type: email = excluded.email for each columns
sql = '''INSERT INTO temp.tickets
(id, created_at, updated_at, emails, status)
VALUES
(%s, %s, %s, %s, %s)
ON CONFLICT (id)
DO UPDATE SET emails = excluded.eamils, status = excluded.status
'''
cursor = cm.cursor()
# cm is a custom module
cursor.execute(sql, (ticket['id'],
ticket['created_at'],
ticket['updated_at'],
ticket['emails'], ticket['status'], )
There are two Relevant Questions link1, link2
I would try something like this:
sql = '''INSERT INTO temp.tickets
(id, created_at, updated_at, emails, status)
VALUES
(%s, %s, %s, %s, %s)
ON CONFLICT (id)
DO UPDATE SET ( emails, status) values (%s,%s)
'''
cursor = cm.cursor()
## cm is a custom module
cursor.execute(sql, (ticket['id'],
ticket['created_at'],
ticket['updated_at'],
ticket['emails'],
ticket['status'],
ticket['emails'],
ticket['status'] )
Thre number of %s must match the number of parameters.
When Postgres encounters a captured conflict it basically creates a record called EXCLUDED that contains the values you attempted to insert, You can refer to this record in DO UPDATE. Try the following:
INSERT INTO temp.tickets
(id, created_at, updated_at, emails, status)
VALUES
(%s, %s, %s, %s, %s)
ON CONFLICT (id)
DO UPDATE
SET emails = excluded.emails
, status = excluded.status
, updated_at = excluded.updated_at -- my assumption.
...
You will have to format is into the requirements of your source language.
I am using prepared statements for my SQL insert query, and I am receiving the message that there is an error in the syntax.
I have tried using PHPMyAdmin and used the same query in that and substituted the placeholders for the real values and that query worked fine, therefore I am assuming it is something to do with my use of prepared statements.
def create_user(f_name, s_name, email, sex, dob, mobile=None):
try:
conn = mysql.connector.connect(host=host, user=user, passwd=password) # create a connection to the database
cursor = conn.cursor(prepared=True) # Creates a cursor that is expecting prepared
if mobile is not None: # if the mobile number is specified
sql_parameterized_query = ("""BEGIN;
INSERT INTO users (FirstName, Surname, Email, Dob, Gender, Mobile)
VALUES (%s, %s, %s, %s, %s, %s);
INSERT INTO passwords (UserID, hashedPass)
VALUES (LAST_INSERT_ID(),%s);
COMMIT;""")
query_array = (f_name, s_name, email, date_sql_format(dob), sex, mobile, hash_user_password)
else: # if the mobile number is not specified
sql_parameterized_query = ("""BEGIN;
INSERT INTO users (FirstName, Surname, Email, Dob, Gender)
VALUES(%s, %s, %s, %s, %s);
INSERT INTO passwords (UserID, hashedPass)
VALUES(LAST_INSERT_ID(),%s);
COMMIT;""")
query_array = (f_name, s_name, email, date_sql_format(dob), sex, hash_user_password) # Init array of values
cursor.execute(sql_parameterized_query, query_array) # Execute query
conn.commit()
I would like it to insert the details for a new user into the database all fields are required excluding the mobile phone number, that is why I have used the if statement to separate them, if this is poor practice then please guide me in the correct direction for that too as I could not find a more elegant way of solving that issue, anyway, when calling the function like so create_user("Ollie", "Pugh", "oliver.pugh#icloud.com", "M", "22-04-2001")
The variable query_array has the value of ('Ollie', 'Pugh', 'oliver.pugh#icloud.com', '2001-04-22', 'M', '$2b$12$RU9FRcNjgHlC78kjZA5OIeqT1s1K2LHndC2iDK8mcqkadGc8d9XO2')
The message I receive is: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'INSERT INTO users (FirstName, Surname, Email, Dob, Gender)' at line 2
The structure of the table Users is:
CREATE TABLE `users` (
`UserID` int(11) NOT NULL AUTO_INCREMENT,
`FirstName` varchar(255) NOT NULL,
`Surname` varchar(255) NOT NULL,
`Email` varchar(255) NOT NULL,
`Dob` date NOT NULL,
`Gender` char(1) NOT NULL,
`Mobile` varchar(11) DEFAULT NULL,
`timeOfCreation` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`Authority` varchar(255) NOT NULL DEFAULT 'User',
PRIMARY KEY (`UserID`)
) ENGINE=InnoDB AUTO_INCREMENT=23 DEFAULT CHARSET=latin1
The solution to my problem was to create a procedure, I named it CreateUser and consisted of:
BEGIN
START TRANSACTION;
IF mobile = "NULL" THEN
SET mobile = null;
END IF;
INSERT INTO `cl43-flexfit`.users (FirstName, Surname, Email, Dob, Gender,Mobile)
VALUES (f_name, s_name, email, dob, gender, mobile);
INSERT INTO `cl43-flexfit`.passwords (UserID, hashedPass)
VALUES (LAST_INSERT_ID(), passhash);
COMMIT;
END
And I have modified the python script to have two cursors as I could not use a USE statement within the procedure nor could I use one with a cursor that was configured for prepared statements.
try:
conn = mysql.connector.connect(host=host, user=user, passwd=password) # create a connection to the database
cursor = conn.cursor() # Have to have two as you can not select database with a prepared cursor
prep_cursor = conn.cursor(prepared=True) # Creates a cursor that is expecting prepared
if mobile is None: # if the mobile number is not specified
mobile = "NULL" # This is recognised as null in the procedure
sql_parameterized_query = "CALL CreateUser(%s, %s, %s, %s, %s, %s, %s)" # Calls the create user procedure
query_array = (f_name, s_name, email, date_sql_format(dob), sex, mobile, hash_user_password)
cursor.execute("USE `cl43-flexfit`;")
prep_cursor.execute(sql_parameterized_query, query_array) # Execute query
conn.commit()
I'm sure there are still better ways of doing this, but this does the job for now.
I got this code to save some values in a Sql-DB via python. But it doesn't work. Could you please help me to find the problem?
def saveValue(art, value, timestamp):
# Missing: Check if table setting exists or create it
# First check if row exists
cursor.execute('SELECT count(*) FROM setting WHERE parameter=%s', art)
existing = cursor.fetchone()
if existing[0]==0:
cursor.execute('INSERT INTO setting (parameter, wert) VALUES (%s, NULL)', art)
db.commit()
# Update row
cursor.execute('UPDATE setting set value=%s WHERE parameter=%s', value, art)
db.commit()
# Check if measure-table exisits or create ist
cursor.execute('SHOW TABLES LIKE measure_%s', art)
result = cursor.fetchone()
if not result:
cursor.execute('CREATE TABLE measure_%s (id INT NOT NULL AUTO_INCREMENT, time int(11), value float(6), PRIMARY KEY (id))', art)
db.commit()
# Insert measure value
sql = 'INSERT INTO measure_%s (time, value) VALUES (%s, %s)'
args= art, timestamp, value
cursor.execute(sql,args)
db.commit()
Maybe it is even possible to optimize/minimize the code?