I have two variables to insert in my table.
user_id - int
marks - float
and I am having this data for multiple users like this:
user_ids = (-,-,-,-,-,-,-) **TUPLE**
marks = (-,-,-,-,-,-,-,-) **TUPLE**
I want to insert this data into my table using executemany and I am executing this query in my flask snippet:
con = pymysql.connect(
host=host,
user=user,
password=password,
db=db,
charset=charset,
cursorclass=pymysql.cursors.DictCursor,
port=port,
)
cur = con.cursor()
percs = calcattnonull()
# percs contains list of dictionaries.
# {[<'user_id'>: <'marks'>], [<'user_id'>: <'marks'>]........}
id_ = []
perc_ = []
final = []
for perc in tqdm(percs):
id_.append(perc["user_id"])
perc_.append(perc["att_perc"])
id_ = tuple(id_)
perc_ = tuple(perc_)
final.append(id_)
final.append(perc_)
cur.executemany(
"UPDATE dream_offline_calculate SET (user_id,att_percentage) VALUES (?,?)",
final,
)
con.commit()
I am getting this error again and again:
TypeError: not all arguments converted during string formatting
Thanks in advance for helping me.
executemany takes an iterable of the same placeholders you would use when calling execute several times.
So if your original query would be
cur.execute(
"UPDATE dream_offline_calculate SET (user_id,att_percentage) VALUES (?,?)",
(user_id, att_perc),
)
the equivalent executemany would be
cur.executemany(
"UPDATE dream_offline_calculate SET (user_id,att_percentage) VALUES (?,?)",
[(user_id, att_perc)],
)
So that said, simply
cur.executemany(
"UPDATE dream_offline_calculate SET (user_id,att_percentage) VALUES (?,?)",
[(perc["user_id"], perc["att_perc"]) for perc in percs],
)
should do the trick.
Related
I am trying to create a method in python insert records into a table passing in a list of column names, and an associated list of records.
I was able to set it up where the column names populated dynamically via a for loop, but I can't figure out how to do the same thing with values because the psycopg2.executemany function relies on having %s's as placeholders.
Is it possible to have the number of %s's in the string populate dynamically via a loop? Is there another way to do this?
def load_table(dbname,table_name,fields,records):
try:
#Variable Qty Column Loop
sql_fields = []
for i in fields:
i = sql.Identifier(i)
sql_fields.append(i)
#Need similar loop to replace %s values
#Replace (%s,%s,%s) ???
#.....
#.....
sql_values = []
for i in fields:
sql_values.append('%s')
print(sql_values)
flist = sql.SQL(',').join(sql_fields)
connection, cursor = create_connection(dbname)
insert_query = sql.SQL('INSERT INTO {table_name} ({fields}) VALUES (%s,%s,%s)').format(
table_name = sql.Identifier(table_name),
fields = flist,
cursor.executemany(insert_query,records)
print('Records Loaded Successfully')
except (Exception,psycopg2.Error) as error:
print("Failed to insert record into table {error}".format(error = error))
finally:
# closing database connection.
if (connection):
close_connection(connection,cursor)
You can use sql.Placeholder, to populate the sql statement with the amount of %s-placeholders you need:
def load_table(dbname,table_name,fields,records):
con, cur = create_connection('foo')
query = sql.SQL("insert into {} ({}) values ({})").format(
sql.Identifier(table_name),
sql.SQL(', ').join(map(sql.Identifier, fields)),
sql.SQL(', ').join(sql.Placeholder() * len(fields)))
print(query.as_string(con))
if __name__ == '__main__':
dbname = '...'
table_name = 'messages'
fields = ['user_id', 'message_type', 'message_title']
records = [['12345', 'json', 'my first message'], ]
load_table(dbname,table_name,fields,records)
Output:
insert into "messages" ("user_id", "message_type", "message_title") values (%s, %s, %s)
def LiraRateApiCall():
R = requests.get(url)
timestamp = R.json()['buy'][-1][0]/1000
format_date = '%d/%m/%y'
date = datetime.fromtimestamp(timestamp)
buyRate = R.json()['buy'][-1][1]
print(date.strftime(format_date))
print(buyRate)
#ADDDING TO SQL SERVER
conn = odbc.connect("Driver={ODBC Driver 17 for SQL Server};"
'Server=LAPTOP-36NUUO53\SQLEXPRESS;'
'Database=test;'
'Trusted_connection=yes;')
cursor = conn.cursor()
cursor.execute('''
INSERT INTO Data_table (Time1,Price)
VALUES
('date',140),
('Date2' , 142)
''')
conn.commit()
cursor.execute('SELECT * FROM Data_table')
for i in cursor:
print(i)
How do i pass the variables date and buy rate to the table instead of putting in values liek i did (i put in'date' , 140 for example but i want to pass variables not specific values)
You'll need to check the driver version that you're using, but what you're looking for is the concept of bind variables. I'd suggest you look into the concept of fast_executemany as well - that should help speed things up. I've edited your code to show how bind variables typically work (using the (?, ?) SQL syntax), but there are other formats out there.
def LiraRateApiCall():
R = requests.get(url)
timestamp = R.json()['buy'][-1][0]/1000
format_date = '%d/%m/%y'
date = datetime.fromtimestamp(timestamp)
buyRate = R.json()['buy'][-1][1]
print(date.strftime(format_date))
print(buyRate)
#ADDDING TO SQL SERVER
conn = odbc.connect("Driver={ODBC Driver 17 for SQL Server};"
'Server=LAPTOP-36NUUO53\SQLEXPRESS;'
'Database=test;'
'Trusted_connection=yes;')
cursor = conn.cursor()
#Setup data
data = [('date',140), ('Date2' , 142)]
#Use executemany since we have a list
cursor.executemany('''
INSERT INTO Data_table (Time1,Price)
VALUES (?, ?)
''', data)
conn.commit()
cursor.execute('SELECT * FROM Data_table')
for i in cursor:
print(i)
I dont understand at all your question
If you want to pass the variables:
insert_sql = 'INSERT INTO Data_table (Time1,Price) VALUES (' + date + ',' + str(buyRate) + ')'
cursor.execute(insert_sql)
If you want to do dynamic Insert:
You can only insert knowing the values or by inserting with a select
INSERT INTO table
SELECT * FROM tableAux
WHERE condition;
That or you could iterate through the fields you have in a table, extract them and compare it to your variables to do a dynamic insert.
With this select you can extract the columns.
SELECT *
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = N'table1'
I want to insert a new row in my table by using the python-mariadb connector. For that I prefer to use the SET clause.
For some reason it does work if I only want to save ints (i.e y=2), but when I use a string, the following error occurs
Unknown column 'myString' in 'field list'
It seems it thinks the content of the string is a column name? Any idea how to fix that (I can do it with INSERT INTO ... VALUES ..., but I want to use the SET clause here). From my understanding, it should save both an int and a str without throwing an error
Thank you.
See the code example below
def myfunction():
x = 1
y ='myString'
db = connect_db()
cur = db.cursor()
sql = "INSERT INTO Table SET col1={}, col2={}"
cur.execute(sql.format(x, y))
db.commit()
db.close()
return
Here the MariaDB Connector, but this should be fine as it works for other db functions.
import mariadb
def connect_db():
db = mariadb.connect(
user="user",
password="123",
host="localhost",
port=3306,
database="DB"
)
db.autocommit = False
return db
you are not using right syntax for insert
sql = "INSERT INTO Table (col1,col2) values({}, {})"
but if you want to update an existing row:
sql = "UPDATE Table SET col1={}, col2={} WHERE id = {}"
and probably you need a where clause
The code in question produces the SQL statement:
INSERT INTO Table SET col1=1, col2=myString;
This is incorrect syntax, and strings must be in single-quotes:
INSERT INTO Table (col1, col2) VALUES (1, 'myString');
def myfunction():
x = 1
y ='myString'
db = connect_db()
cur = db.cursor()
sql = "INSERT INTO Table (col1, COL2) VALUES ({}, '{}')"
cur.execute(sql.format(x, y))
db.commit()
db.close()
return
But the above is fragile. Don't use string building methods to create SQL statements, it is much better to use parameter binding.
def myfunction():
x = 1
y ='myString'
db = connect_db()
cur = db.cursor()
sql = "INSERT INTO Table (col1, col2) VALUES (?, ?)"
cur.execute(sql, (x, y))
db.commit()
db.close()
return
The MariaDB connector documentation explains these things.
Retrieving Data
Once you have the initial code in place you can start working with the data. The first thing you should do is try to
retrieve information from the database. Here is code for a query
against the employees database:
cur.execute(
"SELECT first_name,last_name FROM employees WHERE first_name=?",
(some_name,))
MariaDB Connector/Python uses prepared statements, sanitizing and inserting the values from the tuple into the position
of the question marks (?). This is safer than inserting through
f-strings or format specifiers when working with user provided
information.
The query results are stored in a list in the cursor object. To view
the results, you can loop over the cursor.
Adding Data
Using the same execute() method with an INSERT statement, you can add rows to the table.
cursor.execute(
"INSERT INTO employees (first_name,last_name) VALUES (?, ?)",
(first_name, last_name))
I am trying to create a method in python insert records into a table passing in a list of column names, and an associated list of records.
I was able to set it up where the column names populated dynamically via a for loop, but I can't figure out how to do the same thing with values because the psycopg2.executemany function relies on having %s's as placeholders.
Is it possible to have the number of %s's in the string populate dynamically via a loop? Is there another way to do this?
def load_table(dbname,table_name,fields,records):
try:
#Variable Qty Column Loop
sql_fields = []
for i in fields:
i = sql.Identifier(i)
sql_fields.append(i)
#Need similar loop to replace %s values
#Replace (%s,%s,%s) ???
#.....
#.....
sql_values = []
for i in fields:
sql_values.append('%s')
print(sql_values)
flist = sql.SQL(',').join(sql_fields)
connection, cursor = create_connection(dbname)
insert_query = sql.SQL('INSERT INTO {table_name} ({fields}) VALUES (%s,%s,%s)').format(
table_name = sql.Identifier(table_name),
fields = flist,
cursor.executemany(insert_query,records)
print('Records Loaded Successfully')
except (Exception,psycopg2.Error) as error:
print("Failed to insert record into table {error}".format(error = error))
finally:
# closing database connection.
if (connection):
close_connection(connection,cursor)
You can use sql.Placeholder, to populate the sql statement with the amount of %s-placeholders you need:
def load_table(dbname,table_name,fields,records):
con, cur = create_connection('foo')
query = sql.SQL("insert into {} ({}) values ({})").format(
sql.Identifier(table_name),
sql.SQL(', ').join(map(sql.Identifier, fields)),
sql.SQL(', ').join(sql.Placeholder() * len(fields)))
print(query.as_string(con))
if __name__ == '__main__':
dbname = '...'
table_name = 'messages'
fields = ['user_id', 'message_type', 'message_title']
records = [['12345', 'json', 'my first message'], ]
load_table(dbname,table_name,fields,records)
Output:
insert into "messages" ("user_id", "message_type", "message_title") values (%s, %s, %s)
I have this situation where I created a method that will insert rows in database. I provide to that method columns, values and table name.
COLUMNS = [['NAME','SURNAME','AGE'],['SURNAME','NAME','AGE']]
VALUES = [['John','Doe',56],['Doe','John',56]]
TABLE = 'people'
This is how I would like to pass but it doesn't work:
db = DB_CONN.MSSQL() #method for connecting to MS SQL or ORACLE etc.
cursor = db.cursor()
sql = "insert into %s (?) VALUES(?)" % TABLE
cursor.executemany([sql,[COLUMNS[0],VALUES[0]],[COLUMNS[1],VALUES[1]]])
db.commit()
This is how it will pass query but problem is that I must have predefined column names and that's not good because what if the other list has different column sort? Than the name will be in surname and surname in name.
db = DB_CONN.MSSQL() #method for connecting to MS SQL or ORACLE etc.
cursor = db.cursor()
sql = 'insert into %s (NAME,SURNAME,AGE) VALUES (?,?,?)'
cursor.executemany(sql,[['John','Doe',56],['Doe','John',56]])
db.commit()
I hope I explained it clearly enough.
Ps. COLUMNS and VALUES are extracted from json dictionary
[{'NAME':'John','SURNAME':'Doe','AGE':56...},{'SURNAME':'Doe','NAME':'John','AGE':77...}]
if that helps.
SOLUTION:
class INSERT(object):
def __init__(self):
self.BASE_COL = ''
def call(self):
GATHER_DATA = [{'NAME':'John','SURNAME':'Doe','AGE':56},{'SURNAME':'Doe','NAME':'John','AGE':77}]
self.BASE_COL = ''
TABLE = 'person'
#check dictionary keys
for DATA_EVAL in GATHER_DATA:
if self.BASE_COL == '': self.BASE_COL = DATA_EVAL.keys()
else:
if self.BASE_COL != DATA_EVAL.keys():
print ("columns in DATA_EVAL.keys() have different columns")
#send mail or insert to log or remove dict from list
exit(403)
#if everything goes well make an insert
columns = ','.join(self.BASE_COL)
sql = 'insert into %s (%s) VALUES (?,?,?)' % (TABLE, columns)
db = DB_CONN.MSSQL()
cursor = db.cursor()
cursor.executemany(sql, [DATA_EVAL.values() for DATA_EVAL in GATHER_DATA])
db.commit()
if __name__ == "__main__":
ins = INSERT()
ins.call()
You could take advantage of the non-random nature of key-value pair listing for python dictionaries.
You should check that all items in the json array of records have the same fields, otherwise you'll run into an exception in your query.
columns = ','.join(records[0].keys())
sql = 'insert into %s (%s) VALUES (?,?,?)' % (TABLE, columns)
cursor.executemany(sql,[record.values() for record in records])
References:
https://stackoverflow.com/a/835430/5189811