I'm creating SQL commands from given input:
def do_add(self, table, full_sql_command="INSERT INTO {} VALUES ({})"):
""" Add a column to a specified table name """
add_command = raw_input("INSERT INTO {} VALUES ".format(table))
self.__create_sql_command(full_sql_command, table, add_command.split(" "))
def __create_sql_command(self, full, tablename, addable):
print full.format(tablename, ', '.join(addable))
What I need this to output is INSERT INTO <table-name> VALUES ('<val-1>', '<val-2>', '<etc..>')
As of right now I get the following output:
INSERT INTO inventory VALUES test test test
# <= INSERT INTO inventory VALUES (test, test, test)
How can I get the quotes around the values to add?
Instead of wrapping your arguments in quotes, use a parametrized query. It will in effect add the quotes as needed, transparently and with no chance of problems due to e.g. quotes in the input itself, either intentional (SQL injection attacks) or accidental.
This is an example of a parametrized query.
cursor.execute("INSERT INTO table VALUES (?, ?, ?)", (var1, var2, var3))
As you can see you need to pass two separate arguments to execute(), the query template and the values, so it will require some rejigging of your code.
Note: Different database modules accept different placeholders for the parametrized query. sqlite3, for example, accepts ? as above, but also allows named parameters (passed in a dictionary instead of a tuple):
cursor.execute("INSERT INTO table VALUES (:who, :age)", {"who":"Bill", "age": 33})
I am no SQL guy but if your enter table data separated by a space then a little for loop will do your job,
Say:
l=raw_input('Enter Table contents separated by space.').split(' ')
l1='('
for i in l:l1=l1+i+', '
l1=l1[:len(l1-3)] #for removing extra comma and space left after loop
l1=l1+')'
Then use l1 where ever you want. You can use private attributes or something, for more protection!
If anyone is curious, this is what I ended up doing:
Never do this in production it is extremely vulnerable to SQL injection
def __create_sql_command(self, full, tablename, addable):
return full.format(tablename, ', '.join(map(lambda x: "'" + x + "'", addable)))
#<= CREATE TABLE test ('test1', 'test2', 'test3')
Related
I'm looking for a way to implement alternating SQL queries - i.e. a function that allows me to filter entries based on different columns. Take the following example:
el=[["a","b",1],["a","b",3]]
def save_sql(foo):
with sqlite3.connect("fn.db") as db:
cur=db.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS et"
"(var1 VARCHAR, var2 VARCHAR, var3 INT)")
cur.executemany("INSERT INTO et VALUES "
"(?,?,?)", foo)
db.commit()
def load_sql(v1,v2,v3):
with sqlite3.connect("fn.db") as db:
cur=db.cursor()
cur.execute("SELECT * FROM et WHERE var1=? AND var2=? AND var3=?", (v1,v2,v3))
return cur.fetchall()
save_sql(el)
Now if I were to use load_sql("a","b",1), it would work. But assume I want to only query for the first and third column, i.e. load_sql("a",None,1) (the None is just intended as a placeholder) or only the last column load_sql(None,None,5), this wouldn't work.
This could of course be done with if statements checking which variables were supplied in the function call, but in tables with larger amounts of columns, this might get messy.
Is there a good way to do this?
What if load_sql() would accept an arbitrary number of keyword arguments, where keyword argument names would correspond to column names. Something along these lines:
def load_sql(**values):
with sqlite3.connect("fn.db") as db:
cur = db.cursor()
query = "SELECT * FROM et"
conditions = [f"{column_name} = :{column_name}" for column_name in values]
if conditions:
query = query + " WHERE " + " AND ".join(conditions)
cur.execute(query, values)
return cur.fetchall()
Note that here we trust keyword argument names to be valid and existing column names (and string-format them into the query) which may potentially be used as an SQL injection attack vector.
As a side note, I cannot stop but think that this feels like a reinventing-the-wheel step towards an actual ORM. Look into lightweight PonyORM or Peewee abstraction layers between Python and a database.
It will inevitably get messy if you want your SQL statements to remain sanitized/safe, but as long as you control your function signature it can remain reasonably safe, e.g.:
def load_sql(var1, var2, var3):
fields = dict(field for field in locals().items() if field[1] is not None)
query = "SELECT * FROM et"
if fields: # if at least one field is not None:
query += " WHERE " + " AND ".join((k + "=?" for k in fields.keys()))
with sqlite3.connect("fn.db") as db:
cur = db.cursor()
cur.execute(query, fields.values())
return cur.fetchall()
You can replace the function signature with load_sql(**kwargs) and then use kwargs.items() instead of locals.items() so that you can pass arbitrary column names, but that can be very dangerous and is certainly not recommended.
The overarching question here is how to get a multirow REPLACE INTO statement that works with None in the format "REPLACE INTO ... VALUES (...), (...).
From https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-executemany.html we have this example where executemany(stmt, params) for INSERT statements ONLY forms the multiple row format:
INSERT INTO employees (first_name, hire_date) VALUES ('Jane', '2005-02-12'), ('Joe', '2006-05-23'), ('John', '2010-10-03')
But for all other statement types, it creates one query per tuple in params. For performance reasons, we want to bundle a REPLACE INTO in this multirow format.
The field list looks like this:
child_item_count, name, type_default, parent_id, version, agent_id, folder_id
and some of them are permitted to be NULL.
Originally, I tried to just build a statement string with all of the tuples comma added to the operational part of the query. Given list_of_tuples looks like [(None,'a string',8,'UUID',190L,'UUID','UUID'),...]:
insert_query = "REPLACE INTO %s ( %s ) VALUES {} " % (table, column_names)
values = ', '.join(map(str, list_of_tuples))
sql = insert_query.format(values)
db_cursor.execute(sql)
but I got:
Error writing to database: OperationalError(1054, "Unknown column 'None' in 'field list'")
I've also tried just shipping the list to execute() as in db_cursor.execute(insert_query, list_of_tuples) and that doesn't work, either. That results in "TypeError('not all arguments converted during string formatting',)"
Warning: Your code contains the possibility of SQL Injection.
The issue is pretty simple:
The map(a, b) function will run the a(el) for each element in b.
In your case, it will get every tuple on the list and convert it to a string, therefore a given tuple (None, 4, 'b') will turn into (None, 4, 'b') - and None is not a valid keyword on MySQL.
The best way to fix this is to rely on the execute command to convert the values correctly, making it sql injection free:
import itertools
columns_count = 10 # Change this value according to your columns count
column_values = '(' + ', '.join(['%s'] * columns_count) + ')'
values = ', '.join([column_values]*len(list_of_tuples))
# (...)
insert_query = insert_query.format(values)
db_cursor.execute(insert_query, list(itertools.chain.from_iterable(list_of_tuples)))
Although there is the second option (Bellow), it would make your code vulnerable to SQL Injection. So don't use it.
Simply to convert the values directly, making the necessary adjustments (In this specific scenario, it is just changing None to NULL):
values = ', '.join('('+', '.join(map(lambda e: str(e) if e else "NULL", t))+')' for t in list_of_tuples)
I have these two lists:
list1=['a','b','c']
list2=['1','2','3']
I am trying to insert these into a database with field names like so:
a | b | c | d | e
I am currently trying putting these lists as strings and then simply adding in the execute, e.g. cur.execute(insert,(strList1,strList2)) where strList1 and strList2 are just strings of list1 and list2 formed using:
strList1=''
for thing in list1:
strList1+=thing+','
strList1=strList1[:-1]
My current SQL statement is:
insert="""insert into tbl_name(%s) values(%s)"""
cur.execute(insert,(strList1,strList2))
I also have a follow up question: how could I ensure that say column a needed to be a primary key that on a duplicate entry it would update the other fields if they were blank?
Do not use %s in queries as this is a security risk. This is due to %s simply inserting the value into the string, meaning it can be a whole separate query all together.
Instead use "?" where you want the value to be, and add a second argument to execute in the form of a tuple like so
curs.execute("SELECT foo FROM bar WHERE foobar = ?",(some_value,))
Or in a slightly longer example
curs.execute("UPDATE foo SET bar = ? WHERE foobar = ?",(first_value,second_value))
Edit:
Hopefully i understood what you want correctly this time, sadly you cannot use "?" for tables so you are stuck with %s. I made a quick little test script.
import sqlite3
list1=['foo','bar','foobar'] #List of tables
list2=['First_value','second_value','Third_value'] #List of values
db_conn = sqlite3.connect("test.db") #I used sqlite to test it quickly
db_curs = db_conn.cursor()
for table in list1: #Create all the tables in the db
query = "CREATE TABLE IF NOT EXISTS %s(foo text, bar text,foobar text)" % table
db_curs.execute(query)
db_conn.commit()
for table in list1: #Insert all the values into all the tables
query = "INSERT INTO %s VALUES (?,?,?)" % table
db_curs.execute(query,tuple(list2))
db_conn.commit()
for table in list1: #Print all the values out to see if it worked
db_curs.execute("SELECT * FROM %s" % table)
fetchall = db_curs.fetchall()
for entry in fetchall:
print entry[0], entry[1],entry[2]
One thing you could do on those lists to make things easier...
list1=['a','b','c']
print ",".join(list1)
#a,b,c
Your insert looks good. Seems like a batch insert would be the only other option.
This is the way prepared statements work (simplified):
* The statement is send to the database with a list of parameters;
* The statement is retrieved from the statement cache or if not present, prepared and added to the statement cache;
* Parameters are applied;
* Statement is executed.
Statements have to be complete except that the parameters are replaced by %s (or ? or :parm depending on the language used). Parameters are your final numerical/string/date/etc values only. So labels or other parts can not be replaced.
In your case that means:
insert="""insert into tbl_name(%s) values(%s)"""
Should become something like:
insert="""insert into tbl_name(a,b,c) values(%s,%s,%s)"""
To use parameters, you must provide a "%s" (or %d, whatever) for each item. You can use two lists/tuples as follows:
insert="""insert into tbl_name (%s,%s,%s) values (%s, %s, %s);"""
strList1=('a','b','c')
strList2=(1,2,3)
curs.execute(insert % (strList1 + strList2))
*I'm using python3, (strList1,StrList2) doesn't work for me, but you might have slight differences.
I would like to use a dictionary to insert values into a table, how would I do this?
import sqlite3
db = sqlite3.connect('local.db')
cur = db.cursor()
cur.execute('DROP TABLE IF EXISTS Media')
cur.execute('''CREATE TABLE IF NOT EXISTS Media(
id INTEGER PRIMARY KEY, title TEXT,
type TEXT, genre TEXT,
onchapter INTEGER, chapters INTEGER,
status TEXT
)''')
values = {'title':'jack', 'type':None, 'genre':'Action', 'onchapter':None,'chapters':6,'status':'Ongoing'}
#What would I Replace x with to allow a
#dictionary to connect to the values?
cur.execute('INSERT INTO Media VALUES (NULL, x)'), values)
cur.execute('SELECT * FROM Media')
meida = cur.fetchone()
print meida
If you're trying to use a dict to specify both the column names and the values, you can't do that, at least not directly.
That's really inherent in SQL. If you don't specify the list of column names, you have to specify them in CREATE TABLE order—which you can't do with a dict, because a dict has no order. If you really wanted to, of course, you could use a collections.OrderedDict, make sure it's in the right order, and then just pass values.values(). But at that point, why not just have a list (or tuple) in the first place? If you're absolutely sure you've got all the values, in the right order, and you want to refer to them by order rather than by name, what you have is a list, not a dict.
And there's no way to bind column names (or table names, etc.) in SQL, just values.
You can, of course, generate the SQL statement dynamically. For example:
columns = ', '.join(values.keys())
placeholders = ', '.join('?' * len(values))
sql = 'INSERT INTO Media ({}) VALUES ({})'.format(columns, placeholders)
values = [int(x) if isinstance(x, bool) else x for x in values.values()]
cur.execute(sql, values)
However, this is almost always a bad idea. This really isn't much better than generating and execing dynamic Python code. And you've just lost all of the benefits of using placeholders in the first place—primarily protection from SQL injection attacks, but also less important things like faster compilation, better caching, etc. within the DB engine.
It's probably better to step back and look at this problem from a higher level. For example, maybe you didn't really want a static list of properties, but rather a name-value MediaProperties table? Or, alternatively, maybe you want some kind of document-based storage (whether that's a high-powered nosql system, or just a bunch of JSON or YAML objects stored in a shelve)?
An alternative using named placeholders:
columns = ', '.join(my_dict.keys())
placeholders = ':'+', :'.join(my_dict.keys())
query = 'INSERT INTO my_table (%s) VALUES (%s)' % (columns, placeholders)
print query
cur.execute(query, my_dict)
con.commit()
There is a solution for using dictionaries. First, the SQL statement
INSERT INTO Media VALUES (NULL, 'x');
would not work, as it assumes you are referring to all columns, in the order they are defined in the CREATE TABLE statement, as abarnert stated. (See SQLite INSERT.)
Once you have fixed it by specifying the columns, you can use named placeholders to insert data. The advantage of this is that is safely escapes key-characters, so you do not have to worry. From the Python sqlite-documentation:
values = {
'title':'jack', 'type':None, 'genre':'Action',
'onchapter':None,'chapters':6,'status':'Ongoing'
}
cur.execute(
'INSERT INTO Media (id, title, type, onchapter, chapters, status)
VALUES (:id, :title, :type, :onchapter, :chapters, :status);',
values
)
You could use named parameters:
cur.execute('INSERT INTO Media VALUES (NULL, :title, :type, :genre, :onchapter, :chapters, :status)', values)
This still depends on the column order in the INSERT statement (those : are only used as keys in the values dict) but it at least gets away from having to order the values on the python side, plus you can have other things in values that are ignored here; if you're pulling what's in the dict apart to store it in multiple tables, that can be useful.
If you still want to avoid duplicating the names, you could extract them from an sqlite3.Row result object, or from cur.description, after doing a dummy query; it may be saner to keep them around in python form near wherever you do your CREATE TABLE.
Here's a more generic way with the benefit of escaping:
# One way. If keys can be corrupted don't use.
sql = 'INSERT INTO demo ({}) VALUES ({})'.format(
','.join(my_dict.keys()),
','.join(['?']*len(my_dict)))
# Another, better way. Hardcoded w/ your keys.
sql = 'INSERT INTO demo ({}) VALUES ({})'.format(
','.join(my_keys),
','.join(['?']*len(my_dict)))
cur.execute(sql, tuple(my_dict.values()))
key_lst = ('status', 'title', 'chapters', 'onchapter', 'genre', 'type')
cur.execute('INSERT INTO Media (status,title,chapters,onchapter,genre,type) VALUES ' +
'(?,?,?,?,?,?);)',tuple(values[k] for k in key_lst))
Do your escaping right.
You probably also need a commit call in there someplace.
Super late to this, but figured I would add my own answer. Not an expert, but something I found that works.
There are issues with preserving order when using a dictionary, which other users have stated, but you could do the following:
# We're going to use a list of dictionaries, since that's what I'm having to use in my problem
input_list = [{'a' : 1 , 'b' : 2 , 'c' : 3} , {'a' : 14 , 'b' : '' , 'c' : 43}]
for i in input_list:
# I recommend putting this inside a function, this way if this
# Evaluates to None at the end of the loop, you can exit without doing an insert
if i :
input_dict = i
else:
input_dict = None
continue
# I am noting here that in my case, I know all columns will exist.
# If you're not sure, you'll have to get all possible columns first.
keylist = list(input_dict.keys())
vallist = list(input_dict.values())
query = 'INSERT INTO example (' +','.join( ['[' + i + ']' for i in keylist]) + ') VALUES (' + ','.join(['?' for i in vallist]) + ')'
items_to_insert = list(tuple(x.get(i , '') for i in keylist) for x in input_list)
# Making sure to preserve insert order.
conn = sqlite3.connect(':memory:')
cur = conn.cursor()
cur.executemany(query , items_to_insert)
conn.commit()
dictionary = {'id':123, 'name': 'Abc', 'address':'xyz'}
query = "insert into table_name " + str(tuple(dictionary.keys())) + " values" + str(tuple(dictionary.values())) + ";"
cursor.execute(query)
query becomes
insert into table_name ('id', 'name', 'address') values(123, 'Abc', 'xyz');
I was having the similar problem so I created a string first and then passed that string to execute command. It does take longer time to execute but mapping was perfect for me. Just a work around:
create_string = "INSERT INTO datapath_rtg( Sr_no"
for key in record_tab:
create_string = create_string+ " ," + str(key)
create_string = create_string+ ") VALUES("+ str(Sr_no)
for key in record_tab:
create_string = create_string+ " ," + str(record_tab[key])
create_string = create_string + ")"
cursor.execute(create_string)
By doing above thing I ensured that if my dict (record_tab) doesn't contain a particular field then the script wont throw out error and proper mapping can be done which is why I used dictionary at the first place.
I was having a similar problem and ended up with something not entirely unlike the following (Note - this is the OP's code with bits changed so that it works in the way they requested)-
import sqlite3
db = sqlite3.connect('local.db')
cur = db.cursor()
cur.execute('DROP TABLE IF EXISTS Media')
cur.execute('''CREATE TABLE IF NOT EXISTS Media(
id INTEGER PRIMARY KEY, title TEXT,
type TEXT, genre TEXT,
onchapter INTEGER, chapters INTEGER,
status TEXT
)''')
values = {'title':'jack', 'type':None, 'genre':'Action', 'onchapter':None,'chapters':6,'status':'Ongoing'}
#What would I Replace x with to allow a
#dictionary to connect to the values?
#cur.execute('INSERT INTO Media VALUES (NULL, x)'), values)
# Added code.
cur.execute('SELECT * FROM Media')
colnames = cur.description
list = [row[0] for row in cur.description]
new_list = [values[i] for i in list if i in values.keys()]
sql = "INSERT INTO Media VALUES ( NULL, "
qmarks = ', '.join('?' * len(values))
sql += qmarks + ")"
cur.execute(sql, new_list)
#db.commit() #<-Might be important.
cur.execute('SELECT * FROM Media')
media = cur.fetchone()
print (media)
I have a large SQLite database with a mix of text and lots of other columns var1 ... var 50. Most of these are numeric, though some are text based.
I am trying to extract data from the database, process it in python and write it back - I need to do this for all rows in the db.
So far, the below sort of works:
# get row using select and process
fields = (','.join(keys)) # "var1, var2, var3 ... var50"
results = ','.join([results[key] for key in keys]) # "value for var1, ... value for var50"
cur.execute('INSERT OR REPLACE INTO results (id, %s) VALUES (%s, %s);' %(fields, id, results))
This however, nulls the columns that I don't explicitly add back. I can fix this by re-writing the code, but this feels quite messy, as I would have to surround with quotes using string concatenation and rewrite data that was there to begin with (i.e. the columns I didn't change).
Apparently the way to run updates on rows is something like this:
update table set var1 = 4, var2 = 5, var3="some text" where id = 666;
Presumably the way for me would be to run map , and add the = signs somehow (not sure how), but how would I quote all of the results appropriately (Since I would have to quote the text fields, and they might contain quotes within them too .. )?
I'm a bit confused. Any pointers would be very helpful.
Thanks!
As others have stressed, use parametrized arguments. Here is an example of how you might construct the SQL statement when it has a variable number of keys:
sql=('UPDATE results SET '
+ ', '.join(key+' = ?' for key in keys)
+ 'WHERE id = ?')
args = [results[key] for key in keys] + [id]
cur.execute(sql,args)
Use parameter substitution. It's more robust (and safer I think) than string formatting.
So if you did something like
query = 'UPDATE TABLE SET ' + ', '.join(str(f) + '=?,' for f in fields) + ';'
Or alternatively
query = 'UPDATE TABLE SET %s;' % (', '.join(str(f) + '=?,' for f in fields))
Or using new style formatting:
query = 'UPDATE TABLE SET {0};'.format(', '.join(str(f) + '=?,' for f in fields))
So the complete program would look something like this:
vals = {'var1': 'foo', 'var2': 3, 'var24':999}
fields = vals.keys()
results = vals.values()
query = 'UPDATE TABLE SET {0};'.format(', '.join(str(f) + '=?,' for f in fields))
conn.execute(query, results)
And that should work - and I presume do what you want it to.
You don't have to care about things like quotations etc, and in fact you shouldn't. If you do it like this, it's not only more convenient but also takes care of security issues known as sql injections:
sql = "update table set var1=%s, var2=%s, var3=%s where id=666"
cursor.execute(sql, (4, 5, "some text"))
the key point here ist that the sql and the values in the second statement aren't separated by a "%", but by a "," - this is not a string manipulation, but instead you pass two arguments to the execute function, the actual sql and the values. Each %s is replaced by a value from the value tuple. the database driver then knows how to take care of the individual types of the values.
the insert statement can be rewritten the same way, although I'm not sure and currently can't test whether you can also replace field names that way (the first %s in your insert-sql statement)
so to come back to your overall problem, you can loop over your values and dynamically add ", var%d=%%s" % i for your i-th variable while adding the actual value to a list at the same time