Trying to self teach Python and Sqlite and my head is spinning. How does one 'clean up' the output of a query to get rid of all the brackets, commas, etc... from the result. Would also like to .title() the 2nd column. For example:
def get_all_bdays(self):
print("\n" * 100)
print("The birthdays we know about are: ")
self.c.execute('SELECT * FROM birthdays')
for row in self.c.fetchall():
print(row)
Results in the following output:
The birthdays we know about are:
(1, 'joe smoe', '12-12-1212')
How does one go about reformatting that mess to something like:
The birthdays we know about are:
1. Joe Smoe 12-12-1212
My end goal is to create an inventory system for my small business that my employees can use to find where backstock items are located in my storeroom. Thinking about using Flask for something like that, but I'm a long way off from that point in time.
Each row is a tuple with three values: the number, name, and birthday. print(row) is outputting the tuple, with all its parentheses and quotes, not any formatted version.
In Python, you can deconstruct the tuple and assign the parts of it to different variables, then format using Python's syntax for printf-like formatting:
for row in self.c.fetchall():
number, name, date = row
print("%d. %s on %s" % (number, name.title(), date))
or even:
for number, name, date in self.c.fetchall:
print("%d. %s on %s" % (number, name.title(), date))
When you print(row) you are getting the Python representation of row, which includes the quotes and commas and such. What you want to do is to str.format the data into whatever shape you like:
fmt = "{0}. {1}, on {2}"
for row in self.c.fetchall():
print(fmt.format(*row))
Related
The overarching question here is how to get a multirow REPLACE INTO statement that works with None in the format "REPLACE INTO ... VALUES (...), (...).
From https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-executemany.html we have this example where executemany(stmt, params) for INSERT statements ONLY forms the multiple row format:
INSERT INTO employees (first_name, hire_date) VALUES ('Jane', '2005-02-12'), ('Joe', '2006-05-23'), ('John', '2010-10-03')
But for all other statement types, it creates one query per tuple in params. For performance reasons, we want to bundle a REPLACE INTO in this multirow format.
The field list looks like this:
child_item_count, name, type_default, parent_id, version, agent_id, folder_id
and some of them are permitted to be NULL.
Originally, I tried to just build a statement string with all of the tuples comma added to the operational part of the query. Given list_of_tuples looks like [(None,'a string',8,'UUID',190L,'UUID','UUID'),...]:
insert_query = "REPLACE INTO %s ( %s ) VALUES {} " % (table, column_names)
values = ', '.join(map(str, list_of_tuples))
sql = insert_query.format(values)
db_cursor.execute(sql)
but I got:
Error writing to database: OperationalError(1054, "Unknown column 'None' in 'field list'")
I've also tried just shipping the list to execute() as in db_cursor.execute(insert_query, list_of_tuples) and that doesn't work, either. That results in "TypeError('not all arguments converted during string formatting',)"
Warning: Your code contains the possibility of SQL Injection.
The issue is pretty simple:
The map(a, b) function will run the a(el) for each element in b.
In your case, it will get every tuple on the list and convert it to a string, therefore a given tuple (None, 4, 'b') will turn into (None, 4, 'b') - and None is not a valid keyword on MySQL.
The best way to fix this is to rely on the execute command to convert the values correctly, making it sql injection free:
import itertools
columns_count = 10 # Change this value according to your columns count
column_values = '(' + ', '.join(['%s'] * columns_count) + ')'
values = ', '.join([column_values]*len(list_of_tuples))
# (...)
insert_query = insert_query.format(values)
db_cursor.execute(insert_query, list(itertools.chain.from_iterable(list_of_tuples)))
Although there is the second option (Bellow), it would make your code vulnerable to SQL Injection. So don't use it.
Simply to convert the values directly, making the necessary adjustments (In this specific scenario, it is just changing None to NULL):
values = ', '.join('('+', '.join(map(lambda e: str(e) if e else "NULL", t))+')' for t in list_of_tuples)
Trying to learn Sqlite and I'm not sure I understand why I can't get this code to work:
def get_bday(self):
name = self.input_name()
self.c.execute('SELECT * FROM birthdays WHERE name =?', name)
for row in self.c.fetchall():
print(row)
The name variable is being returned from another method. For this example, I am using "joe smoe" without the quotes as the name variable to perform the query with. When I run the above code I get:
self.c.execute('SELECT * FROM birthdays WHERE name =?', name)
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 8 supplied.
The word "joe smoe" is 8 bindings long if you count the space. But I have no idea what that means. I assumed I could simply pass a variable right to Sqlite just as easily as I pass variables around in Python but that doesn't appear to be the case. I think it has something to do with my very poor understanding of tuples.
SQLite is currently thinking you want to query each individual letter of 'joe smoe'.
All you have to do to avoid this is put name in a container of some kind: a tuple or a list for example:
def get_bday(self):
name = self.input_name()
self.c.execute('SELECT * FROM birthdays WHERE name =?', (name,))
# ^ ^^
for row in self.c.fetchall():
print(row)
i'm practicing on Python and trying to create a class that helps performing database operations, but when inserting to a database here's the code :
def Insert(self, **kwargs):
self.__query.execute("INSERT INTO {} ({}) VALUES ({})".format(self.table, ", ".join(kwargs.keys()), ", ".join(str(v) for v in kwargs.values())))
self.__db.commit()
When i ran this code for testing:
MyTable.Insert(id=3, name="jack", age=23)
I got this error :
sqlite3.OperationalError: no such column: jack
When i replaced the execute command with print i got this :
INSERT INTO testTbl111 (id, name, age) VALUES (3, jack, 23)
I guess jack must be surrounded by quotations.
My question: is how to surround jack with quotation while doing ", ".join(str(v) for v in kwargs.values()) ?
You don't want to try to escape value parameters yourself, instead you want to build the insert query and put placeholders (? works for sqlite3) for values - something like:
query = 'INSERT INTO {} ({}) VALUES({})'.format(self.table, ', '.join(kwargs), ','.join(['?'] * len(kwargs)))
Then, use the second method of execute (either on the db object or cursor object) to pass in the values to be substituted - these will automatically be correctly escaped for the database.
self.__db.execute(query, list(kwargs.values()))
I'm having a bit of trouble loading an CSV file into a mysql database. Here's my code:
for q in csvReader:
name, price, LastUpdate, today = q
co.execute("""INSERT INTO fundata (name, price, LastUpdate) VALUES(name, price, LastUpdate);""",q)
I get an error saying TypeError: not all arguments converted during string formatting.
The name column is a string, price is a float, and LastUpdate is a date. I read a bit and saw some scripts that wrapped the values in %(value)s and %(value)d (in my case instead of d I use f) but then I get a different error:
TypeError: format requires a mapping
Can anyone help show me what I am doing wrong?
Thank you!
If I recall correctly, you should use %s with MySQLdb in query to denote positions you want the argument tuple elements to be formatted. This is different from usual ? placeholders used in most other implementations.
for q in csvReader:
name, price, LastUpdate, today = q
co.execute("INSERT INTO fundata (name, price, LastUpdate) VALUES(%s, %s, %s);",q)
EDIT: Here is also an example of inserting multiple rows at once, that is more efficient than inserting them one by one. From MySQLdb User's Guide:
c.executemany(
"""INSERT INTO breakfast (name, spam, eggs, sausage, price)
VALUES (%s, %s, %s, %s, %s)""",
[
("Spam and Sausage Lover's Plate", 5, 1, 8, 7.95 ),
("Not So Much Spam Plate", 3, 2, 0, 3.95 ),
("Don't Wany ANY SPAM! Plate", 0, 4, 3, 5.95 )
] )
Here we are inserting three rows of five values. Notice that there is
a mix of types (strings, ints, floats) though we still only use %s.
And also note that we only included format strings for one row.
MySQLdb picks those out and duplicates them for each row.
From the error message, the execute() method is substituting your parameters into your SQL statement using %. But you haven't indicated where any of them go, so none of your parameters are being used, they are all left over, and therefore you get the message about having some left over (hey, all is some!). Read the documentation for your database driver to find out what it wants you to use as a substitution token; probably %s.
format requires mapping is because you are setting a name to the string replacement tokens and a dictionary is expected.
if you left them as %s %d %f etc. with out the parenthesis, it will take the arguments in order from a list or tuple (q)
for q in csvReader:
co.execute("""INSERT INTO fundata (name, price, LastUpdate) VALUES(%s, %f, %s);""",q[:-1])
I have a large SQLite database with a mix of text and lots of other columns var1 ... var 50. Most of these are numeric, though some are text based.
I am trying to extract data from the database, process it in python and write it back - I need to do this for all rows in the db.
So far, the below sort of works:
# get row using select and process
fields = (','.join(keys)) # "var1, var2, var3 ... var50"
results = ','.join([results[key] for key in keys]) # "value for var1, ... value for var50"
cur.execute('INSERT OR REPLACE INTO results (id, %s) VALUES (%s, %s);' %(fields, id, results))
This however, nulls the columns that I don't explicitly add back. I can fix this by re-writing the code, but this feels quite messy, as I would have to surround with quotes using string concatenation and rewrite data that was there to begin with (i.e. the columns I didn't change).
Apparently the way to run updates on rows is something like this:
update table set var1 = 4, var2 = 5, var3="some text" where id = 666;
Presumably the way for me would be to run map , and add the = signs somehow (not sure how), but how would I quote all of the results appropriately (Since I would have to quote the text fields, and they might contain quotes within them too .. )?
I'm a bit confused. Any pointers would be very helpful.
Thanks!
As others have stressed, use parametrized arguments. Here is an example of how you might construct the SQL statement when it has a variable number of keys:
sql=('UPDATE results SET '
+ ', '.join(key+' = ?' for key in keys)
+ 'WHERE id = ?')
args = [results[key] for key in keys] + [id]
cur.execute(sql,args)
Use parameter substitution. It's more robust (and safer I think) than string formatting.
So if you did something like
query = 'UPDATE TABLE SET ' + ', '.join(str(f) + '=?,' for f in fields) + ';'
Or alternatively
query = 'UPDATE TABLE SET %s;' % (', '.join(str(f) + '=?,' for f in fields))
Or using new style formatting:
query = 'UPDATE TABLE SET {0};'.format(', '.join(str(f) + '=?,' for f in fields))
So the complete program would look something like this:
vals = {'var1': 'foo', 'var2': 3, 'var24':999}
fields = vals.keys()
results = vals.values()
query = 'UPDATE TABLE SET {0};'.format(', '.join(str(f) + '=?,' for f in fields))
conn.execute(query, results)
And that should work - and I presume do what you want it to.
You don't have to care about things like quotations etc, and in fact you shouldn't. If you do it like this, it's not only more convenient but also takes care of security issues known as sql injections:
sql = "update table set var1=%s, var2=%s, var3=%s where id=666"
cursor.execute(sql, (4, 5, "some text"))
the key point here ist that the sql and the values in the second statement aren't separated by a "%", but by a "," - this is not a string manipulation, but instead you pass two arguments to the execute function, the actual sql and the values. Each %s is replaced by a value from the value tuple. the database driver then knows how to take care of the individual types of the values.
the insert statement can be rewritten the same way, although I'm not sure and currently can't test whether you can also replace field names that way (the first %s in your insert-sql statement)
so to come back to your overall problem, you can loop over your values and dynamically add ", var%d=%%s" % i for your i-th variable while adding the actual value to a list at the same time