We Keep Coding

How to fix syntax error with UPDATE in SQLite3 Python module

I'm using python module sqlite3 as per the below code:
# Enter the randomised data into the dictionary:
for square in cube:
cur.execute("UPDATE cubes SET ? = ? WHERE id = ?", (square, cube[square], session["current_cube_id"]))
con.commit()
Which results in the following error:
cur.execute("UPDATE cubes SET ? = ? WHERE id = ?", (square, cube[square], session["current_cube_id"]))
sqlite3.OperationalError: near "?": syntax error
I don't seem to have a problem with INSERT or SELECT queries, so I assume there is a specific syntax required to UPDATE. From the documentation, tutorials and other examples I can find this seems to be correct - can anyone please assist with what might be the syntax error?

You can't define table, column names, or SQL keywords, using bind variables (the SET ? =) in UPDATE cubes SET ? = ? WHERE id = ?
I am not sure why you feel you need to have a dynamic column name, rather than UPDATE cubes SET mycol = ? WHERE id = ? but you need to specify your column name differently.
You'd have the exact same problem with insert or delete if your target column names, in an insert, or your where condition column names in a delete, were getting specified with ? placeholders. I assume you did not do this so you did not get the error.
Be very careful if you decide to build your query string dynamically as in
myquery = f"UPDATE cubes SET {my_user_supplied_column_name} = ? WHERE id = ?"
cur.execute(myquery, (cube[square], session["current_cube_id"]))
That opens you to a large class of extremely serious vulnerabilities, the SQL Injections because the user may enter anything they want in my_user_supplied_column_name. Best to be very careful as it also has a reputational risk: a savvy prospective employer might for example reject your application if they saw this type of construct, unguarded, in your code because it is an extremely grave, frequent and well-known risk.

Querying SQLite Database for specific cell [duplicate]

This question already has answers here:
sqlite3.OperationalError: no such column - but I'm not asking for a column?
(2 answers)
Closed 2 years ago.
Forgive me if this is a basic question, I'm learning on my own and having some trouble. I have built a database with SQLite and am trying to write something that can display the 'description' of an entry when the name is entered into an entry box.
record_id = call_field.get()
# Query the database
c.execute("SELECT name, abbr, description FROM TQ_QUICKTEXT WHERE name =" + record_id)
records = c.fetchall()
# Loop through results
for record in records:
display.insert(1.0, record[2])
When I type the name of the entry that has been put into the database, an error is returned saying there is no column with that name. However, when I type the actual word 'name' into the entry box and run the function every single description entry is returned. If someone wouldn't mind pointing out where I've made mistakes it would be greatly appreciated. Thank you in advance.

The SELECT statement that is executed currently looks like this:
SELECT name, abbr, description FROM TQ_QUICKTEXT WHERE name = something (where something is the value of record_id)
Because there are no quotes around the value of record_id, it thinks it is a column name, not text. This is why when you try the name of a record, you get an error because that is not a column name, but name works, because it is the name of a column.
Adding quotes will fix the problem, but the database is vulnerable to SQL injection.
It is good security practise to parameterise SQL queries to prevent this. This is done by using ? in place of parameters and then passing a tuple of parameters to the execute function. This protects you from SQL injection.
After these changes, the c.execute statment should look like this:
c.execute("SELECT name, abbr, description FROM TQ_QUICKTEXT WHERE name = ?", (record_id,))

Too many server roundtrips w/ psycopg2

I am making a script, that should create a schema for each customer. I’m fetching all metadata from a database that defines how each customer’s schema should look like, and then create it. Everything is well defined, the types, names of tables, etc. A customer has many tables (fx, address, customers, contact, item, etc), and each table has the same metadata.
My procedure now:
get everything I need from the metadataDatabase.
In a for loop, create a table, and then Alter Table and add each metadata (This is done for each table).
Right now my script runs in about a minute for each customer, which I think is too slow. It has something to do with me having a loop, and in that loop, I’m altering each table.
I think that instead of me altering (which might be not so clever approach), I should do something like the following:
Note that this is just a stupid but valid example:
for table in tables:
con.execute("CREATE TABLE IF NOT EXISTS tester.%s (%s, %s);", (table, "last_seen date", "valid_from timestamp"))
But it gives me this error (it seems like it reads the table name as a string in a string..):
psycopg2.errors.SyntaxError: syntax error at or near "'billing'"
LINE 1: CREATE TABLE IF NOT EXISTS tester.'billing' ('last_seen da...

Consider creating tables with a serial type (i.e., autonumber) ID field and then use alter table for all other fields by using a combination of sql.Identifier for identifiers (schema names, table names, column names, function names, etc.) and regular format for data types which are not literals in SQL statement.
from psycopg2 import sql
# CREATE TABLE
query = """CREATE TABLE IF NOT EXISTS {shm}.{tbl} (ID serial)"""
cur.execute(sql.SQL(query).format(shm = sql.Identifier("tester"),
tbl = sql.Identifier("table")))
# ALTER TABLE
items = [("last_seen", "date"), ("valid_from", "timestamp")]
query = """ALTER TABLE {shm}.{tbl} ADD COLUMN {col} {typ}"""
for item in items:
# KEEP IDENTIFIER PLACEHOLDERS
final_query = query.format(shm="{shm}", tbl="{tbl}", col="{col}", typ=i[1])
cur.execute(sql.SQL(final_query).format(shm = sql.Identifier("tester"),
tbl = sql.Identifier("table"),
col = sql.Identifier(item[0]))
Alternatively, use str.join with list comprehension for one CREATE TABLE:
query = """CREATE TABLE IF NOT EXISTS {shm}.{tbl} (
"id" serial,
{vals}
)"""
items = [("last_seen", "date"), ("valid_from", "timestamp")]
val = ",\n ".join(["{{}} {typ}".format(typ=i[1]) for i in items])
# KEEP IDENTIFIER PLACEHOLDERS
pre_query = query.format(shm="{shm}", tbl="{tbl}", vals=val)
final_query = sql.SQL(pre_query).format(*[sql.Identifier(i[0]) for i in items],
shm = sql.Identifier("tester"),
tbl = sql.Identifier("table"))
cur.execute(final_query)
SQL (sent to database)
CREATE TABLE IF NOT EXISTS "tester"."table" (
"id" serial,
"last_seen" date,
"valid_from" timestamp
)

However, this becomes heavy as there are too many server roundtrips.
How many tables with how many columns are you creating that this is slow? Could you ssh to a machine closer to your server and run the python there?
I don't get that error. Rather, I get an SQL syntax error. A values list is for conveying data. But ALTER TABLE is not about data, it is about metadata. You can't use a values list there. You need the names of the columns and types in double quotes (or no quotes) rather than single quotes. And you can't have a comma between name and type. And you can't have parentheses around each pair. And each pair needs to be introduced with "ADD", you can't have it just once. You are using the wrong tool for the job. execute_batch is almost the right tool, except it will use single quotes rather than double quotes around the identifiers. Perhaps you could add a flag to it tell it to use quote_ident.
Not only is execute_values the wrong tool for the job, but I think python in general might be as well. Why not just load from a .sql file?

Python mysql using variable to select a certain field

Having a little tricky issue with python and mysql. To keep it simple, the following code returns whatever is in the variable 'field', which is a string. Such as 'username' or 'password'.
options = [field, userID]
entries = cursor.execute('select (?) from users where id=(?)', options).fetchall()
print(entries);
This code works correctly if I remove the first (?) and just use the actually name (like 'username') instead. Can anyone provide some input?

Your query is actually formed as:
select "field" from users where id="value"
which returns you a string "field" instead of the actual table field value.
You cannot parameterize column and table names (docs):
Parameter placeholders can only be used to insert column values. They
can not be used for other parts of SQL, such as table names,
statements, etc.
Use string formatting for that part:
options = [userID]
query = 'select {field} from users where id=(?)'.format(field=field)
cursor.execute(query, options).fetchall()
Related threads with some more explanations:
pysqlite: Placeholder substitution for column or table names?
Python MySQLdb: Query parameters as a named dictionary

Query syntax to select exactly one item for each category

class Category(models.Model):
pass
class Item(models.Model):
cat = models.ForeignKey(Category)
I want to select exactly one item for each category, which is the query syntax for do this?

Your question isn't entirely clear: since you didn't say otherwise, I'm going to assume that you don't care which item is selected for each category, just that you need any one. If that isn't the case, please update the question to clarify.
tl;dr version: there is no documented
way to explicitly use GROUP BY
statements in Django, except by using
a raw query. See the bottom for code to do so.
The problem is that in doing what you're looking for in SQL itself requires a bit of a hack. You can easily try this example with by entering sqlite3 :memory: at the command line:
CREATE TABLE category
(
id INT
);
CREATE TABLE item
(
id INT,
category_id INT
);
INSERT INTO category VALUES (1);
INSERT INTO category VALUES (2);
INSERT INTO category VALUES (3);
INSERT INTO item VALUES (1,1);
INSERT INTO item VALUES (2,2);
INSERT INTO item VALUES (3,3);
INSERT INTO item VALUES (4,1);
INSERT INTO item VALUES (5,2);
SELECT id, category_id, COUNT(category_id) FROM item GROUP BY category_id;
returns
4|1|2
5|2|2
3|3|1
Which is what you're looking for (one item id for each category id), albeit with an extraneous COUNT. The count (or some other aggregate function) is needed in order to apply the GROUP BY.
Note: this will ignore categories that don't contain any items, which seems like sensible behaviour.
Now the question becomes, how to do this in Django?
The obvious answer is to use Django's aggregation/annotation support, in particular, combining annotate with values as is recommend elsewhere to GROUP queries in Django.
Reading those posts, it would seem we could accomplish what we're looking for with
Item.objects.values('id').annotate(unneeded_count=Count('category_id'))
However this doesn't work. What Django does here is not just GROUP BY "category_id", but groups by all fields selected (ie GROUP BY "id", "category_id")1. I don't believe there is a way (in the public API, at least) to change this behaviour.
The solution is to fall back to raw SQL:
qs = Item.objects.raw('SELECT *, COUNT(category_id) FROM myapp_item GROUP BY category_id')
1: Note that you can inspect what queries Django is running with:
from django.db import connection
print connection.queries[-1]
Edit:
There are a number of other possible approaches, but most have (possibly severe) performance problems. Here are a couple:
1. Select an item from each category.
items = []
for c in Category.objects.all():
items.append(c.item_set[0])
This is a more clear and flexible approach, but has the obvious disadvantage of requiring many more database hits.
2. Use select_related
items = Item.objects.select_related()
and then do the grouping/filtering yourself (in Python).
Again, this is perhaps more clear than using raw SQL and only requires one query, but this one query could be very large (it will return all items and their categories) and doing the grouping/filtering yourself is probably less efficient than letting the database do it for you.