This question already has answers here:
Dynamic SQL WHERE clause generation
(3 answers)
How to use variables in SQL statement in Python?
(5 answers)
Python sqlite3 parameterized drop table
(1 answer)
Closed 4 years ago.
From my Python code (Flask application, actually), I need to execute sqlite query, of the following structure
SELECT some_column FROM My_table WHERE some_column=some_value;
Now, some_column recurs twice in the query, one way to execute it is:
cursor.execute('SELECT ? FROM Users WHERE ?=?;', (some_column, some_column, some_value))
Which is not very nice/Pythonic. Then I came up with:
cursor.execute('SELECT {0} FROM Users WHERE {0}=?;'.format(some_column), (some_value,))
Finally, I ended up using .format() all the way:
cursor.execute('SELECT {0} FROM Users WHERE {0}={1};'.format(some_column, some_value), ())
I am wondering if there is prettier and/or more Pythonic way to pass recurring arguments into sqlite's cursor.execute()?
First syntax is incorrect. In SQL parameterized queries can use parameters for values, not for table or column names.
Third form is bad because it hard codes a value in a query which is forbidden per best practices. It used to be common some times ago and was the cause for SQL injection security problems.
So the only possibility is second form: use string construction for table and column names, and parameters for values.
But anyway, your query is close to non sense: you ask the value of one single column when you fix that column value. For each selected row, the value will be some_value!
So, I assume that this was a simplified example of a more complex question, but without more context, I simply cannot imagine why you are asking this question and what is your real problem.
It's not a big improvement, but since you tagged Python 3, consider the f-string:
f"SELECT {col} FROM Users WHERE {col}={val}"
As Klaus D. points out in a comment, however, it's not best practice to allow values to be format strings, as in some cases it makes your code vulnerable to SQL injection.
It's a little less compact, but you can use a mix of f-string (for column names) and sqlite ? syntax to input values (basically a mashup of your first two examples):
params = (val,)
q = f"SELECT {col} FROM Users WHERE {col} = ?"
cursor.execute(q, params)
Related
I am wanting to run a Presto SQL query in a for loop so that the query will pull hourly data based on my date variables.
Example query is along the lines of:
x = datetime.strptime('12-10-22', '%d-%m-%y').date()
y = datetime.strptime('13-10-22', '%d-%m-%y').date()
for dt in rrule.rrule(rrule.HOURLY, dtstart=nextProcStart, until=nextProcEnd):
sql_query = "SELECT SUM(sales) FROM a WHERE date between x and y"
I will note I'm using the syntax of writing the SQL query as a variable so along the lines of:
sql_query = """ SELECT... FROM..."""
I have tried just adding the variables into the query but no luck. Unsure what steps will work.
I've also tried using .format(x,y) at the end of my SQL query but keep getting an error saying
KeyError: 'x'
Remember that your SQL statement is no more than a string, so you just need to know how to incorporate a variable into a string.
Try:
sql_query = "SELECT SUM(sales) FROM a WHERE date between {} and {}".format(x, y)
Read How do I put a variable’s value inside a string (interpolate it into the string)? for more info or alternative methods.
Hopefully this answers your immediate question above on how to incorporate variable into string and get your code, as is, to work. However, as #nbk, mentions in comment below, this method is NOT recommended as it is insecure.
Using concatenations in SQL statements like this does open the code up to injection attacks. Even if your database does not contain sensitive information, it is bad practice.
Prepared statements have many advantages, not least of all that they are more secure and more efficient. I would certainly invest some time in researching and understanding SQL prepared statements.
This question already has answers here:
How to use variables in SQL statement in Python?
(5 answers)
Closed 2 months ago.
def update_inv_quant():
new_quant = int(input("Enter the updated quantity in stock: "))
Hello! I'm wondering how to insert a user variable into an sql statement so that a record is updated to said variable. Also, it'd be really helpful if you could also help me figure out how to print records of the database into the actual python console. Thank you!
I tried doing soemthing like ("INSERT INTO Inv(ItemName) Value {user_iname)") but i'm not surprised it didnt work
It would have been more helpful if you specified an actual database.
First method (Bad)
The usual way (which is highly discouraged as Graybeard said in the comments) is using python's f-string. You can google what it is and how to use it more in-depth.
but basically, say you have two variables user_id = 1 and user_name = 'fish', f-string turns something like f"INSERT INTO mytable(id, name) values({user_id},'{user_name}')" into the string INSERT INTO mytable(id,name) values(1,'fish').
As we mentioned before, this causes something called SQL injection. There are many good youtube videos that demonstrate what that is and why it's dangerous.
Second method
The second method is dependent on what database you are using. For example, in Psycopg2 (Driver for PostgreSQL database), the cursor.execute method uses the following syntax to pass variables cur.execute('SELECT id FROM users WHERE cookie_id = %s',(cookieid,)), notice that the variables are passed in a tuple as a second argument.
All databases use similar methods, with minor differences. For example, I believe SQLite3 uses ? instead of psycopg2's %s. That's why I said that specifying the actual database would have been more helpful.
Fetching records
I am most familiar with PostgreSQL and psycopg2, so you will have to read the docs of your database of choice.
To fetch records, you send the query with cursor.execute() like we said before, and then call cursor.fetchone() which returns a single row, or cursor.fetchall() which returns all rows in an iterable that you can directly print.
Execute didn't update the database?
Statements executing from drivers are transactional, which is a whole topic by itself that I am sure will find people on the internet who can explain it better than I can. To keep things short, for the statement to physically change the database, you call connection.commit() after cursor.execute()
So finally to answer both of your questions, read the documentation of the database's driver and look for the execute method.
This is what I do (which is for sqlite3 and would be similar for other SQL type databases):
Assuming that you have connected to the database and the table exists (otherwise you need to create the table). For the purpose of the example, i have used a table called trades.
new_quant = 1000
# insert one record (row)
command = f"""INSERT INTO trades VALUES (
'some_ticker', {new_quant}, other_values, ...
) """
cur.execute(command)
con.commit()
print('trade inserted !!')
You can then wrap the above into your function accordingly.
This question already has an answer here:
How to get variable length placeholders in a Python call to SQLite3
(1 answer)
Closed 7 years ago.
When there are only a few parameters I can do this
cur.execute(INSERT INTO table (col1,col2) VALUES (?,?), (1,2))
but what if I have like 50 parameters? I don't want to write 50 question marks in that case. Is there a way to do something like the following?
cur.execute(INSERT INTO table (listOfColumns) VALUES tuple(['?']*len(listOfColumns)), (listOfValues))
Yes, you just build the SQL statement dynamically:
sql = ('INSERT INTO table ({}) VALUES ({})'.format(','.join(listOfColumns),
','.join('?'*len(listOfColumns))))
cur.execute(sql, listOfValues)
Note that this assumes the list of columns was generated locally and not tainted by user input. If the list of columns could be tainted you need to check it pretty carefully to ensure that it only contains valid column names before inserting it into SQL code.
This question already has answers here:
Python MySQLdb string substitution without added quotations
(2 answers)
Closed 9 years ago.
In Python I'm passing in two variables to MySQL, and the second variable is referenced as {0} and works correctly.
How can I make the first parameter do the same thing ? If '{0}' is the second variable. What is the First ? dbname is incorrect, I need to show dbname's value in this string ?
def checkTableExists(dbname,tablename):
sql2 = """SELECT COUNT(*) FROM information_schema.tables
WHERE table_schema = " dbname "
AND table_name = '{0}' """.format(tablename.replace('\'', '\'\''))
A better way to organize your function to create the query might be:
def checkTableExists(dbname, tablename):
query_args = {"table_name":tablename, "database_name":dbname}
exist_query = """
SELECT EXISTS (
SELECT *
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME = '%(table_name)s'
AND TABLE_SCHEMA = '%(database_name)s') AS `Exist`
"""%(query_args)
# Do stuff with the `Exist` column of whatever your db connection returns.
# Should be 1 if True. Might be good to just return bool(...).
This is what EXISTS is for, so you don't need to do a "hack" and check yourself if the COUNT within INFORMATION_SCHEMA.TABLES is greater than 0.
As #jgranger commented on the OP, if you are using MySQLdb, you can follow that link and let it do the argument substitution. But there are many other flavors of interaction with pyodbc, so string-formatted queries will never go away (I wish they would).
Even Better
Write a stored procedure in SQL that checks if tables exist as a function of the database name and schema name, and then do nothing but pass arguments to this stored procedure. Then any component of the system (Python or otherwise) can expect that stored procedure's interface for existence checking.
If the existence-check logic needs to change (say in your application it suddenly matters that a table has more than 0 rows rather than merely having been created with a CREATE TABLE command, or vice versa), then you can change the internals of the stored procedure without needing to go to every downstream location that had embedded string SQL queries and change the logic. It helps add modularity and encapsulation to the system's interaction with the databases.
I have a very large db that I am working with, and I need to know how to select a large set of id's which doesn't have any real pattern to them. This is segment of code I have so far:
longIdList = [1, 3, 5 ,8 ....................................]
for id in longIdList
sql = "select * from Table where id = %s" %id
result = cursor.execute(sql)
print result.fetchone()
I was thinking, That there must be a quicker way of doing this... I mean my script needs to search through a db that has over 4 million id's. Is there a way that I can use a select command to grab them all in one shot. could I use the where statement with a list of id's? Thanks
Yes, you can use SQL's IN() predicate to compare a column to a set of values. This is standard SQL and it's supported by every SQL database.
There may be a practical limit to the number of values you can put in an IN() predicate before it becomes too inefficient or simply exceeds a length limit on SQL queries. The largest practical list of values depends on what database you use (in Oracle it's 1000, MS SQL Server it's around 2000). My feeling is that if your list exceeds a few dozen values, I'd seek another solution.
For example, #ngroot suggests using a temp table in his answer. For analysis of this solution, see this blog by StackOverflow regular #Quassnoi: Passing parameters in MySQL: IN list vs. temporary table.
Parameterizing a list of values into an SQL query a safe way can be tricky. You should be mindful of the risk of SQL injection.
Also see this popular question on Stack Overflow: Parameterizing a SQL IN clause?
You can use IN to look for multiple items simultaneously:
SELECT * FROM Table WHERE id IN (x, y, z, ...)
So maybe something like:
sql = "select * from Table where id in (%s)" % (', '.join(str(id) for id in longIdList))
Serialize the list in some fashion (comma-separated or XML would be reasonable choices), then have a stored procedure on the other side that will deserialize the list into a temp table. You can then do an INNER JOIN against the temp table.