I want to pass dynamic parameter to the LIKE query using Fast API (Python) coming from the query parameter. I have tried a lot of ways but I don't know what I am doing wrong.
The query I need to use is :
SELECT Time, table_id, Text FROM tablename WHERE Text LIKE '%text%'
The python code for getting query parameter is :
def get_result(text : str):
con = connection.connect()
statement = 'SELECT Time, table_id, Text FROM tablename WHERE Text LIKE '%text%''
How will I pass the dynamic text parameter in this query because this gives the error saying "TypeError: not all arguments converted during string formatting"?
You cannot nest single quotes. Also, it's clearer to use a f-string for string formatting here. Try:
statement = f"SELECT Time, table_id, Text FROM tablename WHERE Text LIKE '%{text}%'"
NEVER substitute values directly in to a SQL string like that. EVER. Use parameterised queries / bind variables.
For example, depending on the driver/library you're using, you may be able to use...
con = connection.connect()
cur = con.cursor()
cur.execute("SELECT Time, table_id, Text FROM tablename WHERE Text LIKE %s", ('%' + text + '%',))
Related
I am aware that queries in Python can be parameterized using either ? or %s in execute query here or here
However I have some long query that would use some constant variable defined at the beginning of the query
Set #my_const = 'xyz';
select #my_const;
-- Query that use #my_const 40 times
select ... coalesce(field1, #my_const), case(.. then #my_const)...
I would like to do the least modif possible to the query from Mysql. So that instead of modifying the query to
pd.read_sql(select ... coalesce(field1, %s), case(.. then %s)... , [my_const, my_const, my_const, ..]
,I could write something along the line of the initial query. Upon trying the following, however, I am getting a TypeError: 'NoneType' object is not iterable
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
Any idea how to use the original variable defined in Mysql query ?
The reason
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
throws that exception is because all you are doing is setting null_value to '' and then selecting that '' - what exactly would you have expected that to give you? EDIT read_sql only seems to execute one query at a time, and as the first query returns no rows it results in that exception.
If you split them in to two calls to read_sql then it will in fact return you the value of your #null value in the second call. Due to this behaviour read_sql is clearly not a good way to do this. I strongly suggest you use one of my suggestions below.
Why are you wanting to set the variable in the SQL using '#' anyway?
You could try using the .format style of string formatting.
Like so:
query_str = "select ... coalesce(field1, {c}), case(.. then {c})...".format(c=my_const)
pd.read_sql(query_str)
Just remember that if you do it this way and your my_const is a user input then you will need to sanitize it manually to prevent SQL injection.
Another possibility is using a dict of params like so:
query_str = "select ... coalesce(field1, %(my_const)s, case(.. then %(my_const)s)..."
pd.read_sql(query_str, params={'my_const': const_value})
However this is dependent on which database driver you use.
From the pandas.read_sql docs:
Check your database driver documentation for which of the five syntax
styles, described in PEP 249’s paramstyle, is supported. Eg. for
psycopg2, uses %(name)s so use params={‘name’ : ‘value’}
Question: Is it possible to use a variable as your table name without having to use string constructors to do so?
Info:
I'm working on a project right now that catalogs data from a star simulation of mine. To do so I'm loading all the data into a sqlite database. It's working pretty well, but I've decided to add a lot more flexibility, efficiency, and usability to my db. I plan on later adding planetoids to the simulation, and wanted to have a table for each star. This way I wouldn't have to query a table of 20m some planetoids for the 1-4k in each solar system.
I've been told using string constructors is bad because it leaves me vulnerable to a SQL injection attack. While that isn't a big deal here as I'm the only person with access to these dbs, I would like to follow best practices. And also this way if I do a project with a similar situation where it is open to the public, I know what to do.
Currently I'm doing this:
cursor.execute("CREATE TABLE StarFrame"+self.name+" (etc etc)")
This works, but I would like to do something more like:
cursor.execute("CREATE TABLE StarFrame(?) (etc etc)",self.name)
though I understand that this would probably be impossible. though I would settle for something like
cursor.execute("CREATE TABLE (?) (etc etc)",self.name)
If this is not at all possible, I'll accept that answer, but if anyone knows a way to do this, do tell. :)
I'm coding in python.
Unfortunately, tables can't be the target of parameter substitution (I didn't find any definitive source, but I have seen it on a few web forums).
If you are worried about injection (you probably should be), you can write a function that cleans the string before passing it. Since you are looking for just a table name, you should be safe just accepting alphanumerics, stripping out all punctuation, such as )(][;, and whitespace. Basically, just keep A-Z a-z 0-9.
def scrub(table_name):
return ''.join( chr for chr in table_name if chr.isalnum() )
scrub('); drop tables --') # returns 'droptables'
For people searching for a way to make the table as a variable, I got this from another reply to same question here:
It said the following and it works. It's all quoted from mhawke:
You can't use parameter substitution for the table name. You need to add the table name to the query string yourself. Something like this:
query = 'SELECT * FROM {}'.format(table)
c.execute(query)
One thing to be mindful of is the source of the value for the table name. If that comes from an untrusted source, e.g. a user, then you need to validate the table name to avoid potential SQL injection attacks. One way might be to construct a parameterised query that looks up the table name from the DB catalogue:
import sqlite3
def exists_table(db, name):
query = "SELECT 1 FROM sqlite_master WHERE type='table' and name = ?"
return db.execute(query, (name,)).fetchone() is not None
I wouldn't separate the data into more than one table. If you create an index on the star column, you won't have any problem efficiently accessing the data.
Try with string formatting:
sql_cmd = '''CREATE TABLE {}(id, column1, column2, column2)'''.format(
'table_name')
db.execute(sql_cmd)
Replace 'table_name' with your desire.
To avoid hard-coding table names, I've used:
table = "sometable"
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS {} (
importantdate DATE,
somename VARCHAR,
)'''.format(table))
c.execute('''INSERT INTO {} VALUES (?, ?)'''.format(table),
(datetime.strftime(datetime.today(), "%Y-%m-%d"),
myname))
As has been said in the other answers, "tables can't be the target of parameter substitution" but if you find yourself in a bind where you have no option, here is a method of testing if the table name supplied is valid.
Note: I have made the table name a real pig in an attempt to cover all of the bases.
import sys
import sqlite3
def delim(s):
delims="\"'`"
use_delim = []
for d in delims:
if d not in s:
use_delim.append(d)
return use_delim
db_name = "some.db"
db = sqlite3.connect(db_name)
mycursor = db.cursor()
table = 'so""m ][ `etable'
delimiters = delim(table)
if len(delimiters) < 1:
print "The name of the database will not allow this!"
sys.exit()
use_delimiter = delimiters[0]
print "Using delimiter ", use_delimiter
mycursor.execute('SELECT name FROM sqlite_master where (name = ?)', [table])
row = mycursor.fetchall()
valid_table = False
if row:
print (table,"table name verified")
valid_table = True
else:
print (table,"Table name not in database", db_name)
if valid_table:
try:
mycursor.execute('insert into ' +use_delimiter+ table +use_delimiter+ ' (my_data,my_column_name) values (?,?) ',(1,"Name"));
db.commit()
except Exception as e:
print "Error:", str(e)
try:
mycursor.execute('UPDATE ' +use_delimiter+ table +use_delimiter+ ' set my_column_name = ? where my_data = ?', ["ReNamed",1])
db.commit()
except Exception as e:
print "Error:", str(e)
db.close()
you can use something like this
conn = sqlite3.connect()
createTable = '''CREATE TABLE %s (# );''' %dateNow)
conn.execute(createTable)
basically, if we want to separate the data into several tables according to the date right now, for example, you want to monitor a system based on the date.
createTable = '''CREATE TABLE %s (# );''' %dateNow) means that you create a table with variable dateNow which according to your coding language, you can define dateNow as a variable to retrieve the current date from your coding language.
You can save your query in a .sql or txt file and use the open().replace() method to use variables in any part of your query. Long time reader but first time poster so I apologize if anything is off here.
```SQL in yoursql.sql```
Sel *
From yourdbschema.tablenm
```SQL to run```
tablenm = 'yourtablename'
cur = connect.cursor()
query = cur.execute(open(file = yoursql.sql).read().replace('tablenm',tablenm))
You can pass a string as the SQL command:
import sqlite3
conn = sqlite3.connect('db.db')
c = conn.cursor()
tablename, field_data = 'some_table','some_data'
query = 'SELECT * FROM '+tablename+' WHERE column1=\"'+field_data+"\""
c.execute(query)
Question: Is it possible to use a variable as your table name without having to use string constructors to do so?
Info:
I'm working on a project right now that catalogs data from a star simulation of mine. To do so I'm loading all the data into a sqlite database. It's working pretty well, but I've decided to add a lot more flexibility, efficiency, and usability to my db. I plan on later adding planetoids to the simulation, and wanted to have a table for each star. This way I wouldn't have to query a table of 20m some planetoids for the 1-4k in each solar system.
I've been told using string constructors is bad because it leaves me vulnerable to a SQL injection attack. While that isn't a big deal here as I'm the only person with access to these dbs, I would like to follow best practices. And also this way if I do a project with a similar situation where it is open to the public, I know what to do.
Currently I'm doing this:
cursor.execute("CREATE TABLE StarFrame"+self.name+" (etc etc)")
This works, but I would like to do something more like:
cursor.execute("CREATE TABLE StarFrame(?) (etc etc)",self.name)
though I understand that this would probably be impossible. though I would settle for something like
cursor.execute("CREATE TABLE (?) (etc etc)",self.name)
If this is not at all possible, I'll accept that answer, but if anyone knows a way to do this, do tell. :)
I'm coding in python.
Unfortunately, tables can't be the target of parameter substitution (I didn't find any definitive source, but I have seen it on a few web forums).
If you are worried about injection (you probably should be), you can write a function that cleans the string before passing it. Since you are looking for just a table name, you should be safe just accepting alphanumerics, stripping out all punctuation, such as )(][;, and whitespace. Basically, just keep A-Z a-z 0-9.
def scrub(table_name):
return ''.join( chr for chr in table_name if chr.isalnum() )
scrub('); drop tables --') # returns 'droptables'
For people searching for a way to make the table as a variable, I got this from another reply to same question here:
It said the following and it works. It's all quoted from mhawke:
You can't use parameter substitution for the table name. You need to add the table name to the query string yourself. Something like this:
query = 'SELECT * FROM {}'.format(table)
c.execute(query)
One thing to be mindful of is the source of the value for the table name. If that comes from an untrusted source, e.g. a user, then you need to validate the table name to avoid potential SQL injection attacks. One way might be to construct a parameterised query that looks up the table name from the DB catalogue:
import sqlite3
def exists_table(db, name):
query = "SELECT 1 FROM sqlite_master WHERE type='table' and name = ?"
return db.execute(query, (name,)).fetchone() is not None
I wouldn't separate the data into more than one table. If you create an index on the star column, you won't have any problem efficiently accessing the data.
Try with string formatting:
sql_cmd = '''CREATE TABLE {}(id, column1, column2, column2)'''.format(
'table_name')
db.execute(sql_cmd)
Replace 'table_name' with your desire.
To avoid hard-coding table names, I've used:
table = "sometable"
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS {} (
importantdate DATE,
somename VARCHAR,
)'''.format(table))
c.execute('''INSERT INTO {} VALUES (?, ?)'''.format(table),
(datetime.strftime(datetime.today(), "%Y-%m-%d"),
myname))
As has been said in the other answers, "tables can't be the target of parameter substitution" but if you find yourself in a bind where you have no option, here is a method of testing if the table name supplied is valid.
Note: I have made the table name a real pig in an attempt to cover all of the bases.
import sys
import sqlite3
def delim(s):
delims="\"'`"
use_delim = []
for d in delims:
if d not in s:
use_delim.append(d)
return use_delim
db_name = "some.db"
db = sqlite3.connect(db_name)
mycursor = db.cursor()
table = 'so""m ][ `etable'
delimiters = delim(table)
if len(delimiters) < 1:
print "The name of the database will not allow this!"
sys.exit()
use_delimiter = delimiters[0]
print "Using delimiter ", use_delimiter
mycursor.execute('SELECT name FROM sqlite_master where (name = ?)', [table])
row = mycursor.fetchall()
valid_table = False
if row:
print (table,"table name verified")
valid_table = True
else:
print (table,"Table name not in database", db_name)
if valid_table:
try:
mycursor.execute('insert into ' +use_delimiter+ table +use_delimiter+ ' (my_data,my_column_name) values (?,?) ',(1,"Name"));
db.commit()
except Exception as e:
print "Error:", str(e)
try:
mycursor.execute('UPDATE ' +use_delimiter+ table +use_delimiter+ ' set my_column_name = ? where my_data = ?', ["ReNamed",1])
db.commit()
except Exception as e:
print "Error:", str(e)
db.close()
you can use something like this
conn = sqlite3.connect()
createTable = '''CREATE TABLE %s (# );''' %dateNow)
conn.execute(createTable)
basically, if we want to separate the data into several tables according to the date right now, for example, you want to monitor a system based on the date.
createTable = '''CREATE TABLE %s (# );''' %dateNow) means that you create a table with variable dateNow which according to your coding language, you can define dateNow as a variable to retrieve the current date from your coding language.
You can save your query in a .sql or txt file and use the open().replace() method to use variables in any part of your query. Long time reader but first time poster so I apologize if anything is off here.
```SQL in yoursql.sql```
Sel *
From yourdbschema.tablenm
```SQL to run```
tablenm = 'yourtablename'
cur = connect.cursor()
query = cur.execute(open(file = yoursql.sql).read().replace('tablenm',tablenm))
You can pass a string as the SQL command:
import sqlite3
conn = sqlite3.connect('db.db')
c = conn.cursor()
tablename, field_data = 'some_table','some_data'
query = 'SELECT * FROM '+tablename+' WHERE column1=\"'+field_data+"\""
c.execute(query)
I have following PostgreSQL query
data = {'token':"123",'kart_user':"tablename"}
select_stmt = "SELECT * FROM %(kart_user)s WHERE token = %(token)s"
self.cur.execute(select_stmt, data)
result = self.cur.fetchone()
In the select query %(kart_user)s is represent in the form of single quotes
My question is how can i remove the quotes from the same query??.
The whole point of using SQL parameters is to prevent data being interpreted as SQL objects or syntax. You can't use SQL parameters to name tables; you can only use SQL parameters for data.
You'll have to interpolate table names separately. This does means you run a risk of opening up your code to SQL injections; I'd test the table name against a list of known tablenames if this is sourced from user input.
assert data['kart_user'] in known_tables
select_stmt = "SELECT * FROM {} WHERE token = %(token)s".format(data['kart_user'])
self.cur.execute(select_stmt, data)
I have two queries in SQL which are the following:
q1 = select date_hour from table
And, the second query is:
q2 = select date(date_hour) from table
The only difference between these queries is the string date_hour and date(date_hour). SO, I tried parameterising my query in the following manner:
q1 = select %s from table
cur.execute(q1,'date')
cur.execute(q1,'date(date_hour)')
However, this throws an error which is:
not all arguments converted during string formatting
Why am I getting this error? How can I fix it?
Change the comma in cur.execute to %
Change this:
q1 = "select %s from table"
cur.execute(q1,'date')
cur.execute(q1,'date(date_hour)')
to:
q1 = "select %s from table"
cur.execute(q1 % 'date')
cur.execute(q1 % 'date(date_hour)')
It's unclear wich sql library you're using but assuming it uses the Python DB API:
Sql parameters are typically used for values, not columns names (while this is possible using stored procedures).
It seems you're confused between string formatting in python and sql parametized queries.
While %s can be used to format a string (see formatting strings) this is not the way to set sql parameters.
See this response to use sql parameters in python.
By the way i can't see anything wrong with this simple code:
cursor=cnx.curor()
query="select date_hour from table"
cursor.execute(query)
query="select date(date_hour) from table"
cursor.execute(query)
Change your code to something like this:
q1 = "select %s from table"
cur.execute(q1,['date'])
cur.execute(q1,['date(date_hour)'])
Check this