I am new to working on Python. I m not able to understand how can I send the correct input t0 the query.
list_of_names = []
for country in country_name_list.keys():
list_of_names.append(getValueMethod(country))
sql_query = f"""SELECT * FROM table1
where name in (%s);"""
db_results = engine.execute(sql_query, list_of_names).fetchone()
Give the error " not all arguments converted during string formatting"
As implied by John Gordon's comment, the number of placeholders in the SQL statement should match the number of elements in the list. However SQLAlchemy 2.0+ no longer accepts raw SQL statements. A future-proof version of the code would be:
import sqlalchemy as sa
...
# SQL statements should be wrapped with text(), and should used
# the "named" parameter style.
sql_query = sa.text("""SELECT * FROM table1 where name in :names)"""
# Values should be dictionaries of lists of dictionaries,
values = {'names': list_of_names}
# Execute statements using a context manager.
with engine.connect() as conn:
db_results = conn.execute(sql_query, values).fetchone()
If I know right, there are a simpler solution. If you write curly bracets {}, not bracets (), and you place inside the bracets a variable, which contains the %s value, should work. I don't know, how sql works, but you should use one " each side, not three.
Sorry, I'm not english. From this, maybe I wasn't help with the question, because I don't understand correctly.
Related
I am wanting to run a Presto SQL query in a for loop so that the query will pull hourly data based on my date variables.
Example query is along the lines of:
x = datetime.strptime('12-10-22', '%d-%m-%y').date()
y = datetime.strptime('13-10-22', '%d-%m-%y').date()
for dt in rrule.rrule(rrule.HOURLY, dtstart=nextProcStart, until=nextProcEnd):
sql_query = "SELECT SUM(sales) FROM a WHERE date between x and y"
I will note I'm using the syntax of writing the SQL query as a variable so along the lines of:
sql_query = """ SELECT... FROM..."""
I have tried just adding the variables into the query but no luck. Unsure what steps will work.
I've also tried using .format(x,y) at the end of my SQL query but keep getting an error saying
KeyError: 'x'
Remember that your SQL statement is no more than a string, so you just need to know how to incorporate a variable into a string.
Try:
sql_query = "SELECT SUM(sales) FROM a WHERE date between {} and {}".format(x, y)
Read How do I put a variable’s value inside a string (interpolate it into the string)? for more info or alternative methods.
Hopefully this answers your immediate question above on how to incorporate variable into string and get your code, as is, to work. However, as #nbk, mentions in comment below, this method is NOT recommended as it is insecure.
Using concatenations in SQL statements like this does open the code up to injection attacks. Even if your database does not contain sensitive information, it is bad practice.
Prepared statements have many advantages, not least of all that they are more secure and more efficient. I would certainly invest some time in researching and understanding SQL prepared statements.
I have a function that executes many SQL queries with different dates.
What I want is to pass all dates and other query variables as function parameters and then just execute the function. I have figured out how to do this for datetime variables as below. But I also have a query that looks at specific campaign_names in a database and pulls those as strings. I want to be able to pass those strings as function parameters but I haven't figured out the correct syntax for this in the SQL query.
def Camp_eval(start_date,end_1M,camp1,camp2,camp3):
query1 = f"""SELECT CONTACT_NUMBER, OUTCOME_DATE
FROM DATABASE1
where OUTCOME_DATE >= (to_date('{start_date}', 'dd/mm/yyyy'))
and OUTCOME_DATE < (to_date('{end_1M}', 'dd/mm/yyyy'))"""
query2 = """SELECT CONTACT_NUMBER
FROM DATABASE2
WHERE (CAMP_NAME = {camp1} or
CAMP_NAME = {camp2} or
CAMP_NAME = {camp3})"""
Camp_eval('01/04/2022','01/05/2022','Camp_2022_04','Camp_2022_05','Camp_2022_06')
The parameters start_date and end_1M work fine with the {} brackets but the camp variables, which are strings don't return any results even though there are results in the database with those conditions if I were to write them directly in the query.
Any help would be appreciated!!
Please, do not use f-strings for creating SQL queries!
Most likely, any library you use for accessing a database already has a way of creating queries: SQLite docs (check code examples).
Another example: cur.execute("SELECT * FROM tasks WHERE priority = ?", (priority,)).
Not only this way is safer (fixes SQL Injection problem mentioned by #d-malan in comments), but it also eliminates the need to care about how data is represented in SQL - the library will automatically cast dates, strings, etc. in what they need to be casted into. Therefore, your problem can be fixed by using proper instruments.
I am aware that queries in Python can be parameterized using either ? or %s in execute query here or here
However I have some long query that would use some constant variable defined at the beginning of the query
Set #my_const = 'xyz';
select #my_const;
-- Query that use #my_const 40 times
select ... coalesce(field1, #my_const), case(.. then #my_const)...
I would like to do the least modif possible to the query from Mysql. So that instead of modifying the query to
pd.read_sql(select ... coalesce(field1, %s), case(.. then %s)... , [my_const, my_const, my_const, ..]
,I could write something along the line of the initial query. Upon trying the following, however, I am getting a TypeError: 'NoneType' object is not iterable
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
Any idea how to use the original variable defined in Mysql query ?
The reason
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
throws that exception is because all you are doing is setting null_value to '' and then selecting that '' - what exactly would you have expected that to give you? EDIT read_sql only seems to execute one query at a time, and as the first query returns no rows it results in that exception.
If you split them in to two calls to read_sql then it will in fact return you the value of your #null value in the second call. Due to this behaviour read_sql is clearly not a good way to do this. I strongly suggest you use one of my suggestions below.
Why are you wanting to set the variable in the SQL using '#' anyway?
You could try using the .format style of string formatting.
Like so:
query_str = "select ... coalesce(field1, {c}), case(.. then {c})...".format(c=my_const)
pd.read_sql(query_str)
Just remember that if you do it this way and your my_const is a user input then you will need to sanitize it manually to prevent SQL injection.
Another possibility is using a dict of params like so:
query_str = "select ... coalesce(field1, %(my_const)s, case(.. then %(my_const)s)..."
pd.read_sql(query_str, params={'my_const': const_value})
However this is dependent on which database driver you use.
From the pandas.read_sql docs:
Check your database driver documentation for which of the five syntax
styles, described in PEP 249’s paramstyle, is supported. Eg. for
psycopg2, uses %(name)s so use params={‘name’ : ‘value’}
So here is my problem: I am trying to select a specific value from a table
comparing it with a unicode string. The value is also unicode. I am using
mysql.connector. The server settings are all utf8 oriented. When I run
following query - I get an empty list. When I run it without 'WHERE Title like '%s'' part, I get a full set of values, and they properly displayed in the
output. The same query works in the command line on the server. The value is
there for sure. What is it that I am missing?
conn = sql.connect(host='xxxxxxx', user='xxx', password='xxx', database='db', charset="utf8")
cur = conn.cursor()
townQuery = (u"""SELECT * FROM Towns WHERE Title like '%s' """)
tqd = (u"%" +u"Серов"+u"%")
cur.execute(townQuery, tqd)
for i in cur:
print i
When you use the 2-argument form of cur.execute (thus passing the arguments, tqd, to the parametrized sql, townQuery), the DB adaptor will quote the arguments for you. Therefore, remove the single quotes from around the %s in townQuery:
townQuery = u"""SELECT * FROM Towns WHERE Title like %s"""
tqd = [u"%Серов%"]
cur.execute(townQuery, tqd)
Also note that the second argument, tqd, must be a sequence such as a list or tuple. The square brackets around u"%Серов%" makes [u"%Серов%"] a list. Parentheses around u"%Серов%" do NOT make (u"%Серов%") a tuple because Python evaluates the quantity in parentheses to a unicode. To make it a tuple, add a comma before the closing parenthesis: (u"%Серов%",).
Does psycopg2 have a function for escaping the value of a LIKE operand for Postgres?
For example I may want to match strings that start with the string "20% of all", so I want to write something like this:
sql = '... WHERE ... LIKE %(myvalue)s'
cursor.fetchall(sql, { 'myvalue': escape_sql_like('20% of all') + '%' }
Is there an existing escape_sql_like function that I could plug in here?
(Similar question to How to quote a string value explicitly (Python DB API/Psycopg2), but I couldn't find an answer there.)
Yeah, this is a real mess. Both MySQL and PostgreSQL use backslash-escapes for this by default. This is a terrible pain if you're also escaping the string again with backslashes instead of using parameterisation, and it's also incorrect according to ANSI SQL:1992, which says there are by default no extra escape characters on top of normal string escaping, and hence no way to include a literal % or _.
I would presume the simple backslash-replace method also goes wrong if you turn off the backslash-escapes (which are themselves non-compliant with ANSI SQL), using NO_BACKSLASH_ESCAPE sql_mode in MySQL or standard_conforming_strings conf in PostgreSQL (which the PostgreSQL devs have been threatening to do for a couple of versions now).
The only real solution is to use the little-known LIKE...ESCAPE syntax to specify an explicit escape character for the LIKE-pattern. This gets used instead of the backslash-escape in MySQL and PostgreSQL, making them conform to what everyone else does and giving a guaranteed way to include the out-of-band characters. For example with the = sign as an escape:
# look for term anywhere within title
term= term.replace('=', '==').replace('%', '=%').replace('_', '=_')
sql= "SELECT * FROM things WHERE description LIKE %(like)s ESCAPE '='"
cursor.execute(sql, dict(like= '%'+term+'%'))
This works on PostgreSQL, MySQL, and ANSI SQL-compliant databases (modulo the paramstyle of course which changes on different db modules).
There may still be a problem with MS SQL Server/Sybase, which apparently also allows [a-z]-style character groups in LIKE expressions. In this case you would want to also escape the literal [ character with .replace('[', '=['). However according to ANSI SQL escaping a character that doesn't need escaping is invalid! (Argh!) So though it will probably still work across real DBMSs, you'd still not be ANSI-compliant. sigh...
I was able to escape % by using %% in the LIKE operand.
sql_query = "select * from mytable where website like '%%.com'"
cursor.fetchall(sql_query)
If you're using a prepared statement, then the input will be wrapped in '' to prevent sql injection. This is great, but also prevents input + sql concatenation.
The best and safest way around this would be to pass in the %(s) as part of the input.
cursor.execute('SELECT * FROM goats WHERE name LIKE %(name)s', { 'name': '%{}%'.format(name)})
You can also look at this problem from a different angle. What do you want? You want a query that for any string argument executes a LIKE by appending a '%' to the argument. A nice way to express that, without resorting to functions and psycopg2 extensions could be:
sql = "... WHERE ... LIKE %(myvalue)s||'%'"
cursor.execute(sql, { 'myvalue': '20% of all'})
I found a better hack. Just append '%' to your search query_text.
con, queryset_list = psycopg2.connect(**self.config), None
cur = con.cursor(cursor_factory=RealDictCursor)
query = "SELECT * "
query += " FROM questions WHERE body LIKE %s OR title LIKE %s "
query += " ORDER BY questions.created_at"
cur.execute(query, ('%'+self.q+'%', '%'+self.q+'%'))
I wonder if all of the above is really needed. I am using psycopg2 and was simply able to use:
data_dict['like'] = psycopg2.Binary('%'+ match_string +'%')
cursor.execute("SELECT * FROM some_table WHERE description ILIKE %(like)s;", data_dict)
Instead of escaping the percent character, you could instead make use of PostgreSQL's regex implementation.
For example, the following query against the system catalogs will provide a list of active queries which are not from the autovacuuming sub-system:
SELECT procpid, current_query FROM pg_stat_activity
WHERE (CURRENT_TIMESTAMP - query_start) >= '%s minute'::interval
AND current_query !~ '^autovacuum' ORDER BY (CURRENT_TIMESTAMP - query_start) DESC;
Since this query syntax doesn't utilize the 'LIKE' keyword, you're able to do what you want... and not muddy the waters with respect to python and psycopg2.
Having failed to find a built-in function so far, the one I wrote is pretty simple:
def escape_sql_like(s):
return s.replace('\\', '\\\\').replace('%', '\\%').replace('_', '\\_')
You can create a Like class subclassing str and register an adapter for it to have it converted in the right like syntax (e.g. using the escape_sql_like() you wrote).
I made some modifications to the code above to do the following:
def escape_sql_like(SQL):
return SQL.replace("'%", 'PERCENTLEFT').replace("%'", 'PERCENTRIGHT')
def reescape_sql_like(SQL):
return SQL.replace('PERCENTLEFT', "'%").replace('PERCENTRIGHT', "%'")
SQL = "SELECT blah LIKE '%OUCH%' FROM blah_tbl ... "
SQL = escape_sql_like(SQL)
tmpData = (LastDate,)
SQL = cur.mogrify(SQL, tmpData)
SQL = reescape_sql_like(SQL)
cur.execute(SQL)
It just requires to concatenate double % before and after it. Using "ilike" instead of "like" makes it case insensitive.
query = """
select
*
from
table
where
text_field ilike '%%' || %(search_text)s || '%%'
"""
I think it would be simpler and more readable to use f-strings.
query = f'''SELECT * FROM table where column like '%%{my_value}%%' '''
cursor.execute(query)