We Keep Coding

Python sqlite operation,SQL statement 'where field in (1,2)' syntax error

Python sqlite operation,SQL statement 'where field in (1,2)' syntax error
The error is:sqlite3.OperationalError: near ":id": syntax error
My search of the Official Python documentation and Google failed to find the answer:
https://docs.python.org/3/library/sqlite3.html
How should arguments be passed?
'''first create test.db:table and field
CREATE TABLE test_tab (
id INTEGER PRIMARY KEY ASC,
test_num INT,
test_field TEXT
);
'''
import sqlite3
con = sqlite3.connect('test.db')
con.set_trace_callback(print) # start the debug
d = [
(111,'aaa'),
(111,'bbb'),
(111,'ccc'),
(444,'ddd')
]
sql = "insert into `test_tab` (`test_num`, `test_field`) values (?,?)"
cursor = con.executemany(sql, d)
con.commit() # Execute successfully
#####################
# wrong code begin,why sql 'in ()' is wrong?
sql = "SELECT * from `test_tab` where `test_num`=:num AND `id` in :id"
par = {'num': 111, 'id': (1,2)} # The number of 'id' parameters is uncertain
x = con.execute(sql, par)
print(x.fetchall())

In the second query, you would need actually separate placeholders for every value in the IN clause. In addition, I would use ? here:
num = 111
ids = (1, 2)
par = (num,) + ids
sql = "select * from test_tab where test_num = ? AND id in "
in_clause = '(?' + ', ?'*(len(ids) - 1) + ')'
sql = sql + in_clause
x = con.execute(sql, par)
print(x.fetchall())
The SQL query generated by the above script is:
select * from test_tab where test_num = ? AND in (?, ?)
and we bind (111, 1, 2) to the three ? placeholders.

Correct function definition for optional parameters of a SELECT query in Python using psycopg2 with POSTGRESQL

I want to SELECT from my PostgreSQL database using optional filter parameters in a function, but am struggling to get the syntax right.
def search(title: str = '', author: str = "", year: int, isbn: int):
conn = psycopg2.connect(
" dbname='mydb' user='postgres' password='pwd' host='localhost' port='5432' ")
curs = conn.cursor()
curs.execute("SELECT * FROM book WHERE title=? OR author=? OR year=? OR isbn=?",
(title, author, year, isbn))
rows = curs.fetchall()
conn.close()
return rows
I then want to search by passing only one parameter (author name) to the function, and have the SELECT-statement be able to ignore/skip the fields that have not been passed. When I execute
print(search(author="John"))
The output is
Traceback (most recent call last):
File "backend.py", line 48, in <module>
print(search(author="John"))
File "backend.py", line 39, in search
(title, author, year, isbn))
psycopg2.errors.UndefinedFunction: operator does not exist: text =?
LINE 1: SELECT * FROM book WHERE title=? OR author=? OR year=? OR is...
^
HINT: No operator matches the given name and argument type. You might need to add an explicit type cast.
I understand this relates to a mismatch between the datatype of the SELECT-statement and the database column type, and that the error reflects that there is no = operator for comparing those two data types. But how do I define my function parameters and the SELECT statement to get my query to work?

psycopg2 uses %s as the placeholder for string substitution, not ?. Try changing the ? to %s and it should work.
Edit:
#Parfait brought up a good point about your where logic. Changing ? to %s will fix your syntax error, but still won't fix the logic. One common way of handling this is to default all of your parameters to None and change your query to something like this:
curs.execute("""
SELECT * FROM BOOK
WHERE (title = %(title)s OR %(title)s IS NULL)
AND (author = %(author)s OR %(author)s IS NULL)
AND (year = %(year)s OR %(year)s IS NULL)
AND (isbn = %(isbn)s OR %(isbn)s IS NULL)""",
{"title": title, "author": author, "year": year, "isbn": isbn})
This is not the most performant method, but it's probably the simplest.

Alternatively, consider passing None as default which translates to NULL and use SQL's COALESCE to select the first not null value. If params are None, then the columns in query will equal each other which is essentially applying no filter.
def search(title = None, author = None, year = None, isbn = None):
conn = psycopg2.connect(
" dbname='mydb' user='postgres' password='pwd' host='localhost' port='5432' ")
sql = """SELECT * FROM book
WHERE COALESCE(title, 'N/A') = COALESCE(%s, title, 'N/A')
AND COALESCE(author, 'N/A') = COALESCE(%s, author, 'N/A')
AND COALESCE(year, -1) = COALESCE(%s, year, -1)
AND COALESCE(isbn, -1) = COALESCE(%s, isbn, -1)
"""
curs = conn.cursor()
curs.execute(sql, (title, author, year, isbn))
rows = curs.fetchall()
conn.close()
return rows

How to use like pattern matching with PostgreSQL and Python with multiple percentage (%) symbols?

I am trying to pattern match with the LIKE LOWER('% %') command however I think the fact that I am using a python variable with %s is mucking it up. I can't seem to find any escape characters for the percentage symbol and my program gives me no errors. Is this the problem or is there something else I'm missing. It does work if I just run LIKE %s however I need to be able to search like not equals.
# Ask for the database connection, and get the cursor set up
conn = database_connect()
if(conn is None):
return ERROR_CODE
cur = conn.cursor()
print("search_term: ", search_term)
try:
# Select the bays that match (or are similar) to the search term
sql = """SELECT fp.name AS "Name", fp.size AS "Size", COUNT(*) AS "Number of Fish"
FROM FishPond fp JOIN Fish f ON (fp.pondID = f.livesAt)
WHERE LOWER(fp.name) LIKE LOWER('%%s%') OR LOWER(fp.size) LIKE LOWER('%%s%')
GROUP BY fp.name, fp.size"""
cur.execute(sql, (search_term, ))
rows = cur.fetchall()
cur.close() # Close the cursor
conn.close() # Close the connection to the db
return rows
except:
# If there were any errors, return a NULL row printing an error to the debug
print("Error with Database - Unable to search pond")
cur.close() # Close the cursor
conn.close() # Close the connection to the db
return None

Instead of embedding the ampersands in the query string, you could wrap the search term string in ampersands, and then pass that to cursor.execute():
sql = 'SELECT * from FishPond fp WHERE LOWER(fp.name) LIKE LOWER(%s)'
search_term = 'xyz'
like_pattern = '%{}%'.format(search_term)
cur.execute(sql, (like_pattern,))
The query is simplified for the purpose of example.
This is more flexible because the calling code can pass any valid LIKE pattern to the query.
BTW: In Postgresql you can use ILIKE for case insensitive pattern matching, so the example query could be written as this:
sql = 'SELECT * from FishPond fp WHERE fp.name ILIKE %s'
As noted in the documentation ILIKE is a Postgresql extension, not standard SQL.

You can escape % with another %
>>> test = 'test'
>>> a = 'LIKE %%s%'
>>> a % test
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: incomplete format
>>>
>>> a = 'LIKE %%%s%%'
>>> a % test
'LIKE %test%'
P.S. you also have two placeholders, but you are passing only one argument in execute

WHERE IN Clause in python list [duplicate]

This question already has answers here:
imploding a list for use in a python MySQLDB IN clause
(8 answers)
Closed 1 year ago.
I need to pass a batch of parameters to mysql in python. Here is my code:
sql = """ SELECT * from my_table WHERE name IN (%s) AND id=%(Id)s AND puid=%(Puid)s"""
params = {'Id':id,'Puid' : pid}
in_p=', '.join(list(map(lambda x: '%s', names)))
sql = sql %in_p
cursor.execute(sql, names) #todo: add params to sql clause
The problem is I want to pass the name list to sql IN clause, meanwhile I also want to pass the id and puid as parameters to the sql query clause. How do I implement these in python?

Think about the arguments to cursor.execute that you want. You want to ultimately execute
cursor.execute("SELECT * FROM my_table WHERE name IN (%s, %s, %s) AND id = %s AND puid = %s;", ["name1", "name2", "name3", id, pid])
How do you get there? The tricky part is getting the variable number of %ss right in the IN clause. The solution, as you probably saw from this answer is to dynamically build it and %-format it into the string.
in_p = ', '.join(list(map(lambda x: '%s', names)))
sql = "SELECT * FROM my_table WHERE name IN (%s) AND id = %s AND puid = %s;" % in_p
But this doesn't work. You get:
TypeError: not enough arguments for format string
It looks like Python is confused about the second two %ss, which you don't want to replace. The solution is to tell Python to treat those %ss differently by escaping the %:
sql = "SELECT * FROM my_table WHERE name IN (%s) AND id = %%s AND puid = %%s;" % in_p
Finally, to build the arguments and execute the query:
args = names + [id, pid]
cursor.execute(sql, args)

sql = """ SELECT * from my_table WHERE name IN (%s) AND id=%(Id)s AND puid=%(Puid)s""".replace("%s", "%(Clause)s")
print sql%{'Id':"x", 'Puid': "x", 'Clause': "x"}
This can help you.

Python sqlite3 operational error when string formatting?

I have this small decorator function where I am going to find an unknown number of entries in a table:
def Deco(func):
func
conn = sqlite3.connect('/home/User/vocab_database/vocab.db')
with conn:
cur = conn.cursor()
cur.execute("SELECT name FROM sqlite_master WHERE type='table'")
total = cur.fetchall()
print "You have %d tables " % len(total)
## this line below, is where I wanted to use a formatted string ##
cur.execute("SELECT * FROM %s") % total[0]
entries = cur.fetchall()
print "You have %d entries" % len(entries)
Then I get this error message:
Traceback (most recent call last):
File "./database_trial.py", line 19, in <module>
class Vocab:
File "./database_trial.py", line 25, in Vocab
#Deco
File "./database_trial.py", line 15, in Deco
cur.execute("SELECT * FROM %s") % total[0]
sqlite3.OperationalError: near "%": syntax error
Does sqlite3 only accept ? operators? Or is there something I'm mucking up?

You're trying to replace metadata, so unfortunately a parametrized query won't work. You must use interpolation or the like here, but make sure that the value is sanitized; this is a possible vector for a SQL injection.
cur.execute("SELECT * FROM %s" % (total[0],))

In the line cur.execute("SELECT * FROM %s") % total[0], you are applying the % operator to the result of the cur.execute call. I think you want to do the substitution inside the call, e.g., cur.execute("SELECT * FROM ?", (total[0],)).