I'm new to Python and Psycopg2... I'am trying to do a query that uses IN sql statement and other WHERE clauses but I'm getting an error like this:
psycopg2.ProgrammingError: argument formats can't be mixed
From what I understand I'm mixing a Python tuple with strings, here is the SELECT statement:
cur2.execute("SELECT hash FROM jobsads_text\
WHERE\
date_inserted::timestamp::date - now()::timestamp::date <= 0\
AND date_inserted::timestamp::date - now()::timestamp::date >= -7\
AND hash NOT IN %s \
AND lower((%s)) LIKE '%(%s)%'\
ORDER BY date_inserted asc;", ((not_in_sql,), search_field, search_string))
I get error in the query above.
This query bellow runs OK:
cur2.execute("SELECT hash FROM jobsads_text\
WHERE\
date_inserted::timestamp::date - now()::timestamp::date <= 0\
AND date_inserted::timestamp::date - now()::timestamp::date >= -7\
AND hash NOT IN %s \
ORDER BY date_inserted asc;", (not_in_sql,))
My question is... How can I mix the tuple not_in_sql with the strings search_field and search_string?
Any clues?
Best Regards,
t = (1, 3)
search_field = 'c'
search_string = '%something%'
print cursor.mogrify("""\
select *
from p
where
c in %%s
and
lower (%s) like %%s
""" % search_field, (t, search_string))
Will output this:
select *
from p
where
c in (1, 3)
and
lower (c) like '%something%'
psycopg2 will not substitute identifiers like column names so you must substitute then before passing the query as the first argument of the method.
Related
I want to run a query that filter two columns based on values from two list.
Basically, I want to simulate two filters like this:
SELECT *
FROM my_table
WHERE customers in ("John","Peter") AND customers_numbers IN ('1','2')
But the values from customers and customers_number are in two lists. To try this I am making the following code:
list1 = ["John","Peter"]
list2 = [1,2]
query_sql = "DELETE FROM vw_bill_details WHERE customers in (%s) and customers_numbers in (%s)" % ','.join(['?'] * len(list1)) % ','.join(['?'] * len(list2))
cursor.execute(query_sql, list1,list2)
But I am getting the following error:
query_sql = "DELETE FROM vw_bill_details WHERE customers in (%s) and customers_numbers in (%s)" % ','.join(['?'] * len(list1)) % ','.join(['?'] * len(list2))
TypeError: not enough arguments for format string
How can I make the above query using python?
Thanks!
You've got an error in your query, there's an extra % between the two terms instead of a comma. Also, when you use % formatting with more than one term, you need to have the entire variables section after the % in parentheses:
query_sql = "DELETE FROM vw_bill_details WHERE customers in (%s) and customers_numbers in (%s)" % (','.join(['?'] * len(list1)), ','.join(['?'] * len(list2)))
Improvements:
consider writing your query in a docstring so that it's easier read, write, and debug:
query_sql = """DELETE FROM vw_bill_details
WHERE customers in (%s)
and customers_numbers in (%s)""" % (
','.join(['?'] * len(list1)), ','.join(['?'] * len(list2)))
str.join() works on any iterable, including strings, so the ','.join(['?'] * len(list1)) parts could be written as ','.join('?' * len(list1)) - the ? mark is a single string rather than a list with a single element.
There is potential for matching the wrong records: the WHERE customers in ("John","Peter") AND customers_numbers IN ('1','2') doesn't care/check of 'John' has cust_number 1 or 2. So it could match with a John-2 and a Peter-1, instead of you're intended John-1 and Peter-2.
Examples of the mismatch can be seen here: http://sqlfiddle.com/#!9/caa7f3/2
You can avoid this mismatch by specifying the name and number matches each:
WHERE (customers = 'John' AND customers_numbers = '1')
OR (customers = 'Peter' AND customers_numbers = '2')
which could also be written as as a matching pair:
WHERE (customers, customers_numbers) = ('John', 1)
and you can extend it for multiple options with:
WHERE (customers, customers_numbers) IN (('John', 1), ('Peter', 2))
which is easier to parameterize with ?s than the extended AND/OR version above.
I would like get a list of values from a DB table in python using cx_oracle. I am unable to write a query with two where conditions one of single value and another of a list.
I am able to achieve it when I filter it two strings separately or only filter it by a list of string. But could not achieve it together!!
output_list=[]
catlist = ','.join(":x%d" % i for i, _ in enumerate(category_list))
db_cursor = connection.cursor()
db_cursor.execute("""
SELECT LWEX_WORD_EXCLUDE
FROM WCG_SRC_WORD_EXCLUDE
WHERE LWEX_CATEGORY IN (%s) and LWIN_USER_UPDATED = :did""" % catlist, category_list, did =argUser)
for word in db_cursor :
output_list.append(word[0])
The current code throws an error. But if I have either of the conditions separately then it works fine. The python version that I am using is 3.5
You cannot mix and match "bind by position" and "bind by name", which is what you are doing in the above code. My suggestion would be to do something like this instead:
output_list=[]
catlist = ','.join(":x%d" % i for i, _ in enumerate(category_list))
bindvals = category_list + [arguser]
db_cursor = connection.cursor()
db_cursor.execute("""
SELECT LWEX_WORD_EXCLUDE
FROM WCG_SRC_WORD_EXCLUDE
WHERE LWEX_CATEGORY IN (%s) and LWIN_USER_UPDATED = :did""" % catlist, bindvals)
for word in db_cursor :
output_list.append(word[0])
I have a SQL statement that works in mysql:
SELECT * FROM `ps_message` WHERE `id_order` = 111 ORDER BY id_message asc LIMIT 1
What is wrong with the following statement in Python:
cursor2.execute("SELECT * FROM ps_message WHERE id_order='%s'" % order["id_order"] " ORDER BY id_message asc LIMIT 1")
How should the syntax be in Python to work?
You have a syntax error in string formatting. Should be:
cursor2.execute("SELECT * FROM ps_message WHERE id_order='%s' ORDER BY id_message asc LIMIT 1" % order["id_order"])
Using format() is also preferable over old-style string formatting. Read more about it here.
Pass the order number as a query parameter.
e.g.
cursor2.execute("SELECT * FROM ps_message WHERE id_order=%s ORDER BY id_message asc LIMIT 1", [ order["id_order"] ])
Note that when using query parameters you don't put quotes around the %s.
This approach is recommended to avoid the risk of sql injection attacks.
It should also be more efficient if there are many queries.
https://docs.python.org/2/library/sqlite3.html
http://pymssql.org/en/stable/pymssql_examples.html
I am new to python. What I am trying to achieve is to insert values from my list/tuple into my redshift table without iteration.I have around 1 million rows and 1 column. Below is the code I am using to create my list/tuple.
cursor1.execute("select domain from url limit 5;")
for record, in cursor1:
ext = tldextract.extract(record)
mylist.append(ext.domain + '.' + ext.suffix)
mytuple = tuple(mylist)
I am not sure what is best to use, tuple or list. output of print(mylist) and print(mytuple) are as follows.
List output
['friv.com', 'steep.tv', 'wordpress.com', 'fineartblogger.net',
'v56.org'] Tuple Output('friv.com', 'steep.tv', 'wordpress.com',
'fineartblogger.net', 'v56.org')
Now, below is the code I am using to insert the values into my redshift table but I am getting an error:
cursor2.execute("INSERT INTO sample(domain) VALUES (%s)", mylist) or
cursor2.execute("INSERT INTO sample(domain) VALUES (%s)", mytuple)
Error - not all arguments converted during string formatting
Any help is appreciated. If any other detail is required please let me know, I will edit my question.
UPDATE 1:
Tried using below code and getting different error.
args_str = ','.join(cur.mogrify("(%s)", x) for x in mylist)
cur.execute("INSERT INTO table VALUES " + args_str)
ERROR - INSERT has more expressions than target columns
I think you're looking for Fast Execution helpers:
mylist=[('t1',), ('t2',)]
execute_values(cursor2, "INSERT INTO sample(domain) %s", mylist, page_size=100)
what this does is it replaces the %s with 100 VALUES. I'm not sure how high you can set page_size, but that should be far more performant.
Finally found a solution. For some reason cur.mogrify was not giving me proper sql string for insert. Created my own SQl string and it works alot faster than cur.executeall()
list_size = len(mylist)
for len in range(0,list_size):
if ( len != list_size-1 ):
sql = sql + ' ('+ "'"+ mylist[len] + "'"+ ') ,'
else:
sql = sql + '('+ "'"+ mylist[len] + "'"+ ')'
cursor1.execute("INSERT into sample(domain) values " + sql)
Thanks for your help guys!
I have the following code in python. I get this error ->tuple indices must be integers, not str
How can I pass these values into the query? I have other examples where this approach works perfectly, i don't understand why it's failling here.
def request_events_json(uei,interval,conn):
cur = conn.cursor()
events_query ="""select e.nodeid,n.nodelabel,e.ipaddr,count(*) as total,min(e.eventcreatetime),max(e.eventcreatetime),(regexp_matches (e.eventlogmsg,E': %(.*)'))[1] as msglog
from events e, node n where e.eventuei = (%s) and e.eventcreatetime > now() - interval (%s) and n.nodeid=e.nodeid
group by n.nodelabel,e.nodeid,e.ipaddr,msglog
order by e.nodeid, count(*) desc limit 10;"""
try:
print('## Requesting events ##')
cur.execute(events_query,('uei.opennms.org/syslogd/cisco/line','5 min'))
.......
With my version of PostgreSQL the round brackets after interval are forbidden.
Update:
It is the percent-sign in the regexp. Double it.