Version:
Postgresql = 11.3
Python = 3.7.3
I have a python script that is collecting data from the web and storing it in a database for further analysis.
The below sql statement always returns false when it attempts to compare the variable that has been fetched from the web against what already exists in the database. I think this happens because of a mismatch in the encoding of the variable.
As an example, the variable = Σ1, and Σ1 already exists in the database, so the below sql statement should return TRUE but instead returns FALSE.
cur.execute("SELECT EXISTS(SELECT 1 FROM table_name WHERE column_name = %s)", (variable,))
When I run "cur.query" it shows the following query is being executed:
b"SELECT EXISTS(SELECT 1 FROM table_name WHERE column_name = '\xce\xa31')"
In pgadmin4, the variable is correctly stored as Σ1. If I run the sql query above directly in pgadmin4 with "Σ1" it returns true as expected.
But when I query to check if that variable already exists in the database from the python script, it returns false because it's comparing "Σ1" in the database against "\xce\xa31" in the sql query.
When I run "show CLIENT_ENCODING;" in pgadmin4 it shows "UTF8" and when I print "print(conn.encoding)" immediately before the sql statement in my script it also shows "UTF8".
Where am I going wrong?
If I do this:
>>> b"SELECT EXISTS(SELECT 1 FROM table_name WHERE column_name = '\xce\xa31')".decode("utf-8")
"SELECT EXISTS(SELECT 1 FROM table_name WHERE column_name = 'Σ1')"
it shows that you're not doing anything wrong. The query that you are seeing displayed back is shown as UTF-8 bytes but it is exactly the same as the Unicode query you want. I suspect that the Greek symbol in the database isn't exactly what your select expects to find. That may be because there are in fact several Unicode sigma signs: U+03A3, U+2211 and a few others besides.
Related
I have been working with an SQL database through the mySQL library in Python. I have recently found that when I try searching for a string in my database, it is not correctly returning the output I expect. I think this may be due to my variable not being properly inserted into my SQL command.
code = data['code']
sql = "SELECT 200 FROM mytable WHERE mycolumn = '%s';"
query.execute(sql, teamCode)
print(str(query.fetchall()))
My problem is that printing query.fetchall() prints an empty list ([]) instead of the expected [[200,]] which means the program thinks the code value it is using does not exist in the SQL database, which it does.
The parameters in an execute call need to be a sequence, like a tuple. Try:
query.excute(sql, (teamCode,))
That turns it into a one-element tuple. BTW, did you really mean "code" there?
I am trying to retrieve a value from a table stored in Mysql database using python(pycharm). But instead of outputting the stored value it outputs number of rows instead.
import pymysql
connection = pymysql.connect(host='localhost',
user='root',
password='passw',
database='database1',
charset='utf8',
port=3306
)
x=connection.cursor()
select = x.execute('''SELECT
update_id
FROM
telegram;
''')
print(select)
Output: 1
^Wrong output(Output equals number of rows). As I keep adding on rows the output changes to the number of rows but never returns the value stored.
The command works from MySql perfectly.
SELECT
update_id
FROM
telegram;
Output:233
^This is the correct output.
Why is this happening? What changes should I make in my python code?
According to the documentation on pymysql, this is how you are supposed to do the print out:
sql = "SELECT `id`, `password` FROM `users` WHERE `email`=%s"
cursor.execute(sql, ('webmaster#python.org',))
result = cursor.fetchone()
print(result)
You are missing "cursor.fetchone()"
I hope that helps
According to the documentation, cursor.execute() returns the number of affected rows. You then need to fetch the content with fetch() or fetchall(). See the example at PyMySQL.
I'm working on a bit of python code to run a query against a redshift (postgres) SQL database, and I'm running into an issue where I can't strip off the surrounding single quotes from a variable I'm passing to the query. I'm trying to drop a number of tables from a list. This is the basics of my code:
def func(table_list):
drop_query = 'drop table if exists %s' #loaded from file
table_name = table_list[0] #table_name = 'my_db.my_table'
con=psycopg2.connect(dbname=DB, host=HOST, port=PORT, user=USER, password=PASS)
cur=con.cursor()
cur.execute(drop_query, (table_name, )) #this line is giving me trouble
#cleanup statements for the connection
table_list = ['my_db.my_table']
when func() gets called, I am given the following error:
syntax error at or near "'my_db.my_table'"
LINE 1: drop table if exists 'my_db.my_table...
^
Is there a way I can remove the surrounding single quotes from my list item?
for the time being, I've done it (what think is) the wrong way and used string concatenation, but know this is basically begging for SQL-injection.
This is not how psycopg2 works. You are using a string operator %s to replace with a string. The reason for this is to tokenize your string safely to avoid SQL injection, psycopg2 handles the rest.
You need to modify the query before it gets to the execute statement.
drop_query = 'drop table if exists {}'.format(table_name)
I warn you however, do not allow these table names to be create by outside sources, or you risk SQL injection.
However a new version of PSYCOPG2 kind of allows something similar
http://initd.org/psycopg/docs/sql.html#module-psycopg2.sql
from psycopg2 import sql
cur.execute(
sql.SQL("insert into {} values (%s, %s)").format(sql.Identifier('my_table')),[10, 20]
)
I am using python and trying to update rows using
db=sqlite3.connect('db')
cursor=db.execute('select * from infos where is_processed=0')
films=cursor.fetchall()
cursor.close()
db.close()
for film in films:
inputLayer=np.array([film[1],film[2],film[3]],dtype=float)
name=film[0]
#print inputLayer
NeuralNetwork3.nn(inputLayer,film[4])
sql="update infos set is_processed=1 where file_name='"+name+"'"
db = sqlite3.connect('db')
db.execute(sql)
db.commit()
db.close()
I get: sqlite3.OperationalError: near "t": syntax error what is wrong?
Note it points at line "db.excute(sql)" and says at that line is the error
Suppose name contains a single quote followed by a t, as in
name = "don't look now"
sql = "update foo set is_processed=1 where bar='"+name+"'"
Then sql would equal
In [156]: sql
Out[156]: "update foo set is_processed=1 where bar='don't look now'"
and sqlite3 will think the conditional is where bar='don' followed by a syntax error, t look now'. sqlite3 then raises
sqlite3.OperationalError: near "t": syntax error
This is an example of why you should always use parametrized SQL. To avoid this problem (and protect your code from SQL injection attacks), use parametrized SQL and pass a sequence (or, depending on the paramstyle, a mapping) of values as the second argument to cursor.execute:
sql = "update foo set is_processed=1 where bar=?"
cursor.execute(sql, [name])
When you pass arguments (such as [name]) as the second argument to
cursor.execute, sqlite3 will escape the single-quote for you.
Per the Python Database API, when you pass parameters as the second argument to cursor.execute (my emphasis):
The module will use the __getitem__ method of the parameters object to map
either positions (integers) or names (strings) to parameter values. This
allows for both sequences and mappings to be used as input.
The term bound refers to the process of binding an input value to a database
execution buffer. In practical terms, this means that the input value is
directly used as a value in the operation. The client should not be required
to "escape" the value so that it can be used — the value should be equal to
the actual database value
Here is a runnable toy example to help see the problem and how it is avoided using parametrized SQL:
import sqlite3
with sqlite3.connect(':memory:') as conn:
cursor = conn.cursor()
cursor.execute('''CREATE TABLE foo
(id INTEGER PRIMARY KEY AUTOINCREMENT,
bar TEXT,
is_processed BOOL)''')
name = "don't look now"
sql = "update foo set is_processed=1 where bar='"+name+"'"
print(sql)
cursor.execute(sql)
# comment out `cursor.execute(sql)` above and compare with
# sql = "update foo set is_processed=1 where bar=?"
# cursor.execute(sql, [name])
I am trying to insert a CLOB using the following code.
cursor = connection.cursor()
cursor.setinputsizes(HERP = cx_Oracle.CLOB)
cursor.execute("INSERT INTO myTable (FOO, BAR) VALUES (:FOO, :BAR)", FOO=val1, BAR=val2)
cursor.execute("INSERT INTO myTable2 (HERP) VALUES (:HERP)", HERP=val3)
#len(HERP) 39097
When I run the script WITHOUT cursor.setinputsizes(HERP = cx_Oracle.CLOB) it fails on the second query WITH ValueError: string data too large, when I run the script with cursor.setinputsizes(HERP = cx_Oracle.CLOB) it fails on the first query with DatabaseError: ORA-01036: illegal variable name/number. The CLOB I am trying to insert contains a code snippet (i.e. it has a lot of semi colons, commas and parenthesis), "string".decode("ascii") returns u'string', so unicode isn't the problem... right? I don't know if either of these things are problems. The field in the database is a currently a CLOB, however I have tried it with an NCLOB and the behavior did not change.
I have also tried the field as a BLOB and then used .encode("hex") on the value I was inserting, again same behavior.
I have also tried HERP = cursor.var(cx_Oracle.CLOB) instead of cursor.setinputsizes(HERP = cx_Oracle.CLOB), same issues.
I've been going through the discussions on the cx-oracle-users list, but no luck yet.
It works if I use this line cursor.execute("INSERT INTO myTable2 (HERP) VALUES (:HERP)", HERP="".join(set(val3)).encode("hex")), so I don't think it's an issue with the data's content (This is with the BLOB).
How can I use cx_Oracle to insert a CLOB into an Oracle database?
There are a few possible solutions:
Upgrade cx_Oracle. I'm not sure which version you're using, but I'm using python 2.7.2 with cx_Oracle 5.1 and I don't get any errors when inserting 150,000 chars into a CLOB column without using setinputsizes at all.
Since setinputsizes applies to every subsequent cursor use, just change it between these different cursor.execute statements.
eg:
cursor = connection.cursor()
cursor.setinputsizes(FOO=None, BAR=None)
cursor.execute("INSERT INTO myTable (FOO, BAR) VALUES (:FOO, :BAR)",
FOO=val1, BAR=val2)
cursor.setinputsizes(HERP = cx_Oracle.CLOB)
cursor.execute("INSERT INTO myTable2 (HERP) VALUES (:HERP)", HERP=val3)