I am using pymyql/mysql-connector to write the messages to mysql database. The messages are processed on callback (paho.mqtt callback) from mqtt broker.I have 4 different tables and based on the message type, I am inserting messages into database. I have written the insert queries as below. this way of writing leads to sql injections it seems.Any suggestions how can I improve the insert query statements?
# callback attached to paho.mqtt.client
def on_message(self, client, userdata, msg):
if msg.topic.startswith("topic1/"):
self.bulkpayload += "(" + msg.payload.decode("utf-8") + "," + datetime + "),"
elif msg.topic.startswith("topic2/"):
self.insertStatement += "INSERT INTO mydatabase.table1 VALUES (" + msg.payload.decode("utf-8") + "," + datetime + ");"
elif msg.topic.startswith("topic3/")
self.insertStatement += "INSERT INTO mydatabase.table2 VALUES (" +msg.payload.decode("utf-8") + "," + datetime + ");"
elif msg.topic.startswith("messages"):
self.insertStatement += "INSERT INTO mydatabase.table3 VALUES ('" + msg.topic + "'," + msg.payload.decode("utf-8") + "," + datetime + ");"
else:
return # do not store in DB
cursor.execute(self.insertStatement)
cursor.commit()
Make your query use parameters. Much less chance of injection:
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
credit (and more info) here: How to use variables in SQL statement in Python?
Also, Dan Bracuk is correct - make sure you validate your params before executing the SQL if you aren't already
Related
conn = sqlite3.connect('business_database.db')
c = conn.cursor()
c.execute("INSERT INTO business VALUES(self.nob_text_input.text, self.post_text_input.text, self.descrip_text_input.text )")
conn.commit()
conn.close()
I want to add records into my database using the TextInput in kivy hence the 'self.post_text_input.text' etc, but I get this error:
OperationalError: no such column: self.nob_text_input.text
I tried putting the columns next to table name in the query:
c.execute("INSERT INTO business(column1, column2,column3) VALUES(self.nob_text_input.text....)
But I still get the same error.
Turning my comment into a more detailed answer.
If you're trying to use the values of the variables (self.nob_text_input.text and friends) in the string, you need to embed those values in the string.
One way is to use a format string:
"INSERT INTO business VALUES(%s, %s, %s)" % (self.nob_text_input.text, self.post_text_input.text, self.descrip_text_input.text)
And another is to just concatenate the strings:
"INSERT INTO business VALUES(" + self.nob_text_input.text + ", " + self.post_text_input.text + ", " + self.descrip_text_input.text + ")"
I'm inserting data into SQLite using Python
database_tasks.bookId is a class variable
def insert(self, conn,dbname, name, writer='none'):
c = conn.cursor()
if dbname == "Book_database":
database_tasks.bookId += 1
sql= "INSERT INTO Book_database VALUES(" + database_tasks.bookId + "," + name + "," + writer + ")"
else:
database_tasks.studentId += 1
sql= "INSERT INTO Student VALUES("+ database_tasks.studentId + "," + name + ")"
c.execute(sql)
conn.commit()
return None
I'm getting an error with Insert command.
sql= "INSERT INTO Book_database VALUES(" + database_tasks.bookId + "," + name + "," + writer + ")"
TypeError: can only concatenate str (not "int") to str
You should never build a query string including values to insert that way but used parameterized queries. It is an anti-pattern that has been used for decades to build SQL injection attacks. You should use:
if dbname == "Book_database":
database_tasks.bookId += 1
c.execute("INSERT INTO Book_database VALUES(?,?,?)",
(database_tasks.bookId, name, writer))
else:
...
it looks like your bookId is int. Cast it to str before combining.
sql= "INSERT INTO Book_database VALUES(" + str(database_tasks.bookId) + "," + name + "," + writer + ")"
The error is fairly self explanatory - you are trying to concatenate an integer to a string.
my assumption is that database_tasks.studentId is an int, so changing your query to
sql= "INSERT INTO Book_database VALUES(" + str(database_tasks.bookId) + "," + name + "," + writer + ")"
would fix this, however be aware that if this is public facing, it would be very easy to exploit this, I'd recommend fully validating input, or changing your approach to this problem.
I am trying to insert a row into my postgresql database with a table created from
CREATE TABLE public.coinbase_btc_usd
(
id bigserial primary key,
price integer NOT NULL,
buy integer NOT NULL,
sell integer NOT NULL,
"timestamp" timestamp with time zone
)
However when my python 3.6 script runs and tries to add a row using psycopg2 like this it returns an error saying "no results to fetch" and nothing is added to my db.
sql_query = "INSERT INTO coinbase_btc_usd(price, buy, sell, timestamp)" \
" VALUES (" + exchange_rate + ', ' + buy_rate + ', ' + sell_rate + ", \'2015-10-10 06:44:33.8672177\')"
print(sql_query)
cur.execute(sql_query)
I also printed the sql_query variable to see exactly what was getting attempted to execute and this was printed to the output
INSERT INTO coinbase_btc_usd(price, buy, sell, timestamp) VALUES (16392.10, 16563.40, 16235.42, '2015-10-10 06:44:33.8672177')
Make sure that you are committing the transaction:
cur.execute(sql_query)
conn.commit()
Or you can enable auto commit to commit each query immediately after execution:
conn.autocommit = True
Furthermore, it costs nothing to prevent SQL injection attack - just use parametersied queries. In fact your code will actually be cleaner as well as safer:
sql_query = "INSERT INTO coinbase_btc_usd(price, buy, sell, timestamp) VALUES (%s, %s, %s, %s)"
cur.execute(sql_query, (exchange_rate, buy_rate, sell_rate, timestamp))
conn.commit()
change the
sql_query = "INSERT INTO coinbase_btc_usd(price, buy, sell, timestamp)" \
" VALUES (" + exchange_rate + ', ' + buy_rate + ', ' + sell_rate + ", \'2015-10-10 06:44:33.8672177\')"
to:
sql_query = "INSERT INTO coinbase_btc_usd(price, buy, sell, timestamp)" \
" VALUES (" + exchange_rate + ', ' + buy_rate + ', ' + sell_rate + ", \'2015-10-10 06:44:33.8672177\') returning *"
this should fix no results to fetch in my assumption.
If you see no row added, you most probably begin transaction and never commit it.
When trying to execute the following:
def postToMySQL(date,data,date_column_name,data_column_name,table):
cursor = conn.cursor ()
sql = "\"\"\"INSERT INTO " + table + " (" + date_column_name + ", " + data_column_name + ") VALUES(%s, %s)" + "\"\"\"" #+ ", " + "(" + date + ", " + data + ")"
cursor.execute(sql,(date,data))
I get this error:
_mysql_exceptions.ProgrammingError: (1064, 'You have an error in your SQL syntax... near:
\'"""INSERT INTO natgas (Date, UK) VALUES(\'2012-05-01 13:00:34\', \'59.900\')"""\' at line 1')
I'm puzzled as to where the syntax is wrong, because the following hardcoded example works fine:
def postUKnatgastoMySQL(date, UKnatgas):
cursor = conn.cursor ()
cursor.execute("""INSERT INTO natgas (Date, UK)VALUES(%s, %s)""", (date, UKnatgas))
Can you spot the error?
Alternately, could you tell me how to pass parameters to the field list as well as the value list?
Thanks a lot!
Those triple quotes are a way of representing a string in python. They aren't supposed to be part of the actual query.
On another note, be very sure you trust your input with this approach. Look up SQL Injection.
\'"""INSERT INTO natgas (Date, UK) VALUES(\'2012-05-01 13:00:34\',
\'59.900\')"""\' at line 1')
this is obviously not a vlaid SQL command. You need to get the backslashes out of there, you are probably escaping stuff you shouldn't.
the triple quotes for example sure are unnecessary there.
I am new in python, and using Python & PostgreSQL (9.03) (and psycopg2 to interface between the two) in Windows XP environment.
I am working on a huge spatial dataset road network dataset, and seperating the data per Country through ArcGIS Geoprocessing, and automatically store and them in a PostGIS (1.5) Database.
While when retrieving values from the database everything works as planned:
...
try:
conn = psycopg2.connect("host = '" + HostName + "' dbname='" + DBName + "' user='" + Username + "' password='" + Password + "'")
curs = conn.cursor()
except:
print "Unable to connect to the database"
SQLStatement = "SELECT data_partition FROM datasets WHERE map_partition='" + MapPartitions[0] + "'"
curs.execute(SQLStatement)
...
When I am trying to pass the following Union Statement to Postgres, there is no resulting table, while if I take the printed SQL Statement and run it in as an SQL Statement and run it PostgresSQL, it creates the desired resulting table:
conn = psycopg2.connect("host = '" + HostName + "' dbname='" + DBName + "' user='" + Username + "' password='" + Password + "'")
cur = conn.cursor()
SQLStatement = (
"CREATE TABLE " + Schema + "." + PartitionTableName + " AS \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net0 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net1 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net2 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net3 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net4 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net5;\n"
"\n"
"\n"
"ALTER TABLE " + Schema + "." + partName + "_Lines_" + Rel + "\n"
"DROP COLUMN gid;\n"
cur.execute(SQLStatement)
conn.commit()
cur.close()
If we print the SQL Statement, this is the resulting query:
print SQLStatement
CREATE TABLE compresseddata.FRA24_Lines_2011_03 AS
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net0 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net1 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net2 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net3 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net4 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net5;
ALTER TABLE compresseddata.FRA24_Lines_2011_03
DROP COLUMN gid;
I am using variables in the to Merge different Road Network Classes, and due to different Partitions of my dataset, I need to iterate through, them, but for some reason that I cannot still understand, there is no table being produced.
Any ideas?
Thanx in advance for the help
THe SQL you are sending are actually 3 statements, not 1.
I never tried this but I expect execute to complain about this.
Additionally there is a semicolon missing in the ALTER TABLE statement.
I would recommend to add exception handling to your code and execute each SQL statement separately so you get better error reporting on what might go wrong.
Indeed Peter, this seems to be the case.
More specifically Each SQL Statement must be passed separately through:
curs.execute(SQLStatement)
and them committed via:
conn.commit()
All the changes will then be apparent in the database.
Thanx again
As already mentioned, individually executing each statement and checking the exception can provide good insight to what is occurring.
In particular psycopg2 will raise psycopg2.ProgrammingError. If the error message is not useful, you may have better luck looking up the exception's pgcode and then investigating that.
PGCodes for 9.1:
http://www.postgresql.org/docs/9.1/static/errcodes-appendix.html ).
try:
cur.execute(SQLQUERY)
except psycopg2.ProgrammingError as e:
# Err code lookup at http://www.postgresql.org/docs/9.1/static/errcodes-appendix.html
print "psycopg2 error code %s" % e.pgcode
raise e
NOTE: A cursors execute statement CAN take multiple sql statements in a single string.
ex: cur.execute('create table ABBA (); create table BETA ();') is a perfectly legitimate statement.
For this reason, do not expect cursor.execute to perform any sanity checks on a string only input!
I'd suggest (except for special rare circumstances) to execute each statement individually.