I'm totally new to Python and SQL but I'm following a tutorial and I really can't get this working. Please ignore the poor string construct but I'll tidy it up when it's working. I'm trying to send two values to insert into a new row on a table, they get passed to it in the cursor execute command.
def update_height(conn, location, timestamp_height):
sql_update_height_table = """ INSERT INTO """ + str(location) + """_cushion(timestamp,height)
VALUES(?,?) """
print (sql_update_height_table)
cur = conn.cursor()
cur.execute(sql_update_height_table,timestamp_height)
conn.commit()
return cur.lastrowid
Result of print(sql_update_height_table):
INSERT INTO 0xEE738a9e_height(timestamp,height)
VALUES(?,?)
and this is the error:
sqlite3.OperationalError: near "0xEE738a9e": syntax error
I'm following this tutorial and I really can't see what I'm doing wrong, I've been looking for hours.
timestamp_height is a tuple with two entries.
I don't know why you need to use a table name like this, but 0x means as usually a HEX representation. Quote the name with backticks.
sql_update_height_table = """ INSERT INTO `""" + str(location) + """_cushion`(timestamp,height)
VALUES(?,?) """
Related
I'm writing a program to extract a lot of data from another source and record it in a Postgres database. I need a function that takes in the destination table and a dictionary with variable fields to be added and then inserts it as appropriate. It seems like it should be simple enough, but I'm running into problems generating the insert query. The examples I've found online are either partial, outdated, or simply don't work when I modify them for my data.
Here's a simple version I've put together to work it out. I've tried a lot of variations of this, so it's probably not as clean as it should be at this point. It feels like there's something really simple that I'm just missing, but if so I'm just not seeing it.
def insert_record():
table = "test"
record = {"name": "Jack", "id": 1}
fields = record.keys()
values = ", ".join(str(n) for n in record.values())
query = sql.SQL("INSERT INTO {} ({}) VALUES ({});".format(
sql.Identifier(table),
sql.SQL(",").join(map(sql.Identifier, fields)),
sql.SQL(",").join(sql.Placeholder() * len(fields))
))
cursor = connection.cursor()
print(query.as_string(connection))
try:
cursor.execute(query, (values,))
connection.commit()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
cursor.close()
This returns the error:
syntax error at or near "'test'"
LINE 1: INSERT INTO Identifier('test') (Composed([Identifier('name')...
It looks like it's not actually formatting the query for whatever reason, since the as_string function also returns the unformatted:
"INSERT INTO Identifier('test') (Composed([Identifier('name'), SQL(','), Identifier('id')])) VALUES (Composed([Placeholder(''), SQL(','), Placeholder('')]));"
Any suggestions on how to fix this, or better ways to handle dynamic queries in general?
edit: Here's my import statement
import psycopg2
from psycopg2 import extras, Error, sql
You are calling the function .format() from the object string, you must call the .format() function from sql.SQL() object class.
query = sql.SQL("INSERT INTO {} ({}) VALUES ({});").format(
sql.Identifier(table),
sql.SQL(",").join(map(sql.Identifier, fields)),
sql.SQL(",").join(sql.Placeholder() * len(fields))
)
Ref: https://www.psycopg.org/docs/sql.html?highlight=literal#module-usage
I'm working on a bit of python code to run a query against a redshift (postgres) SQL database, and I'm running into an issue where I can't strip off the surrounding single quotes from a variable I'm passing to the query. I'm trying to drop a number of tables from a list. This is the basics of my code:
def func(table_list):
drop_query = 'drop table if exists %s' #loaded from file
table_name = table_list[0] #table_name = 'my_db.my_table'
con=psycopg2.connect(dbname=DB, host=HOST, port=PORT, user=USER, password=PASS)
cur=con.cursor()
cur.execute(drop_query, (table_name, )) #this line is giving me trouble
#cleanup statements for the connection
table_list = ['my_db.my_table']
when func() gets called, I am given the following error:
syntax error at or near "'my_db.my_table'"
LINE 1: drop table if exists 'my_db.my_table...
^
Is there a way I can remove the surrounding single quotes from my list item?
for the time being, I've done it (what think is) the wrong way and used string concatenation, but know this is basically begging for SQL-injection.
This is not how psycopg2 works. You are using a string operator %s to replace with a string. The reason for this is to tokenize your string safely to avoid SQL injection, psycopg2 handles the rest.
You need to modify the query before it gets to the execute statement.
drop_query = 'drop table if exists {}'.format(table_name)
I warn you however, do not allow these table names to be create by outside sources, or you risk SQL injection.
However a new version of PSYCOPG2 kind of allows something similar
http://initd.org/psycopg/docs/sql.html#module-psycopg2.sql
from psycopg2 import sql
cur.execute(
sql.SQL("insert into {} values (%s, %s)").format(sql.Identifier('my_table')),[10, 20]
)
i'm using sqlite to organize data fetched with a scraper script, having trouble executing an 'insert into' command.
I'm making a scraper for an electronics website as a newbie Python enthusiast.
I already had a working script that would scrape all the pages up until when i decided to modify the code to create a new column with prices and name that column using today's date.
Now for some reason the SQL command that inserts the data to the table refuses to execute with the new column i add.
Tried adding the new column to the SQL command with ? method and .format() method with no success.
Tried all kinds for ' locations around the ?s and the {}s.
This is the Code:
class Product:
def __init__(self, prodId, title, price=None):
self.prodId = prodId
self.title = title
self.price = price
self.currDatePriceCloumnName = date + 'Price'
def insertToTable(self):
self.addColumn()
conn = sqlite3.connect(databaseName)
c = conn.cursor()
c.execute("insert into {} (?,?,?) values (?,?,?)".format(table),('Product_ID','Title',str(self.currDatePriceCloumnName),str(self.prodId),str(self.title),str(self.price)))
conn.commit()
conn.close()
def addColumn(self):
conn = sqlite3.connect(databaseName)
c = conn.cursor()
try:
c.execute("alter table {} add column '?'".format(table),(str(self.currDatePriceCloumnName),))
conn.commit()
conn.close()
except:
pass
i expected the c.execute in the insertToTable to insert the data to the table, but what i get is this error:
File "/home/sergio/Desktop/test/scraper.py", line 67, in insertToTable
c.execute("insert into {} (?,?,?) values (?,?,?)".format(table),('Product_ID','Title',str(self.currDatePriceCloumnName),str(self.prodId),str(self.title),str(self.price)))
sqlite3.OperationalError: near "?": syntax error
The weird thing is that the the column is created but not filled.
When i used the .format() method, the error had the desired column name instead of the ?, this tells me that the issue might be with the fact im using self.currDatePriceCloumnName but i'm stuck from here.
Please help..
Thanks in advance! =]
You have a typo in:
c.execute("insert into {} (?,?,?) values (?,?,?)".format(table),('Product_ID','Title',str(self.currDatePriceCloumnName),str(self.prodId),str(self.title),str(self.price)))
At:
values (?,?,?)".format(table),
The end of .format(table) is all you have inserted into the string. The extra ) is causing the .format() to end throwing a syntax error because it's not expecting a ,. You're not passing any other values.
That being said don't use string formatting with SQL statements in the future (as a general side note) as it is bad practice for security purposes:
https://www.w3schools.com/sql/sql_injection.asp
I am receiving an error when trying to write data to a database table when using a variable for the table name that I do not get when using a static name. For some reason on the line where I insert, if I insert an integer as the column values the code runs and the table is filled, however, if I try to use a string I get a SQL syntax error
cursor = db.cursor()
cursor.execute('DROP TABLE IF EXISTS %s' %data[1])
sql ="""CREATE TABLE %s (IP TEXT, AVAILIBILITY INT)""" %data[1]
cursor.execute(sql)
for key in data[0]:
cur_ip = key.split(".")[3]
cursor.execute("""INSERT INTO %s VALUES (%s,%s)""" %(data[1],key,data[0][key]))
db.commit()
the problem is where I have %(data[1], key, data[0][key]) any ideas?
It's a little hard to analyse your problem when you don't post the actual error, and since we have to guess what your data actually is. But some general points as advise:
Using a dynamic table name is often not way DB-systems want to be used. Try thinking if the problem could be used by using a static table name and adding an additional key column to your table. Into that field you can put what you did now as a dynamic table name. This way the DB might be able to better optimize your queries, and your queries are less likely to get errors (no need to create extra tables on the fly for once, which is not a cheap thing to do. Also you would not have a need for dynamic DROP TABLE queries, which could be a security risk.
So my advice to solve your problem would be to actually work around it by trying to get rid of dynamic table names altogether.
Another problem you have is that you are using python string formatting and not parameters to the query itself. That is a security problem in itself (SQL-Injections), but also is the problem of your syntax error. When you use numbers, your expression evaluates to
INSERT INTO table_name VALUES (100, 200)
Which is valid SQL. But with strings you get
INSERT INTO table_name VALUES (Some Text, some more text)
which is not valid (since you have no quotes ' around the strings.
To get rid of your syntax problem and of the sql-injection-problem, don't add the values to the string, pass them as a list to execute():
cursor.execute("INSERT INTO table_name VALUES (%s,%s)", (key, data[0][key]))
If you must have a dynamic table name, put that in your query string first (e.g. with % formatting), and give the actual values for your query as parameters as above (since I cannot imagine that execute will accept the table name as a parameter).
To put it in some simple sample code. Right now you are trying to do it like this:
# don't do this, this won't even work!
table_name = 'some_table'
user_name = 'Peter Smith'
user_age = 47
query = "INSERT INTO %s VALUES (%s, %s)" % (table_name, user_name, user_age)
cursor.execute(query)
That creates query
INSERT INTO some_table VALUES (Peter Smith, 100)
Which cannot work, because of the unquoted string. So you needed to do:
# DON'T DO THIS, it's bad!
query = "INSERT INTO %s VALUES ('%s', %s)" % (table_name, user_name, user_age)
That's not a good idea, because you need to know where to put quotes and where not (which you will mess up at some point). Even worse, imagine a user named named Connor O'Neal. You would get a syntax error:
INSERT INTO some_table VALUES ('Connor O'Neal', 100)
(This is also the way sql-injections are used to crush your system / steal your data). So you would also need to take care of escaping the values that are strings. Getting more complicated.
Leave those problems to python and mysql, by passing the date (not the table name) as arguments to execute!
table_name = 'some_table'
user_name = 'Peter Smith'
user_age = 47
query = "INSERT INTO " + table_name + " VALUES (%s, %s)"
cursor.execute(query, (user_name, user_age))
This way you can even pass datetime objects directly. There are other ways to put the data than using %s, take a look at this examples http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html (that is python3 used there, I don't know which you use - but except of the print statements it should work with python2 as well, I think).
I have the same question as asked here:
Default value for empty integer fields when importing CSV data in MySQL
I keep getting the warning "Incorrect Integer value" when importing a csv file into Mysql. I've read all the relevant questions/answers here, and the link above is my direct question. But I'm trying to implement the answer given there in Python 2.7, and I can't seem to get it working.
My code below is my attempt to implement the answer from the above post. I think the issue is the syntax for using the "Set" clause in my Load DATA Local Infile statement...I would really appreciate any help since MySQL automatically converts empty INT's to "0" and that would mess up my analysis since I want blanks to be null. My code is:
import MySQLdb
db = MySQLdb.connect(blah, blah, blah)
cursor = db.cursor()
sql = """CREATE TABLE PLAYER(
PLAYER_ID INT,
DATE DATETIME,
BATTING_AVERAGE INT
)"""
cursor.execute(sql)
statement = """LOAD DATA LOCAL INFILE 'path/file.csv'
INTO TABLE PLAYER
COLUMNS TERMINATED BY ','
SET BATTING_AVERAGE = IF(#BATTING_AVERAGE='',NULL,#BATTING_AVERAGE)
IGNORE 1 LINES;"""
The error that this code gives is:
ProgramingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right synatx to use near 'IGNORE 1 Lines at line 5")
The SET and IGNORE clauses are backwards. Try swapping them:
statement = """LOAD DATA LOCAL INFILE 'path/file.csv'
INTO TABLE PLAYER
COLUMNS TERMINATED BY ','
IGNORE 1 LINES
SET BATTING_AVERAGE = IF(#BATTING_AVERAGE='',NULL,#BATTING_AVERAGE);"""