From the PostgreSQL documentation if you do a INSERT without specifying a schema it should be a public schema.
conn = psycopg2.connect(dbname = 'orion',
host = 'localhost',
port = 5432,
user = 'earthling',
password = 'mysupersecretpassword')
sql = conn.cursor()
def INSERT(table, info, text):
date = datetime.date.today()
query = "INSERT INTO %s (info, text, date) " \
"VALUES (%s, %s, %s)" %(table, info, text, date)
sql.execute(query)
INSERT("main", "www.capecod.edu", "test")
For some reason I'm seeing the following error?
psycopg2.ProgrammingError: schema "www" does not exist
You are using string interpolation to create the query. This is what psycopg2 executes:
INSERT INTO main (info, text, date)
VALUES (www.capecod.edu, test, 2015-09-12)
If it's not obvious what's wrong here, it's that none of the values are quoted. Here is the properly quoted version:
INSERT INTO main (info, text, date)
VALUES ('www.capecod.edu', 'test', '2015-09-12')
The error is caused by the unquoted www.capecod.edu. Due to the dots, it's being interpreted as schema.table.column.
The "right" way to do this is with a parameterized query.
query = "INSERT INTO main (info, text, date) VALUES (%s, %s, %s)"
params = (info, text, date)
sql.execute(query, params)
psycopg2 will figure out what should be quoted and how. This is a safer option than simply interpolating the string yourself, which often leaves you open to SQL injection attack.
http://initd.org/psycopg/articles/2012/10/01/prepared-statements-psycopg/
Unfortunately, you can't just toss identifiers such as the table name in as a parameter, because then they are quoted as string values, which is bad SQL syntax. I found an answer (python adds "E" to string) that points to psycopg2.extensions.AsIs as a way to pass identifiers such as table names safely as parameters. I wasn't able to make this work in my testing, though.
If you go the AsIs route, you should be cautious about checking the table names are valid, if they somehow come from user input. Something like
valid_tables = ["main", "foo", "bar", "baz"]
if table not in valid_tables:
return False
Related
I have a syntax error in my python which which stops MySQLdb from inserting into my database. The SQL insert is below.
cursor.execute("INSERT INTO %s (description, url) VALUES (%s, %s);", (table_name.encode("utf-8"), key.encode("utf-8"), data[key].encode("utf-8")))
I get the following error in my stack trace.
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your
SQL syntax; check the manual that corresponds to your MariaDB server
version for the right syntax to use near ''four' (description, url) VALUES ('', 'http://imgur.com/a/V8sdH')' at line 1")
I would really appreciate assistance as I cannot figure this out.
EDIT:
Fixed it with the following line:
cursor.execute("INSERT INTO " + table_name + " (description, url) VALUES (%s, %s);", (key.encode("utf-8"), data[key].encode("utf-8")))
Not the most sophisticated, but I hope to use it as a jumping off point.
It looks like this is your SQL statement:
cursor.execute("INSERT INTO %s (description, url) VALUES (%s, %s);", (table_name.encode("utf-8"), key.encode("utf-8"), data[key].encode("utf-8")))
IIRC, the name of the table is not able to be parameterized (because it gets quoted improperly). You'll need to inject that into the string some other way (preferably safely -- by checking that the table name requested matches a whitelisted set of table names)... e.g.:
_TABLE_NAME_WHITELIST = frozenset(['four'])
...
if table_name not in _TABLE_NAME_WHITELIST:
raise Exception('Probably better to define a specific exception for this...')
cursor.execute("INSERT INTO {table_name} (description, url) VALUES (%s, %s);".format(table_name=table_name),
(table_name.encode("utf-8"),
key.encode("utf-8"),
data[key].encode("utf-8")))
First-time question. I am writing a Python application for personal use, which reads metadata from MP3 files I've collected on CD-ROMs and inserts it into a MySQL database. At least, it should--but when it comes to the actual INSERT statement, the program is throwing a ProgrammingError: "1064: You have an error in your SQL syntax," etc.
In trying to write this program, I've learned how to create parameterized queries, instantiate the cursor object with cursor_class = MySQLCursorPrepared, and instantiate the database connection with use_pure = True. I've searched the Web for similar problems but come up dry.
Here is the offending code (it's the cursor.execute line specifically that throws the exception; for debugging purposes I've temporarily removed the try/except blocks):
table = "mp3_t"
# Parameterized query for SQL INSERT statement
query = '''
INSERT INTO %s
(track_num, title, artist, album, album_year, genre, discname)
VALUES
(%s, '%s', '%s', '%s', %s, '%s', '%s')
'''
conn = self.opendb(self.config)
cursor = conn.cursor(cursor_class = MySQLCursorPrepared)
for track in tracklist:
print("Counter: {}".format(counter))
# Tuple for parameterized query
input = (table, track['track_num'], track['title'],
track['artist'], track['album'], track['album_year'],
track['genre'], track['discname'])
print(query % input) # What is the actual query?
cursor.execute(query, input)
The database table is defined with the following SQL:
CREATE TABLE mp3_t (
id int NOT NULL PRIMARY KEY AUTO_INCREMENT,
track_num int NOT NULL,
title VARCHAR(255) NOT NULL,
artist VARCHAR(255) NOT NULL,
album VARCHAR(255) NOT NULL,
album_year int NOT NULL,
genre VARCHAR(255) NOT NULL,
discname VARCHAR(255) NOT NULL);
For debugging, I've got a print statement that outputs the query that's being sent to MySQL (the error message is particularly unhelpful for trying to pinpoint the cause of the problem), for example:
INSERT INTO mp3_t
(track_num, title, artist, album, album_year, genre, discname)
VALUES
(1, 'Moribund the Burgermeister', 'Peter Gabriel', 'I', 1977, 'Rock', 'Rock 19')
I don't see any error visually, and if I paste directly into the MySQL CLI and add the required semicolon, it inserts a row into the table as expected.
I'm completely stymied where the problem lies.
If it's any help, I'm running Python 3.6.7 and MariaDB 10.1.37 with Connector/Python 8.0.15, on Ubuntu 18.04 64-bit.
Table name should not be replaced by %s. I think your error message should like:
You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near "mp3_t"
So just use table name in your query template.
INSERT INTO mp3_t
(track_num, title, artist, album, album_year, genre, discname)
VALUES
(%s, %s, %s, %s, %s, %s, %s)
If you want to know the query will be executed, you should use these codes:
conn = connect()
cur = conn.cursor()
print(query % cur._get_db().literal(params))
Don't use %s for table name. Use the table name directly.
I'm inserting data using execute_values, which takes a sql query. The query is constructed using psycopg2.sql.SQL as recommended in the documentation, but execute_values won't take that object.
Here's the code I have:
import psycopg2 as pg
from psycopg2 import extras
from psycopg2 import sql
config = {
'host' : 'localhost',
'user' : 'username',
'password' : 'password',
'dbname' : 'myDatabase'
}
connection = pg.connect(**config)
cursor = connection.cursor()
tableName = 'myTable'
dataset = [[1,2],[3,4],[5,6]]
queryText = "INSERT INTO {table} (uid,value) VALUES %s"
query = sql.SQL(queryText).format(table=sql.Identifier(tableName))
extras.execute_values(cursor,query,dataset)
The last line gives the following error:
AttributeError: 'Composed' object has no attribute 'encode'
If the query is specified directly as a string, as below, then the execution runs.
query = """INSERT INTO "myTable" (uid,value) VALUES %s"""
It's possible to insert the table name into the query using string format, but apparently that shouldn't be done, even at gunpoint. How can I safely insert a variable table name into the query and use execute_values? I can't find a built-in way to convert the SQL object to a string.
The parameter sql in execute_values(cur, sql, argslist, template=None, page_size=100) is supposed to be a string:
sql – the query to execute. It must contain a single %s placeholder, which will be replaced by a VALUES list. Example: "INSERT INTO mytable (id, f1, f2) VALUES %s".
Use the as_string(context) method:
extras.execute_values(cursor, query.as_string(cursor), dataset)
connection.commit()
As execute_values() expect the sql statement to be a string you can simply user:
queryText = "INSERT INTO {table} (uid,value) VALUES %s".format(table=sql.Identifier(tableName)
I am trying to take a dataframe and import it into a postgresql database. I've done this before and can always see the data in the database using pgAdmin. However, this time, there's no data visable in pgAdmin.
This is the SQL code I've used to create my table in the postgresql database:
CREATE TABLE Twitterapp
(
tweetid character varying(255),
tweettext character varying(255),
tweetretweetct integer,
tweetfavoritect integer,
tweetsource character varying(255),
tweetcreated character varying(255),
userid character varying(255),
userscreen character varying(255),
username character varying(255),
usercreatedt character varying(255),
user_desc character varying(255),
userfollowerct integer,
userfriendsct integer,
userlocation character varying(255),
usertimezone character varying(255)
)
WITH (
OIDS=FALSE
);
Then, using the sqlalchemy library in python, I tried to push my dataframe to the database:
#First attempt to load data
engine =create_engine('postgresql://username:password#link:5432/db_name')
conn = engine.connect()
a.to_sql('public.twitterapp', con=conn, if_exists='replace',
index=False)
Since that operation was successful, I try to read a table from the database to check sure it worked:
##call data from database
result_set = engine.execute("SELECT * FROM twitterapp")
for r in result_set:
print(r)
This code successfully returns the data in my dataframe but I still can't see it in my pgAdmin database. So I tried this second approach:
b = a.values.tolist()
sql2 = """INSERT INTO twitterapp VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);"""
conn = psycopg2.connect("dbname='db_name' user='username' host='link' password='password'")
c = conn.cursor()
for row in b:
c.execute(sql2, row)
print(row)
conn.commit()
Again, it successfully executes (ie, doesn't throw an error) but no data shows in pgAdmin.
What am I doing wrong?!
If you can show data during the session but not from pgAdmin (a different session) it's sure that your mistake is a wrong commit.
I'm not an expert of sqlalchemy but I think you have to read how to manage the session and how to commit it.
This is a good link:
http://docs.sqlalchemy.org/en/latest/orm/session_basics.html
You have to create a session object and execute a commit command:
session.commit()
You should check the port in psql if it's right or not. In my case local postgresql was listening to port 5433
I am trying to insert some data into a database using the variable test as the table name. But unfortunately I cant seem to achieve this. Can anyone help me out?
From my raise I am getting:
TypeError: unsupported operand type(s) for %: 'tuple' and 'tuple'
My code:
test = "hello"
# Open database connection
db = MySQLdb.connect("127.0.0.1","admin","password","table" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
# Prepare SQL query to INSERT a record into the database.
sql = ("""INSERT INTO %s (name,
age, gender)
VALUES (%s, %s, %s)""",(test))
try:
# Execute the SQL command
cursor.execute(sql, (name, age, gender))
db.commit()
except:
raise
db.rollback()
# disconnect from server
db.close()
I don't think MySQLdb lets you use parameters for table names -- usually this is used for actual parameters (ones that are sometimes from user input and need sanitization - the name/age/gender part gets this right). You could use Python's string formats to achieve this:
sql = ("""INSERT INTO {table} (name, age, gender)
VALUES (%s, %s, %s)""".format(table=table), (test))
Something like this will work:
sql = """INSERT INTO %s (name, age, gender)
VALUES (%s, %s, %s)""" % (test, "%s", "%s", "%s")
You need to separate Python's string substitution from MySQL's parameter substitution. The above is a crude approach, but minimally different from your own code.