I'm trying to execute a basic INSERT statement on a MySQL table from a Python script using MySQLdb. My table looks like this:
CREATE TABLE `testtable` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`testfield` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
)
Running this query from the MySQL command line works fine:
INSERT INTO `testtable` (`id`, `testfield`) VALUES (NULL, 'testvalue');
But when I try to execute the query from a Python script, no rows get inserted. Here's my code:
conn = MySQLdb.connect(host=db_host, port=db_port, user=db_user, passwd=db_password, db=db_database)
cursor = conn.cursor ()
cursor.execute ("INSERT INTO `testtable` (`id`, `testfield`) VALUES (NULL, 'testvalue')")
print "Number of rows inserted: %d" % cursor.rowcount
cursor.close()
conn.close()
Oddly, this will print "Number of rows inserted: 1." I can also confirm that this query increments the ID field, because when I add another row via the command line, the value of its ID is the same as if the Python script had successfully inserted its rows. However, running a SELECT query returns none of the rows from the script.
Any idea what's going wrong?
You either need to set conn.autocommit(), or you need to do conn.commit() - see the FAQ
you need to commit:
conn.commit()
http://mysql-python.sourceforge.net/FAQ.html#my-data-disappeared-or-won-t-go-away
Related
First-time question. I am writing a Python application for personal use, which reads metadata from MP3 files I've collected on CD-ROMs and inserts it into a MySQL database. At least, it should--but when it comes to the actual INSERT statement, the program is throwing a ProgrammingError: "1064: You have an error in your SQL syntax," etc.
In trying to write this program, I've learned how to create parameterized queries, instantiate the cursor object with cursor_class = MySQLCursorPrepared, and instantiate the database connection with use_pure = True. I've searched the Web for similar problems but come up dry.
Here is the offending code (it's the cursor.execute line specifically that throws the exception; for debugging purposes I've temporarily removed the try/except blocks):
table = "mp3_t"
# Parameterized query for SQL INSERT statement
query = '''
INSERT INTO %s
(track_num, title, artist, album, album_year, genre, discname)
VALUES
(%s, '%s', '%s', '%s', %s, '%s', '%s')
'''
conn = self.opendb(self.config)
cursor = conn.cursor(cursor_class = MySQLCursorPrepared)
for track in tracklist:
print("Counter: {}".format(counter))
# Tuple for parameterized query
input = (table, track['track_num'], track['title'],
track['artist'], track['album'], track['album_year'],
track['genre'], track['discname'])
print(query % input) # What is the actual query?
cursor.execute(query, input)
The database table is defined with the following SQL:
CREATE TABLE mp3_t (
id int NOT NULL PRIMARY KEY AUTO_INCREMENT,
track_num int NOT NULL,
title VARCHAR(255) NOT NULL,
artist VARCHAR(255) NOT NULL,
album VARCHAR(255) NOT NULL,
album_year int NOT NULL,
genre VARCHAR(255) NOT NULL,
discname VARCHAR(255) NOT NULL);
For debugging, I've got a print statement that outputs the query that's being sent to MySQL (the error message is particularly unhelpful for trying to pinpoint the cause of the problem), for example:
INSERT INTO mp3_t
(track_num, title, artist, album, album_year, genre, discname)
VALUES
(1, 'Moribund the Burgermeister', 'Peter Gabriel', 'I', 1977, 'Rock', 'Rock 19')
I don't see any error visually, and if I paste directly into the MySQL CLI and add the required semicolon, it inserts a row into the table as expected.
I'm completely stymied where the problem lies.
If it's any help, I'm running Python 3.6.7 and MariaDB 10.1.37 with Connector/Python 8.0.15, on Ubuntu 18.04 64-bit.
Table name should not be replaced by %s. I think your error message should like:
You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near "mp3_t"
So just use table name in your query template.
INSERT INTO mp3_t
(track_num, title, artist, album, album_year, genre, discname)
VALUES
(%s, %s, %s, %s, %s, %s, %s)
If you want to know the query will be executed, you should use these codes:
conn = connect()
cur = conn.cursor()
print(query % cur._get_db().literal(params))
Don't use %s for table name. Use the table name directly.
When I'm using pymysql to perform operations on MySQL database, it seems that all the operations are temporary and only visible to the pymysql connection, which means I can only see the changes through cur.execute('select * from qiushi') and once I cur.close() and conn.close() and log back in using pymysql, everything seems unchanged.
However, when I'm looking at the incremental id numbers, it does increased, but I can't see the rows that were inserted from pymysql connection. It seems that they were automatically deleted?!
Some of my code is here:
import pymysql
try:
conn = pymysql.connect(host='127.0.0.1',port=3306,user='pymysql',passwd='pymysql',charset='utf8')
cur = conn.cursor()
#cur.execute('CREATE TABLE qiushi (id INT NOT NULL AUTO_INCREMENT, content_id BIGINT(10) NOT NULL, content VARCHAR(1000), created TIMESTAMP DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY(id));')
#cur.execute('DESCRIBE content')
#cur.fetchall()
cur.execute('USE qiushibaike')
for _ in range(0,len(content_ids)):
cur.execute("INSERT INTO qiushi (content,content_id) VALUES (\"%s\",%d)"%(jokes[_],int(content_ids[_])))
finally:
cur.close()
conn.close()
I solved the problem by myself...
Because the config is automatically committed, so after each SQL sentence we should commit the changes.
Approach 1:
add cur.commit() after the cur.execute()
Approach 2:
edit the connection config, add autocommit=True
I'm having problems returning auto-incremented ID columns from a MySQL database using MySQLdb python library.
I have something like:
sql = """INSERT INTO %s (%s) VALUES (\"%s\")""" %(tbl, colsf, valsf)
try:
cursor.execute(sql)
id = cursor.lastrowid
db.close()
except:
print "Failed to add to MySQL database: \n%s" %sql
print sys.exc_info()
db.close()
exit()
However the lastrowid command seems to be returning incorrect values. For instance, I've tried printing out various id columns from the MySQL command line which shows them to be empty, but the lastrowid value keeps increasing by 1 every time the python script is run. Any ideas?
Turned out that the values weren't being committed to the MySQL database properly, adding "db.commit()" command seems to solve the problem.
sql = """INSERT INTO %s (%s) VALUES (\"%s\")""" %(tbl, colsf, valsf)
try:
cursor.execute(sql)
id = cursor.lastrowid
cursor.close()
db.commit()
db.close()
except:
print "Failed to add to MySQL database: \n%s" %sql
print sys.exc_info()
db.close()
exit()
I created a database in psql and in it, created a table called "tweet".
CREATE TABLE tweet
( tid CHARACTER VARYING NOT NULL, DATA json,
CONSTRAINT tid_pkey PRIMARY KEY (tid) );
Then when I use
SELECT * FROM tweet;
in the psql window it works and shows an empty table.
Now I have a python script that takes JSON data and is loading it into this table.
conn_string = "host='localhost' port=5432 dbname='tweetsql' user='tweetsql' password='tweetsql'"
conn = psycopg2.connect(conn_string)
cur = conn.cursor()
That sets up the connection and I don't think it had any issues.
Now I have some logic to read in the JSON file and then to add it in, I say:
cur.execute("INSERT INTO tweet (tid, data) VALUES (%s, %s)", (cur_tweet['id'], json.dumps(cur_tweet, cls=DecimalEncoder), ))
But this always says that the relation tweet doesn't exist. Am I missing something here? Is there an issue with my connection or can my script somehow not see the table? For reference I'm using psycopg2 for the connection.
EDIT: I updated the DDL to include a transaction I could commit but that didn't fix it either. Is it a schema issue?
This is what I did regarding the table creation to commit:
BEGIN;
CREATE TABLE tweet
( tid CHARACTER VARYING NOT NULL, DATA json,
CONSTRAINT tid_pkey PRIMARY KEY (tid) );
COMMIT;
EDIT 2: I'm posting some code here...
import psycopg2
import json
import decimal
import os
import ctypes
conn_string = "host='localhost' port=5432 dbname='tweetsql' user='tweetsql' password='tweetsql'"
conn = psycopg2.connect(conn_string)
cur = conn.cursor()
cur.execute("CREATE TABLE tweet (tid CHARACTER VARYING NOT NULL, DATA json, CONSTRAINT tid_pkey PRIMARY KEY (tid) );")
cur.commit()
for file in os.listdir(path):
if not is_hidden(file):
with open(path+file, encoding='utf-8') as json_file:
tweets = json.load(json_file, parse_float=decimal.Decimal)
for cur_tweet in tweets:
cur.execute("INSERT INTO tweet (tid, data) VALUES (%s, %s)", (cur_tweet['id'], json.dumps(cur_tweet, cls=DecimalEncoder), ))
cur.commit()
cur.close()
conn.close()
You're probably not committing the table creation, and, (I'm assuming; not seeing your complete code) you're starting a new connection via psycopg2 each time. You need to commit right after the table creation, and not in a new connection, as each connection is its own implicit transaction. So, your code flow should be something like this:
connect to the db
create the table using the cursor
fill the table
commit and disconnect from db.
Or, if you must separate creation from filling, just commit and disconnect after (2) and then reconnect before (3).
db = MySQLdb.connect("XXXXXXXX","root", "XXXXXX", database)
cursor = db.cursor()
cursor.execute('INSERT INTO media_files (ID, DATA) VALUES ("test", "test")')
cursor.execute("commit")
This statement errors and I cannot seem to figure out why. Any thoughts? The table media_files btw only has the two columns, ID and DATA each of which are VARCHAR(255)
Thanks
Trent
'INSERT INTO media_files (ID, DATA) VALUES ("test", "test")'
Using the parentheses will fix the problem. You were inserting one piece of data but provided 2 fields.