I HAVE ADDED MY OWN ANSWER THAT WORKS BUT OPEN TO IMPROVEMENTS
After seeing a project at datanitro. I took on getting a connection to MySQL (they use SQLite) and I was able to import a small test table into Excel from MySQL.
Inserting new updated data from the Excel sheet was this next task and so far I can get one row to work like so...
import MySQLdb
db = MySQLdb.connect("xxx","xxx","xxx","xxx")
c = db.cursor()
c.execute("""INSERT INTO users (id, username, password, userid, fname, lname)
VALUES (%s, %s, %s, %s, %s, %s);""",
(Cell(5,1).value,Cell(5,2).value,Cell(5,3).value,Cell(5,4).value,Cell(5,5).value,Cell(5,6).value,))
db.commit()
db.close()
...but attempts at multiple rows will fail. I suspect either issues while traversing rows in Excel. Here is what I have so far...
import MySQLdb
db = MySQLdb.connect(host="xxx.com", user="xxx", passwd="xxx", db="xxx")
c = db.cursor()
c.execute("select * from users")
usersss = c.fetchall()
updates = []
row = 2 # starting row
while True:
data = tuple(CellRange((row,1),(row,6)).value)
if data[0]:
if data not in usersss: # new record
updates.append(data)
row += 1
else: # end of table
break
c.executemany("""INSERT INTO users (id, username, password, userid, fname, lname) VALUES (%s, %s, %s, %s, %s, %s)""", updates)
db.commit()
db.close()
...as of now, I don't get any errors, but my new line is not added (id 3). This is what my table looks like in Excel...
The database holds the same structure, minus id 3. There has to be a simpler way to traverse the rows and pull the unique content for INSERT, but after 6 hours trying different things (and 2 new Python books) I am going to ask for help.
If I run either...
print '[%s]' % ', '.join(map(str, updates))
or
print updates
my result is
[]
So this is likely not passing any data to MySQL in the first place.
LATEST UPDATE AND WORKING SCRIPT
Not exactly what I want, but this has worked for me...
c = db.cursor()
row = 2
while Cell(row,1).value != None:
c.execute("""INSERT IGNORE INTO users (id, username, password, userid, fname, lname)
VALUES (%s, %s, %s, %s, %s, %s);""",
(CellRange((row,1),(row,6)).value))
row = row + 1
Here is your problem:
while True:
if data[0]:
...
else:
break
Your first id is 0, so in the first iteration of the loop data[0] will be falsely and your loop will exit, without ever adding any data. What you probably ment is:
while True:
if data[0] is not None:
...
else:
break
I ended up finding a solution that gets me an Insert on new and allows for UPDATE of those that are changed. Not exactly a Python selection based on a single query, but will do.
import MySQLdb
db = MySQLdb.connect("xxx","xxx","xxx","xxx")
c = db.cursor()
row = 2
while Cell(row,1).value is not None:
c.execute("INSERT INTO users (id, username, password, \
userid, fname, lname) \
VALUES (%s, %s, %s, %s, %s, %s) \
ON DUPLICATE KEY UPDATE \
id=VALUES(id), username=VALUES(username), password=VALUES(password), \
userid=VALUES(userid), fname=VALUES(fname), lname=VALUES(lname);",
(CellRange((row,1),(row,6)).value))
row = row + 1
db.commit()
db.close()
Related
I have a python function which should execute 2 SQL queries. I have found that it is impossible to execute 2 queries in one command at once, so as a workaround I created a list of my queries and try to iterate over it with execute command. However nothing is added to MySQL table. Here is the code:
#app.route('/addComment', methods=['POST'])
def addComment():
try:
if session.get('user'):
_description = request.form['description']
_user = session.get('user')
_term_id = request.form['termID']
_time = datetime.now()
operation = ['"INSERT INTO comments (description, user, termID, time) VALUES (%s, %s, %s, %s)", (_description, _user, _term_id, _time)', '"INSERT INTO history (user, term, time) VALUES (%s, %s, %s)", (_user, _term_id, _time)']
conn = mysql.connect()
cursor = conn.cursor()
for item in operation:
cursor.execute()
conn.commit()
data = cursor.fetchall()
if len(data) == 0:
conn.commit()
return json.dumps({'status':'OK'})
else:
return json.dumps({'status':'ERROR'})
except Exception as e:
return json.dumps({'status':'Unauthorized access'})
finally:
cursor.close()
conn.close()
Could you please help me?
Errors in your code lies in the following areas:
A. On iteration sql statement is not passed to execute()
Should be:
for item in operation:
cursor.execute(item)
conn.commit()
B. Invalid parameterization
'"INSERT INTO comments (description, user, termID, time) VALUES (%s, %s, %s, %s)", (_description, _user, _term_id, _time)'
This string statement doesn't apply variables to SQL statement string. Depending on your value types you should decide whether to add ' (apostrophe) or not. More safely would be to pass parameters to .execute() function. Example below.
cursor.execute(
"INSERT INTO comments (description, user, termID, time) VALUES (:description, :user, :term_id, :time)",
description=_description,
user=_user,
term_id=_term_id,
time=_time
)
I am new to Python and having some basic problems. I am trying to insert into an existing MySQL table (holding) from a csv file (test.csv), skip the header row and then return the number of rows inserted. Both tables have identical column headings.
The code below is not inserting the data from the csv file, but just inserting the values (which are my column heading, I thought I needed to declare these in the values), so this method is wrong as it only seems to add the values manually and not inert data from the csv file. Can someone tell me what I'm doing wrong please?
This is the code:
`import csv
try:
mydb = mysql.connector.connect(
host=localhost, port=5306,
user="XX",
passwd="XX",
database="python_test",
auth_plugin='mysql_native_password'
)
# This skips the first row of the CSV file.
with open((r'c:\TELS\\Uploaded\test.csv')) as f:
reader = csv.reader(f)
#next(reader) # skip header
data = [r for r in reader]
data.pop(0) # remove header
mycur = mydb.cursor()
query = "INSERT INTO hold (`Name`, `Address`, `Age`, `DOB) VALUES (%s, %s, %s, %S)"
values = (`Name`, `Address`, `Age`, `DOB`)
mycur.execute(query, values)
mydb.commit()
print(mycur.rowcount, "records inserted.")
#close the connection to the database.
mycur.close()`
The content of your values variable is what will be inserted so you need to map it to your data variable.
You could either do it one row at a time...
query = "INSERT INTO hold (`Name`, `Address`, `Age`, `DOB`) VALUES (%s, %s, %s, %s)"
# also notice how i converted your last %S into %s
for row in data:
mycur.execute(query, row)
mydb.commit()
mycur.close()
Or use the executemany() function :
query = "INSERT INTO hold (`Name`, `Address`, `Age`, `DOB`) VALUES (%s, %s, %s, %s)"
mycur.executemany(query, data)
mydb.commit()
print(mycur.rowcount, "records inserted.")
mycur.close()
I know this question has been asked a number of times, but i am stuck here unable to proceed further. I am executing a for loop in python to load data to fact table.
I am executing the below code
for index, row in df.iterrows():
# get songid and artistid from song and artist tables
cur.execute(song_select, (row.song, row.artist, row.length))
results = cur.fetchone()
if results:
song_id, artist_id = results
else:
song_id, artist_id = None, None
# insert songplay record
songplay_data = (pd.to_datetime(row.ts, unit='ms'),row.userId,row.level,song_id,artist_id,row.sessionId,row.location,row.userAgent)
cur.execute(songplay_table_insert, songplay_data)
conn.commit()
and getting the error
<ipython-input-22-b8b0e27022de> in <module>()
13
14 songplay_data = (pd.to_datetime(row.ts, unit='ms'),row.userId,row.level,song_id,artist_id,row.sessionId,row.location,row.userAgent)
15 cur.execute(songplay_table_insert, songplay_data)
16 conn.commit()
IndexError: tuple index out of range
My table i am trying to insert is
songplay_table_insert = ("""INSERT INTO songplays (songplay_id, start_time,
user_id, level, song_id, artist_id, session_id, location, user_agent )
VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s)
I am really stuck, any help appreciated.
You have one too many %s markers.
VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s)
has 9 markers, while
songplay_data = (pd.to_datetime(row.ts, unit='ms'),row.userId,row.level,song_id,artist_id,row.sessionId,row.location,row.userAgent)
has 8 elements. When it tries to evaluate the last marker, it looks for the 9th element, i.e. songplay_data[8], and that raises the error.
You will also need to remove songplay_id from the SQL to make the INSERT statement valid. The database should be generating the primary key for you if you don't have a value to provide, if not we should take a look at your table definition.
I have been trying to return the values from two different tables, but can't seem to get the c.execute(query) function to return what I want it to. Currently my code will return the first c.fetchone()[0], but the second fetchone()[5] gives an error that it's out of range, which means it is probably still trying to get data from my 'clients' table which does not have 6 columns. I don't think I fully understand how MySQLdb works it's magic, but can't find any good examples of multi-table queries. Here is my code snippet below! Thanks!
c, conn = connection()
#check if already exists
x = c.execute("SELECT * FROM clients WHERE email = (%s)", (thwart(email),))
if int(x) > 0:
flash("That email already has an account, please try a new email or sign in.")
return render_template('register.html', form=form)
else:
c.execute("INSERT INTO clients (email, phone, password) VALUES (%s, %s, %s)", (thwart(email), thwart(phone), thwart(password)))
c.execute("SELECT cid FROM clients WHERE email = (%s)", (thwart(email),))
clientcid = c.fetchone()[0]
c.execute("INSERT INTO cpersonals (first_name, last_name, address, zip) VALUES (%s, %s, %s, %s)", (thwart(first_name), thwart(last_name), thwart(address), czip))
c.execute("SELECT reg_date FROM cpersonals WHERE cid = (%s)", (clientcid,))
reg_date = c.fetchone()[5]
rating = c.execute("SELECT rating FROM clients WHERE email = (%s)", (thwart(email),))
conn.commit()
flash("Thanks for registering!")
c.close()
conn.close()
Your query is SELECT reg_date FROM cpersonals .... You are only selecting one column. The reason fetchone()[5] fails is, there is no 6th column in the fetched record. Try 0 in place of 5.
Why were you using 5?
I am trying to store some TV information in a MySQLdb. I have tried about everything and I cannot get the variables to post. There is information in the variables as I am able to print the information.
My Code:
import pytvmaze
import MySQLdb
AddShow = pytvmaze.get_show(show_name='dexter')
MazeID = AddShow.maze_id
ShowName = "Show" + str(MazeID)
show = pytvmaze.get_show(MazeID, embed='episodes')
db = MySQLdb.connect("localhost","root","XXXXXXX","TVshows" )
cursor = db.cursor()
for episode in show.episodes:
Show = show.name
ShowStatus = show.status
ShowSummary = show.summary
Updated = show.updated
Season = episode.season_number
Episode = episode.episode_number
Title = episode.title
AirDate = episode.airdate
ShowUpdate = show.updated
EpisodeSummary = episode.summary
try:
sql = "INSERT INTO " + ShowName + " VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)""" (Show,ShowStatus,ShowSummary,Updated,Season,Episode,Title,AirDate,ShowUpdate,EpisodeSummary)
cursor.execute(sql)
db.commit()
except:
db.rollback()
db.close()
Any thoughts? Thanks in advance.
EDIT - WORKING CODE
import pytvmaze
import MySQLdb
AddShow = pytvmaze.get_show(show_name='dexter')
MazeID = AddShow.maze_id
ShowNameandID = "Show" + str(MazeID)
show = pytvmaze.get_show(MazeID, embed='episodes')
db = MySQLdb.connect("localhost","root","letmein","TVshows" )
cursor = db.cursor()
for episode in show.episodes:
ShowName = show.name
ShowStatus = show.status
ShowSummary = show.summary
Updated = show.updated
Season = episode.season_number
Episode = episode.episode_number
Title = episode.title
AirDate = episode.airdate
ShowUpdate = show.updated
EpisodeSummary = episode.summary
sql = "INSERT INTO " + ShowNameandID + """ VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"""
cursor.execute(sql, (ShowName, ShowStatus, ShowSummary, Updated, Season, Episode, Title, AirDate, ShowUpdate, EpisodeSummary))
db.commit()
print sql ##Great for debugging
db.close()
First of all, you've actually made things more difficult for yourself by catching all the exceptions via bare try/expect and then silently rolling back. Temporarily remove the try/except and see what the real error is, or log the exception in the except block. I bet the error would be related to a syntax error in the query since you would miss the quotes around the column value(s).
Anyway, arguably the biggest problem you have is how you pass the variables into the query. Currently, you are using string formatting, which is highly not recommended because of the SQL injection attack danger and problems with type conversions. Parameterize your query:
sql = """
INSERT INTO
{show}
VALUES
(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""".format(show=Show)
cursor.execute(sql, (ShowStatus, ShowSummary, Updated, Season, Episode, Title, AirDate, ShowUpdate, EpisodeSummary))
Note that it is not possible to parameterize the table name (Show in your case) - we are using string formatting for it - make sure you either trust your source, or escape it manually via MySQLdb.escape_string(), or validate it with a separate custom code.