I'm usually a C# Dev, in an attempt to get some python experience.
So at this current time I got a List of values I want to import into MYSQL, as there will be quite a few lines I want go with a single big import than thousands of small ones.
Looking at the docs on pynative I can see the example being provided as:
mySql_insert_query = """INSERT INTO Laptop (Id, Name, Price, Purchase_date)
VALUES (%s, %s, %s, %s) """
records_to_insert = [(4, 'HP Pavilion Power', 1999, '2019-01-11'),
(5, 'MSI WS75 9TL-496', 5799, '2019-02-27'),
(6, 'Microsoft Surface', 2330, '2019-07-23')]
cursor = connection.cursor()
cursor.executemany(mySql_insert_query, records_to_insert)
connection.commit()
As my records to insert will be data pulled from an API, I had the "briliant" idea of building the list the most basic and probably the worst method possible.
By creating the list with a pre-formatted string.
val = []
sql = """INSERT INTO equipment (ItemStaticID, ItemID, ItemType, Element, Tier, Level, Hp, Atk, Hit, Def, Spd) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"""
for x in range(len(test)):
print(test[x].id, test[x].itemId, test[x].itemSubType, test[x].elementalType, test[x].grade, test[x].level, test[x].hP, test[x].aTK, test[x].hIT, test[x].dEF, test[x].sPD)
value = "({},'{}','{}','{}','{}','{}','{}','{}','{}','{}','{}')".format(test[x].id, test[x].itemId, test[x].itemSubType, test[x].elementalType, test[x].grade, test[x].level, test[x].hP, test[x].aTK, test[x].hIT, test[x].dEF, test[x].sPD)
print(value)
val.append(value)
insertcur = mydb.cursor()
insertcur.executemany(sql, val)
When running this code I get a "Not all parameters were used in the SQL statement", I verified that the SQL Query was accurate and the fields listed matched, 11 column names and I got 11 params.
So I'm a bit lost why it believes that not all params are being used in the SQL statement.
The below is the content of my list "val"
["(10340000,'d3252f40-5cbb-4a7c-a140-dcc2badac580','BELT','NORMAL','4','1','0','0','555','0','1683')", "(10340000,'e8e16a7b-4271-4cd4-8700-4dfb17a9525f','BELT','NORMAL','4','2','0','0','534','0','1848')", "(10341000,'66e70f12-f13b-4572-a222-cd61740d78b8','BELT','FIRE','4','0','0','0','503','0','1921')", "(10130000,'7cc2027b-110e-4ba0-9d95-ce563eca4be9','WEAPON','NORMAL','3','0','0','99','0','0','0')", "(10130000,'88440779-0794-431e-a726-3cb63af1dae1','WEAPON','NORMAL','3','0','0','99','0','0','0')", "(10130000,'f513afc3-9035-40b4-a0bb-6d4a74b88c7d','WEAPON','NORMAL','3','0','0','100','0','0','88')", "(10130000,'e22a771e-3fd7-4ce4-93c4-254913e89551','WEAPON','NORMAL','3','0','0','100','0','0','93')", "(10230000,'bf97a302-41e1-4b21-b7ff-3d7786f96ec3','ARMOR','NORMAL','3','0','1707','0','0','33','0')", "(10233000,'69f4811f-f45b-4227-99eb-2f45f0705c2a','ARMOR','LAND','3','0','11254','0','0','0','0')", "(10431000,'0813b427-de85-4fd6-82a1-fe8b8708ad45','NECKLACE','FIRE','3','0','0','0','677','30','0')", "(10531000,'48fc6562-44b5-49ba-8ad2-837d0e981a70','RING','FIRE','3','0','0','65','0','90','0')", "(10120000,'e5ebcdb9-ae59-4fc2-adb8-50e3b673c26d','WEAPON','NORMAL','2','0','0','39','0','0','0')", "(10120000,'22194497-91d1-405d-8a46-9c25b36c7ffd','WEAPON','NORMAL','2','1','0','42','0','0','37')", "(10124000,'487e9efc-a5f6-4944-924a-38ff8bcfb014','WEAPON','WIND','2','0','0','421','0','0','0')", "(10124000,'d015eced-58ec-48c4-bdbd-4f790c9ccbe4','WEAPON','WIND','2','0','0','453','0','0','572')", "(10124000,'1c624b33-889d-42f1-ae6b-2ddda234c481','WEAPON','WIND','2','1','0','462','0','0','0')", "(10220000,'225f94e7-2342-4a50-a1b7-07ec789457f1','ARMOR','NORMAL','2','1','737','0','0','13','0')", "(10220000,'64bd3ce5-572a-4b83-b83c-5f8f7f728f9e','ARMOR','NORMAL','2','1','727','0','0','13','0')", "(10224000,'206d6a67-3d1e-4b22-ad94-1ee19bdb4e00','ARMOR','WIND','2','3','9597','0','0','67','0')", "(10224000,'0ddc84d4-b65d-4b6d-b332-b3de72defce2','ARMOR','WIND','2','3','8972','0','0','88','0')", "(10321000,'3d0f39cb-0684-42a4-a9b0-ec2466679d18','BELT','FIRE','2','0','0','0','0','0','176')", "(10423000,'4ebb7192-15d4-4e9d-999c-be4a2e7ac85a','NECKLACE','LAND','2','3','0','0','1833','63','0')", "(10523000,'d1260941-ba0f-4a2a-b452-b3c3a6d6ec81','RING','LAND','2','0','0','0','384','172','0')", "(10523000,'fdef3e38-88ba-4970-923e-1d46ea35d1c5','RING','LAND','2','3','0','0','295','282','0')", "(10523000,'e3657fe9-481e-45ea-9086-065a79e3bc06','RING','LAND','2','0','0','0','376','173','0')", "(10523000,'68ab4ca0-683f-43a8-a2c6-358788e4e5e8','RING','LAND','2','2','0','0','408','195','0')", "(10523000,'be3921cc-fc3a-48bb-8542-c385d83d99cb','RING','LAND','2','0','0','0','0','172','0')", "(10523000,'422bb5ab-bef6-42a8-8562-a572d07c7970','RING','LAND','2','0','0','0','0','175','0')", "(10523000,'708df542-4622-4849-899c-846938f98ef2','RING','LAND','2','0','0','0','392','173','0')", "(10111000,'54896e7e-9799-4c81-a704-124522ab9b91','WEAPON','FIRE','1','3','0','23','0','0','0')", "(10210000,'09f995cd-6a8c-47ae-8c46-18f799322c9a','ARMOR','NORMAL','1','2','264','0','0','5','0')", "(10212000,'4effa256-6fe1-4051-866d-0f77d4294004','ARMOR','WATER','1','0','374','0','0','0','0')", "(10312000,'326908bb-ba10-4be9-abf2-4499d413bbb0','BELT','WATER','1','2','0','0','23','0','110')", "(10412000,'390c39f3-756b-488a-ae3f-eb03021453af','NECKLACE','WATER','1','3','0','0','156','8','0')", "(10512000,'7f8f0faf-da5c-435c-a27b-14ccbf94dd08','RING','WATER','1','3','159','0','0','20','0')", "(10514000,'08049c6d-5923-49d6-ac5f-fb569fc69ed3','RING','WIND','1','2','0','0','0','224','232')", "(10514000,'8dce0420-4d09-4ab6-8d73-86343155f8b9','RING','WIND','1','3','0','0','0','230','433')"]
Been playing with this for 14 hours (I am a beginner)
Data is pulled from one database table to search on yahoo for all the data on that ticker and then its "meant" to upload it.
I orginally had it as panda df but got "ambiguous error" so I have now put it as [] again. New error. I rack my brains :( However, it does work if I leave it blank.
from __future__ import print_function
import yfinance as yf
import pandas as pd
import datetime
import warnings
import MySQLdb as mdb
import requests
import numpy as np
import MySQLdb as mdb
import requests
# Obtain a database connection to the MySQL instance
con = mdb.connect("localhost","sec_user","","securities_master")
def obtain_list_of_db_tickers():
"""
Obtains a list of the ticker symbols in the database.
"""
with con:
cur = con.cursor()
cur.execute("SELECT id, ticker FROM symbol")
data = cur.fetchall()
print(data)
return [(d[0], d[1]) for d in data]
def get_daily_historic_data_yahoo(ticker):
blow = yf.download(ticker)
data = []
data.append(yf.download(ticker).reset_index())
return data
def insert_daily_data_into_db(data_vendor_id, symbol_id, daily_data):
'''
Takes a list of tuples of daily data and adds it to the MySQL database.
Appends the vendor ID and symbol ID to the data.
daily_data: List of tuples of the OHLC data (with adj_close and volume)
'''
# Create the time now
now = datetime.datetime.utcnow()
df = pd.DataFrame(data=daily_data[0])
df.insert(0, 'data_vendor_id', data_vendor_id)
df.insert(1, 'symbol_id', symbol_id)
df.insert(3, 'created_date', now)
df.insert(4, 'last_updated_date', now)
daily_data = []
daily_data.append(df)
#df = daily_data
# Amend the data to include the vendor ID and symbol ID
# Connect to the MySQL instance
db_host = 'localhost'
db_user = ''
db_pass = ''
db_name = 'securities_master'
con = mdb.connect("localhost", "sec_user", "", "securities_master"
# host=db_host, user=db_user, passwd=db_pass, db=db_name
)
try:
mdb.connect
# If connection is not successful
except:
print("Can't connect to database")
return 0
# If Connection Is Successful
print("Connected")
final_str = """INSERT INTO daily_price (data_vendor_id, symbol_id, price_date, created_date,
last_updated_date, open_price, high_price, low_price, close_price, volume, adj_close_price) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"""
with con:
cur = con.cursor()
cur.executemany(final_str, daily_data)
con.commit()
if __name__ == "__main__":
# This ignores the warnings regarding Data Truncation
# from the Yahoo precision to Decimal(19,4) datatypes
warnings.filterwarnings('ignore')
# Loop over the tickers and insert the daily historical
# data into the database
tickers = obtain_list_of_db_tickers()
lentickers = len(tickers)
for i, t in enumerate(tickers):
print(
"Adding data for %s: %s out of %s" %
(t[1], i+1, lentickers)
)
yf_data = get_daily_historic_data_yahoo(t[1])
insert_daily_data_into_db('1', t[0], yf_data)
print("Successfully added Yahoo Finance pricing data to DB.")
Errors
Traceback (most recent call last):
File "/home/quant/price_retrieval.py", line 106, in <module>
insert_daily_data_into_db('1', t[0], yf_data)
File "/home/quant/price_retrieval.py", line 88, in insert_daily_data_into_db
cur.executemany(final_str, daily_data)
File "/home/quant/.local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 230, in executemany
return self._do_execute_many(
File "/home/quant/.local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 255, in _do_execute_many
v = values % escape(next(args), conn)
TypeError: not enough arguments for format string
I'm no data scientist so there's probably a more elegant way to fix it directly with pandas. But the way I usually work with MySQL (and really any SQL drivers) is to give it lists of python tuples.
If you parse each row of the pandas data frame with for row in df.itertuples(): and craft each tuple carefully - making sure the types match the SQL table, all should work ;)
Example:
def insert_daily_data_into_db(data_vendor_id, symbol_id, daily_data):
'''
Takes a list of tuples of daily data and adds it to the MySQL database.
Appends the vendor ID and symbol ID to the data.
daily_data: List of tuples of the OHLC data (with adj_close and volume)
'''
# Create the time now
now = datetime.datetime.utcnow()
df = pd.DataFrame(data=daily_data[0])
daily_data = []
created_date = now
last_updated_date = now
for row in df.itertuples():
_index = row[0] # discard
date = row[1]
open = row[2]
high = row[3]
low = row[4]
close = row[5]
adj_close_price = row[6]
volume = row[7]
daily_data.append((int(data_vendor_id), symbol_id, date, created_date, last_updated_date, open, high, low, close, volume, adj_close_price))
# Connect to the MySQL instance
con = mdb.connect(host="localhost", user="user", password="yourpassword",
db="yourdbname", port=3306)
final_str = """
INSERT INTO daily_price (data_vendor_id, symbol_id, price_date, created_date,
last_updated_date, open_price, high_price, low_price, close_price, volume, adj_close_price)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
"""
with con:
cur = con.cursor()
cur.executemany(final_str, daily_data)
con.commit()
I've tried not to tamper with your existing code too much. Just enough to make it work.
I think what was happening there for you was that you're technically passing it a list of pandas dataframes with only a single pandas dataframe in the list. Instead what you want is a list of tuples with 11 fields to unpack per tuple.
Maybe you mean to pass the dataframe directly i.e. not contained inside of a list but I still don't think that would be right because 1) there's an "Index" column in the dataframe which would give erroneous results 2) you'd need to call some methods on the dataframe to retrieve only the values (not the headers to the columns) and transform it to the correct list of tuples. It's probably very doable but I will leave that to you to find out.
I am also assuming your table schema is something like this:
CREATE TABLE IF NOT EXISTS daily_price (
data_vendor_id INT,
symbol_id INT,
price_date DATETIME,
created_date DATETIME,
last_updated_date TIMESTAMP,
open_price VARCHAR(256),
high_price VARCHAR(256),
low_price VARCHAR(256),
close_price VARCHAR(256),
volume INT,
adj_close_price VARCHAR(256)
);
I am trying to store some TV information in a MySQLdb. I have tried about everything and I cannot get the variables to post. There is information in the variables as I am able to print the information.
My Code:
import pytvmaze
import MySQLdb
AddShow = pytvmaze.get_show(show_name='dexter')
MazeID = AddShow.maze_id
ShowName = "Show" + str(MazeID)
show = pytvmaze.get_show(MazeID, embed='episodes')
db = MySQLdb.connect("localhost","root","XXXXXXX","TVshows" )
cursor = db.cursor()
for episode in show.episodes:
Show = show.name
ShowStatus = show.status
ShowSummary = show.summary
Updated = show.updated
Season = episode.season_number
Episode = episode.episode_number
Title = episode.title
AirDate = episode.airdate
ShowUpdate = show.updated
EpisodeSummary = episode.summary
try:
sql = "INSERT INTO " + ShowName + " VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)""" (Show,ShowStatus,ShowSummary,Updated,Season,Episode,Title,AirDate,ShowUpdate,EpisodeSummary)
cursor.execute(sql)
db.commit()
except:
db.rollback()
db.close()
Any thoughts? Thanks in advance.
EDIT - WORKING CODE
import pytvmaze
import MySQLdb
AddShow = pytvmaze.get_show(show_name='dexter')
MazeID = AddShow.maze_id
ShowNameandID = "Show" + str(MazeID)
show = pytvmaze.get_show(MazeID, embed='episodes')
db = MySQLdb.connect("localhost","root","letmein","TVshows" )
cursor = db.cursor()
for episode in show.episodes:
ShowName = show.name
ShowStatus = show.status
ShowSummary = show.summary
Updated = show.updated
Season = episode.season_number
Episode = episode.episode_number
Title = episode.title
AirDate = episode.airdate
ShowUpdate = show.updated
EpisodeSummary = episode.summary
sql = "INSERT INTO " + ShowNameandID + """ VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"""
cursor.execute(sql, (ShowName, ShowStatus, ShowSummary, Updated, Season, Episode, Title, AirDate, ShowUpdate, EpisodeSummary))
db.commit()
print sql ##Great for debugging
db.close()
First of all, you've actually made things more difficult for yourself by catching all the exceptions via bare try/expect and then silently rolling back. Temporarily remove the try/except and see what the real error is, or log the exception in the except block. I bet the error would be related to a syntax error in the query since you would miss the quotes around the column value(s).
Anyway, arguably the biggest problem you have is how you pass the variables into the query. Currently, you are using string formatting, which is highly not recommended because of the SQL injection attack danger and problems with type conversions. Parameterize your query:
sql = """
INSERT INTO
{show}
VALUES
(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""".format(show=Show)
cursor.execute(sql, (ShowStatus, ShowSummary, Updated, Season, Episode, Title, AirDate, ShowUpdate, EpisodeSummary))
Note that it is not possible to parameterize the table name (Show in your case) - we are using string formatting for it - make sure you either trust your source, or escape it manually via MySQLdb.escape_string(), or validate it with a separate custom code.
I HAVE ADDED MY OWN ANSWER THAT WORKS BUT OPEN TO IMPROVEMENTS
After seeing a project at datanitro. I took on getting a connection to MySQL (they use SQLite) and I was able to import a small test table into Excel from MySQL.
Inserting new updated data from the Excel sheet was this next task and so far I can get one row to work like so...
import MySQLdb
db = MySQLdb.connect("xxx","xxx","xxx","xxx")
c = db.cursor()
c.execute("""INSERT INTO users (id, username, password, userid, fname, lname)
VALUES (%s, %s, %s, %s, %s, %s);""",
(Cell(5,1).value,Cell(5,2).value,Cell(5,3).value,Cell(5,4).value,Cell(5,5).value,Cell(5,6).value,))
db.commit()
db.close()
...but attempts at multiple rows will fail. I suspect either issues while traversing rows in Excel. Here is what I have so far...
import MySQLdb
db = MySQLdb.connect(host="xxx.com", user="xxx", passwd="xxx", db="xxx")
c = db.cursor()
c.execute("select * from users")
usersss = c.fetchall()
updates = []
row = 2 # starting row
while True:
data = tuple(CellRange((row,1),(row,6)).value)
if data[0]:
if data not in usersss: # new record
updates.append(data)
row += 1
else: # end of table
break
c.executemany("""INSERT INTO users (id, username, password, userid, fname, lname) VALUES (%s, %s, %s, %s, %s, %s)""", updates)
db.commit()
db.close()
...as of now, I don't get any errors, but my new line is not added (id 3). This is what my table looks like in Excel...
The database holds the same structure, minus id 3. There has to be a simpler way to traverse the rows and pull the unique content for INSERT, but after 6 hours trying different things (and 2 new Python books) I am going to ask for help.
If I run either...
print '[%s]' % ', '.join(map(str, updates))
or
print updates
my result is
[]
So this is likely not passing any data to MySQL in the first place.
LATEST UPDATE AND WORKING SCRIPT
Not exactly what I want, but this has worked for me...
c = db.cursor()
row = 2
while Cell(row,1).value != None:
c.execute("""INSERT IGNORE INTO users (id, username, password, userid, fname, lname)
VALUES (%s, %s, %s, %s, %s, %s);""",
(CellRange((row,1),(row,6)).value))
row = row + 1
Here is your problem:
while True:
if data[0]:
...
else:
break
Your first id is 0, so in the first iteration of the loop data[0] will be falsely and your loop will exit, without ever adding any data. What you probably ment is:
while True:
if data[0] is not None:
...
else:
break
I ended up finding a solution that gets me an Insert on new and allows for UPDATE of those that are changed. Not exactly a Python selection based on a single query, but will do.
import MySQLdb
db = MySQLdb.connect("xxx","xxx","xxx","xxx")
c = db.cursor()
row = 2
while Cell(row,1).value is not None:
c.execute("INSERT INTO users (id, username, password, \
userid, fname, lname) \
VALUES (%s, %s, %s, %s, %s, %s) \
ON DUPLICATE KEY UPDATE \
id=VALUES(id), username=VALUES(username), password=VALUES(password), \
userid=VALUES(userid), fname=VALUES(fname), lname=VALUES(lname);",
(CellRange((row,1),(row,6)).value))
row = row + 1
db.commit()
db.close()