We Keep Coding

Python - Multiline SQL with List - Not all parameters were used in the SQL statement

I'm usually a C# Dev, in an attempt to get some python experience.
So at this current time I got a List of values I want to import into MYSQL, as there will be quite a few lines I want go with a single big import than thousands of small ones.
Looking at the docs on pynative I can see the example being provided as:
mySql_insert_query = """INSERT INTO Laptop (Id, Name, Price, Purchase_date)
VALUES (%s, %s, %s, %s) """
records_to_insert = [(4, 'HP Pavilion Power', 1999, '2019-01-11'),
(5, 'MSI WS75 9TL-496', 5799, '2019-02-27'),
(6, 'Microsoft Surface', 2330, '2019-07-23')]
cursor = connection.cursor()
cursor.executemany(mySql_insert_query, records_to_insert)
connection.commit()
As my records to insert will be data pulled from an API, I had the "briliant" idea of building the list the most basic and probably the worst method possible.
By creating the list with a pre-formatted string.
val = []
sql = """INSERT INTO equipment (ItemStaticID, ItemID, ItemType, Element, Tier, Level, Hp, Atk, Hit, Def, Spd) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"""
for x in range(len(test)):
print(test[x].id, test[x].itemId, test[x].itemSubType, test[x].elementalType, test[x].grade, test[x].level, test[x].hP, test[x].aTK, test[x].hIT, test[x].dEF, test[x].sPD)
value = "({},'{}','{}','{}','{}','{}','{}','{}','{}','{}','{}')".format(test[x].id, test[x].itemId, test[x].itemSubType, test[x].elementalType, test[x].grade, test[x].level, test[x].hP, test[x].aTK, test[x].hIT, test[x].dEF, test[x].sPD)
print(value)
val.append(value)
insertcur = mydb.cursor()
insertcur.executemany(sql, val)
When running this code I get a "Not all parameters were used in the SQL statement", I verified that the SQL Query was accurate and the fields listed matched, 11 column names and I got 11 params.
So I'm a bit lost why it believes that not all params are being used in the SQL statement.
The below is the content of my list "val"
["(10340000,'d3252f40-5cbb-4a7c-a140-dcc2badac580','BELT','NORMAL','4','1','0','0','555','0','1683')", "(10340000,'e8e16a7b-4271-4cd4-8700-4dfb17a9525f','BELT','NORMAL','4','2','0','0','534','0','1848')", "(10341000,'66e70f12-f13b-4572-a222-cd61740d78b8','BELT','FIRE','4','0','0','0','503','0','1921')", "(10130000,'7cc2027b-110e-4ba0-9d95-ce563eca4be9','WEAPON','NORMAL','3','0','0','99','0','0','0')", "(10130000,'88440779-0794-431e-a726-3cb63af1dae1','WEAPON','NORMAL','3','0','0','99','0','0','0')", "(10130000,'f513afc3-9035-40b4-a0bb-6d4a74b88c7d','WEAPON','NORMAL','3','0','0','100','0','0','88')", "(10130000,'e22a771e-3fd7-4ce4-93c4-254913e89551','WEAPON','NORMAL','3','0','0','100','0','0','93')", "(10230000,'bf97a302-41e1-4b21-b7ff-3d7786f96ec3','ARMOR','NORMAL','3','0','1707','0','0','33','0')", "(10233000,'69f4811f-f45b-4227-99eb-2f45f0705c2a','ARMOR','LAND','3','0','11254','0','0','0','0')", "(10431000,'0813b427-de85-4fd6-82a1-fe8b8708ad45','NECKLACE','FIRE','3','0','0','0','677','30','0')", "(10531000,'48fc6562-44b5-49ba-8ad2-837d0e981a70','RING','FIRE','3','0','0','65','0','90','0')", "(10120000,'e5ebcdb9-ae59-4fc2-adb8-50e3b673c26d','WEAPON','NORMAL','2','0','0','39','0','0','0')", "(10120000,'22194497-91d1-405d-8a46-9c25b36c7ffd','WEAPON','NORMAL','2','1','0','42','0','0','37')", "(10124000,'487e9efc-a5f6-4944-924a-38ff8bcfb014','WEAPON','WIND','2','0','0','421','0','0','0')", "(10124000,'d015eced-58ec-48c4-bdbd-4f790c9ccbe4','WEAPON','WIND','2','0','0','453','0','0','572')", "(10124000,'1c624b33-889d-42f1-ae6b-2ddda234c481','WEAPON','WIND','2','1','0','462','0','0','0')", "(10220000,'225f94e7-2342-4a50-a1b7-07ec789457f1','ARMOR','NORMAL','2','1','737','0','0','13','0')", "(10220000,'64bd3ce5-572a-4b83-b83c-5f8f7f728f9e','ARMOR','NORMAL','2','1','727','0','0','13','0')", "(10224000,'206d6a67-3d1e-4b22-ad94-1ee19bdb4e00','ARMOR','WIND','2','3','9597','0','0','67','0')", "(10224000,'0ddc84d4-b65d-4b6d-b332-b3de72defce2','ARMOR','WIND','2','3','8972','0','0','88','0')", "(10321000,'3d0f39cb-0684-42a4-a9b0-ec2466679d18','BELT','FIRE','2','0','0','0','0','0','176')", "(10423000,'4ebb7192-15d4-4e9d-999c-be4a2e7ac85a','NECKLACE','LAND','2','3','0','0','1833','63','0')", "(10523000,'d1260941-ba0f-4a2a-b452-b3c3a6d6ec81','RING','LAND','2','0','0','0','384','172','0')", "(10523000,'fdef3e38-88ba-4970-923e-1d46ea35d1c5','RING','LAND','2','3','0','0','295','282','0')", "(10523000,'e3657fe9-481e-45ea-9086-065a79e3bc06','RING','LAND','2','0','0','0','376','173','0')", "(10523000,'68ab4ca0-683f-43a8-a2c6-358788e4e5e8','RING','LAND','2','2','0','0','408','195','0')", "(10523000,'be3921cc-fc3a-48bb-8542-c385d83d99cb','RING','LAND','2','0','0','0','0','172','0')", "(10523000,'422bb5ab-bef6-42a8-8562-a572d07c7970','RING','LAND','2','0','0','0','0','175','0')", "(10523000,'708df542-4622-4849-899c-846938f98ef2','RING','LAND','2','0','0','0','392','173','0')", "(10111000,'54896e7e-9799-4c81-a704-124522ab9b91','WEAPON','FIRE','1','3','0','23','0','0','0')", "(10210000,'09f995cd-6a8c-47ae-8c46-18f799322c9a','ARMOR','NORMAL','1','2','264','0','0','5','0')", "(10212000,'4effa256-6fe1-4051-866d-0f77d4294004','ARMOR','WATER','1','0','374','0','0','0','0')", "(10312000,'326908bb-ba10-4be9-abf2-4499d413bbb0','BELT','WATER','1','2','0','0','23','0','110')", "(10412000,'390c39f3-756b-488a-ae3f-eb03021453af','NECKLACE','WATER','1','3','0','0','156','8','0')", "(10512000,'7f8f0faf-da5c-435c-a27b-14ccbf94dd08','RING','WATER','1','3','159','0','0','20','0')", "(10514000,'08049c6d-5923-49d6-ac5f-fb569fc69ed3','RING','WIND','1','2','0','0','0','224','232')", "(10514000,'8dce0420-4d09-4ab6-8d73-86343155f8b9','RING','WIND','1','3','0','0','0','230','433')"]

importing single .csv into mysql with python

when running this code i am getting a Error while connecting to MySQL Not all parameters were used in the SQL statement
I have tried also to ingest these with another technique
import mysql.connector as msql
from mysql.connector import Error
import pandas as pd
empdata = pd.read_csv('path_to_file', index_col=False, delimiter = ',')
empdata.head()
try:
conn = msql.connect(host='localhost', user='test345',
password='test123')
if conn.is_connected():
cursor = conn.cursor()
cursor.execute("CREATE DATABASE timetheft")
print("Database is created")
except Error as e:
print("Error while connecting to MySQL", e)
try:
conn = msql.connect(host='localhost', database='timetheft', user='test345', password='test123')
if conn.is_connected():
cursor = conn.cursor()
cursor.execute("select database();")
record = cursor.fetchone()
print("You're connected to database: ", record)
cursor.execute('DROP TABLE IF EXISTS company;')
print('Creating table....')
create_contracts_table = """
CREATE TABLE company ( ID VARCHAR(40) PRIMARY KEY,
Company_Name VARCHAR(40),
Country VARCHAR(40),
City VARCHAR(40),
Email VARCHAR(40),
Industry VARCHAR(30),
Employees VARCHAR(30)
);
"""
cursor.execute(create_company_table)
print("Table is created....")
for i,row in empdata.iterrows():
sql = "INSERT INTO timetheft.company VALUES (%S, %S, %S, %S, %S,%S,%S,%S)"
cursor.execute(sql, tuple(row))
print("Record inserted")
# the connection is not auto committed by default, so we must commit to save our changes
conn.commit()
except Error as e:
print("Error while connecting to MySQL", e)
second technique I tried
LOAD DATA LOCAL INFILE 'path_to_file'
INTO TABLE copmany
FIELDS TERMINATED BY ';'
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES;
worked better but many errors. only 20% of rows ingested.
Finally here is an excerpt from the .csv (data is consistent throughout all 1K rows)
"ID";"Company_Name";"Country";"City";"Email";"Industry";"Employees"
217520699;"Enim Corp.";"Germany";"Bamberg";"posuere#diamvel.edu";"Internet";"51-100"
352428999;"Lacus Vestibulum Consulting";"Germany";"Villingen-Schwenningen";"egestas#lacusEtiambibendum.org";"Food Production";"100-500"
371718299;"Dictum Ultricies Ltd";"Germany";"Anklam";"convallis.erat#sempercursus.co.uk";"Primary/Secondary Education";"100-500"
676789799;"A Consulting";"Germany";"Andernach";"massa#etrisusQuisque.ca";"Government Relations";"100-500"
718526699;"Odio LLP";"Germany";"Eisenhüttenstadt";"Quisque.varius#euismod.org";"E-Learning";"11-50"

I fixed these issues to get the code to work:
make the number of placeholders in the insert statement equal to the number of columns
the placeholders should be lower-case '%s'
the cell delimiter appears to be a semi-colon, not a comma.
For simply reading a csv with ~1000 rows Pandas is overkill (and iterrows seems not to behave as you expect). I've used the csv module from the standard library instead.
import csv
...
sql = "INSERT INTO company VALUES (%s, %s, %s, %s, %s, %s, %s)"
with open("67359903.csv", "r", newline="") as f:
reader = csv.reader(f, delimiter=";")
# Skip the header row.
next(reader)
# For large files it may be more efficient to commit
# rows in batches.
cursor.executemany(sql, reader)
conn.commit()
If using the csv module is not convenient, the dataframe's itertuples method may be used to iterate over the data:
empdata = pd.read_csv('67359903.csv', index_col=False, delimiter=';')
for tuple_ in empdata.itertuples(index=False):
cursor.execute(sql, tuple_)
conn.commit()
Or the dataframe can be dumped to the database directly.
import sqlalchemy as sa
engine = sa.create_engine('mysql+mysqlconnector:///test')
empdata.to_sql('company', engine, index=False, if_exists='replace')

How to Import Big JSON file to MYSQL [duplicate]

I am having a hard time using the MySQLdb module to insert information into my database. I need to insert 6 variables into the table.
cursor.execute ("""
INSERT INTO Songs (SongName, SongArtist, SongAlbum, SongGenre, SongLength, SongLocation)
VALUES
(var1, var2, var3, var4, var5, var6)
""")
Can someone help me with the syntax here?

Beware of using string interpolation for SQL queries, since it won't escape the input parameters correctly and will leave your application open to SQL injection vulnerabilities. The difference might seem trivial, but in reality it's huge.
Incorrect (with security issues)
c.execute("SELECT * FROM foo WHERE bar = %s AND baz = %s" % (param1, param2))
Correct (with escaping)
c.execute("SELECT * FROM foo WHERE bar = %s AND baz = %s", (param1, param2))
It adds to the confusion that the modifiers used to bind parameters in a SQL statement varies between different DB API implementations and that the mysql client library uses printf style syntax instead of the more commonly accepted '?' marker (used by eg. python-sqlite).

You have a few options available. You'll want to get comfortable with python's string iterpolation. Which is a term you might have more success searching for in the future when you want to know stuff like this.
Better for queries:
some_dictionary_with_the_data = {
'name': 'awesome song',
'artist': 'some band',
etc...
}
cursor.execute ("""
INSERT INTO Songs (SongName, SongArtist, SongAlbum, SongGenre, SongLength, SongLocation)
VALUES
(%(name)s, %(artist)s, %(album)s, %(genre)s, %(length)s, %(location)s)
""", some_dictionary_with_the_data)
Considering you probably have all of your data in an object or dictionary already, the second format will suit you better. Also it sucks to have to count "%s" appearances in a string when you have to come back and update this method in a year :)

The linked docs give the following example:
cursor.execute ("""
UPDATE animal SET name = %s
WHERE name = %s
""", ("snake", "turtle"))
print "Number of rows updated: %d" % cursor.rowcount
So you just need to adapt this to your own code - example:
cursor.execute ("""
INSERT INTO Songs (SongName, SongArtist, SongAlbum, SongGenre, SongLength, SongLocation)
VALUES
(%s, %s, %s, %s, %s, %s)
""", (var1, var2, var3, var4, var5, var6))
(If SongLength is numeric, you may need to use %d instead of %s).

Actually, even if your variable (SongLength) is numeric, you will still have to format it with %s in order to bind the parameter correctly. If you try to use %d, you will get an error. Here's a small excerpt from this link http://mysql-python.sourceforge.net/MySQLdb.html:
To perform a query, you first need a cursor, and then you can execute queries on it:
c=db.cursor()
max_price=5
c.execute("""SELECT spam, eggs, sausage FROM breakfast
WHERE price < %s""", (max_price,))
In this example, max_price=5 Why, then, use %s in the string? Because MySQLdb will convert it to a SQL literal value, which is the string '5'. When it's finished, the query will actually say, "...WHERE price < 5".

As an alternative to the chosen answer, and with the same safe semantics of Marcel's, here is a compact way of using a Python dictionary to specify the values. It has the benefit of being easy to modify as you add or remove columns to insert:
meta_cols = ('SongName','SongArtist','SongAlbum','SongGenre')
insert = 'insert into Songs ({0}) values ({1})'.format(
','.join(meta_cols), ','.join( ['%s']*len(meta_cols)))
args = [ meta[i] for i in meta_cols ]
cursor = db.cursor()
cursor.execute(insert,args)
db.commit()
Where meta is the dictionary holding the values to insert. Update can be done in the same way:
meta_cols = ('SongName','SongArtist','SongAlbum','SongGenre')
update='update Songs set {0} where id=%s'.
.format(','.join([ '{0}=%s'.format(c) for c in meta_cols ]))
args = [ meta[i] for i in meta_cols ]
args.append(songid)
cursor=db.cursor()
cursor.execute(update,args)
db.commit()

The first solution works well. I want to add one small detail here. Make sure the variable you are trying to replace/update it will has to be a type str. My mysql type is decimal but I had to make the parameter variable as str to be able to execute the query.
temp = "100"
myCursor.execute("UPDATE testDB.UPS SET netAmount = %s WHERE auditSysNum = '42452'",(temp,))
myCursor.execute(var)

Here is another way to do it. It's documented on the MySQL official website.
https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html
In the spirit, it's using the same mechanic of #Trey Stout's answer. However, I find this one prettier and more readable.
insert_stmt = (
"INSERT INTO employees (emp_no, first_name, last_name, hire_date) "
"VALUES (%s, %s, %s, %s)"
)
data = (2, 'Jane', 'Doe', datetime.date(2012, 3, 23))
cursor.execute(insert_stmt, data)
And to better illustrate any need for variables:
NB: note the escape being done.
employee_id = 2
first_name = "Jane"
last_name = "Doe"
insert_stmt = (
"INSERT INTO employees (emp_no, first_name, last_name, hire_date) "
"VALUES (%s, %s, %s, %s)"
)
data = (employee_id, conn.escape_string(first_name), conn.escape_string(last_name), datetime.date(2012, 3, 23))
cursor.execute(insert_stmt, data)

Read from csv file and put to Mysql with different field type using python

i have table field of "datetime", "integer", "double", "string (varchar)" type in Mysql, how can i convert from string (from csv file) to each type of Mysql field using python ? example of my csv file :
EURUSD;M5;2011.01.05 02:10:00;1.33193;1.33193;1.33112;1.33135;0;-1;0;0;1.33215
My python code is like this :
import MySQLdb
# Open database connection
db = MySQLdb.connect("localhost","me","xxxxx","test" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
inputFile = "source.csv"
with open(inputFile, 'r') as fr:
data = fr.readlines()
for line in data:
words = line.strip().split(";")
getpair = words[0]
gettf = words[1]
gettime = (datetime) words[2]
getopen = (double) words[3]
gethigh = (double) words[4]
getlow = (double) words[5]
getclose = (double) words[6]
getcUpdown = (int) words[7]
getfractal = (int) words[8]
getzzId = (int) words[9]
getzzUpdown = (int) words[10]
getzzEndPrc = (double) words[11]
print(getpair,"=",gettf,"=",gettime,"=",getopen,"=",gethigh,"=",getlow,"=",getclose,"=",getcUpdown,"=",getfractal,"=",getzzId,"=",getzzUpdown,"=",getzzEndPrc)
# =====================================================================
# Prepare SQL query to INSERT a record into the database.
sql = '''INSERT INTO `mytable` (pair,timeframe,time,open,close,high,low,updown,fractal,zzid,zzupdown,zzlength) \
VALUES (getpair,gettf,gettime,getopen,getclose,gethigh,getlow,getcUpdown,getfractal,getzzId,getzzUpdown,getzzEndPrc)'''
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
fr.close()
Thanks
p

Type conversion in Python is done by:
int(words[7])
not
(int)words[7]
There is no double type in Python. Use float instead.
You can not convert directly to datetime. Use datetime.strptime() to parse the input string:
from datetime import datetime
gettime = datetime.strptime(words[2], '%Y.%m.%d %H:%M:%S')
You might not need to do this (conversion should be handled when using parameterised queries - see below), but you can convert that to a string format that MySQLDB supports. You can use strftime() for that:
gettime = gettime.strftime('%Y-%m-%d %H:%M:%S')
The sql query string needs to substitute the the actual values. Your
query is inserting the names of the variables, not the
values. You can (and should) use a parameterised query like this:
sql = 'INSERT INTO `mytable` (pair, timeframe, time, open, close, high, low, updown, fractal, zzid, zzupdown, zzlength) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)'
cursor.execute(sql, (getpair, gettf, gettime, getopen, getclose, gethigh, getlow, getcUpdown, getfractal, getzzId, getzzUpdown, getzzEndPrc))
cursor.execute(sql, ...) needs to be moved into the body of the for loop.

MySQL not accepting executemany() INSERT, running Python from Excel (datanitro)

I HAVE ADDED MY OWN ANSWER THAT WORKS BUT OPEN TO IMPROVEMENTS
After seeing a project at datanitro. I took on getting a connection to MySQL (they use SQLite) and I was able to import a small test table into Excel from MySQL.
Inserting new updated data from the Excel sheet was this next task and so far I can get one row to work like so...
import MySQLdb
db = MySQLdb.connect("xxx","xxx","xxx","xxx")
c = db.cursor()
c.execute("""INSERT INTO users (id, username, password, userid, fname, lname)
VALUES (%s, %s, %s, %s, %s, %s);""",
(Cell(5,1).value,Cell(5,2).value,Cell(5,3).value,Cell(5,4).value,Cell(5,5).value,Cell(5,6).value,))
db.commit()
db.close()
...but attempts at multiple rows will fail. I suspect either issues while traversing rows in Excel. Here is what I have so far...
import MySQLdb
db = MySQLdb.connect(host="xxx.com", user="xxx", passwd="xxx", db="xxx")
c = db.cursor()
c.execute("select * from users")
usersss = c.fetchall()
updates = []
row = 2 # starting row
while True:
data = tuple(CellRange((row,1),(row,6)).value)
if data[0]:
if data not in usersss: # new record
updates.append(data)
row += 1
else: # end of table
break
c.executemany("""INSERT INTO users (id, username, password, userid, fname, lname) VALUES (%s, %s, %s, %s, %s, %s)""", updates)
db.commit()
db.close()
...as of now, I don't get any errors, but my new line is not added (id 3). This is what my table looks like in Excel...
The database holds the same structure, minus id 3. There has to be a simpler way to traverse the rows and pull the unique content for INSERT, but after 6 hours trying different things (and 2 new Python books) I am going to ask for help.
If I run either...
print '[%s]' % ', '.join(map(str, updates))
or
print updates
my result is
[]
So this is likely not passing any data to MySQL in the first place.
LATEST UPDATE AND WORKING SCRIPT
Not exactly what I want, but this has worked for me...
c = db.cursor()
row = 2
while Cell(row,1).value != None:
c.execute("""INSERT IGNORE INTO users (id, username, password, userid, fname, lname)
VALUES (%s, %s, %s, %s, %s, %s);""",
(CellRange((row,1),(row,6)).value))
row = row + 1

Here is your problem:
while True:
if data[0]:
...
else:
break
Your first id is 0, so in the first iteration of the loop data[0] will be falsely and your loop will exit, without ever adding any data. What you probably ment is:
while True:
if data[0] is not None:
...
else:
break

I ended up finding a solution that gets me an Insert on new and allows for UPDATE of those that are changed. Not exactly a Python selection based on a single query, but will do.
import MySQLdb
db = MySQLdb.connect("xxx","xxx","xxx","xxx")
c = db.cursor()
row = 2
while Cell(row,1).value is not None:
c.execute("INSERT INTO users (id, username, password, \
userid, fname, lname) \
VALUES (%s, %s, %s, %s, %s, %s) \
ON DUPLICATE KEY UPDATE \
id=VALUES(id), username=VALUES(username), password=VALUES(password), \
userid=VALUES(userid), fname=VALUES(fname), lname=VALUES(lname);",
(CellRange((row,1),(row,6)).value))
row = row + 1
db.commit()
db.close()