I have tried many times and searched all over the internet and this is still not working for me.
I am trying to read from a csv file and insert data into a database with python.
This is my code, and I don't understand why it's not working
import mysql.connector
import csv
import pandas as pd
with open(r'files\files1.csv') as csv_file:
csvfile = csv.reader(csv_file, delimiter=';')
allvalues=[]
for row in csvfile:
value = (row[0],row[1],row[2])
allvalues.append(value)
print(allvalues)
db = mysql.connector.connect(
host = 'ip',
user = 'user',
passwd = 'pass',
database = 'db',
auth_plugin='mysql_native_password'
)
cursor = db.cursor()
query = "INSERT INTO table1 (col1, col2, col3) VALUES (%s , %s , %s)"
cursor.execute(query, allvalues)
db.commit()
this gives the following error:
result = self._cmysql.convert_to_mysql(*params)
_mysql_connector.MySQLInterfaceError: Python type tuple cannot be converted
I also want to mentions that I have tried many other things to insert into the table not only the method above, and everytime I get a different error.
Can someone please tell me how do I do it? I would really appreciate it
Thank you very much
use cursor.executemany(query, allvalues)
If you have multiple elements which are saved in a list or tuple then use,
cursor.executemany(query, list) or cursor.executemany(query, tuple)
Or you can use for loop
for value in allvalues:
cursor.execute(query, value)
Related
I am using python to establish db connection and reading csv file. For each line in csv i want to run a PostgreSQL query and get value corresponding to each line read.
DB connection and file reading is working fine. Also if i run query for hardcoded value then it works fine. But if i try to run query for each row in csv file using python variable then i am not getting correct value.
cursor.execute("select team from users.teamdetails where p_id = '123abc'")
Above query works fine.
but when i try it for multiple values fetched from csv file then i am not getting correct value.
cursor.execute("select team from users.teamdetails where p_id = queryPID")
Complete code for Reference:
import psycopg2
import csv
conn = psycopg2.connect(dbname='', user='', password='', host='', port='')
cursor = conn.cursor()
with open('playerid.csv','r') as csv_file:
csv_reader = csv.reader(csv_file)
for line in csv_reader:
queryPID = line[0]
cursor.execute("select team from users.teamdetails where p_id = queryPID")
team = cursor.fetchone()
print (team[0])
conn.close()
DO NOT concatenate the csv data. Use a parameterised query.
Use %s inside your string, then pass the additional variable:
cursor.execute('select team from users.teamdetails where p_id = %s', (queryPID,))
Concatenation of text leaves your application vulnerable to SQL injection.
https://www.psycopg.org/docs/usage.html
I'm trying to update mysql table based on my csv data where sha1 in my csv should update or insert the suggestedname on duplicate. What part am I doing wrong here? Gives me error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'where sha1=#col1' at line 1
Here is my table structure:
date_sourced, sha1, suggested, vsdt, trendx, falcon, notes, mtf
CSV structure:
SHA1,suggestedName
Code:
import mysql.connector
mydb = mysql.connector.connect(user='root', password='',
host='localhost',database='jeremy_db')
cursor = mydb.cursor()
query = "LOAD DATA INFILE %s IGNORE INTO TABLE jeremy_table_test FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (#col1,#col2) set suggested=#col2 where sha1=#col1"
cursor.execute(query, (fullPath))
mydb.commit()
LOAD DATA INFILE can not add condition in it. You can try to read file through pandas then insert value into table, but you need to set up an unique index on sha1 in advance. otherwise, my script will not work(reason).
import pandas as pd
import mysql.connector as mysql
path = "1.xls"
df = pd.read_excel(path)
_sha1 = df["SHA1"].tolist()
_suggestedName = df["suggestedName"].tolist()
conn = mysql.connect(user="xx",passwd="xx",db="xx")
cur = conn.cursor()
sql = """INSERT INTO jeremy_table_test (sha1,suggested) VALUES (%s,%s) ON DUPLICATE KEY UPDATE suggested=VALUES(suggested)"""
try:
cur.executemany(sql,list(zip(_sha1,_suggestedName)))
conn.commit()
except Exception as e:
conn.rollback()
raise e
I'm trying to insert into a MySQL table from data in this Excel sheet: https://www.dropbox.com/s/w7m282386t08xk3/GA.xlsx?dl=0
The script should start from the second sheet "Daily Metrics" at row 16. The MySQL table already has the fields called date, campaign, users, and sessions.
Using Python 2.7, I've already created the MySQL connection and opened the sheet, but I'm not sure how to loop over those rows and insert into the database.
import MySQLdb as db
from openpyxl import load_workbook
wb = load_workbook('GA.xlsx')
sheetranges = wb['Daily Metrics']
print(sheetranges['A16'].value)
conn = db.connect('serverhost','username','password','database')
cursor = conn.cursor()
cursor.execute('insert into test_table ...')
conn.close()
Thank you for you help!
Try this and see if it does what you are looking for. You will need to update to the correct workbook name and location. Also, udate the range that you want to iterate over in for rw in wb["Daily Metrics"].iter_rows("A16:B20"):
from openpyxl import load_workbook
wb = load_workbook("c:/testing.xlsx")
for rw in wb["Daily Metrics"].iter_rows("A16:B20"):
for cl in rw:
print cl.value
Only basic knowledge of MySQL and Openpyxl is needed, you can solve it by reading tutorials on your own.
Before executing the script, you need to create database and table. Assuming you've done it.
import openpyxl
import MySQLdb
wb = openpyxl.load_workbook('/path/to/GA.xlsx')
ws = wb['Daily Metrics']
# map is a convenient way to construct a list. you can get a 2x2 tuple by slicing
# openpyxl.worksheet.worksheet.Worksheet instance and last row of worksheet
# from openpyxl.worksheet.worksheet.Worksheet.max_row
data = map(lambda x: {'date': x[0].value,
'campaign': x[1].value,
'users': x[2].value,
'sessions': x[3].value},
ws[16: ws.max_row])
# filter is another builtin function. Filter blank cells out if needed
data = filter(lambda x: None not in x.values(), data)
db = MySQLdb.connect('host', 'user', 'password', 'database')
cursor = db.cursor()
for row in data:
# execute raw MySQL syntax by using execute function
cursor.execute('insert into table (date, campaign, users, sessions)'
'values ("{date}", "{campaign}", {users}, {sessions});'
.format(**row)) # construct MySQL syntax through format function
db.commit()
I am currently trying to use pyodbc to insert data from a .csv into an Azure SQL Server database. I found a majority of this syntax on Stack Overflow, however for some reason I keep getting one of two different errors.
1) Whenever I use the following code, I get an error that states 'The SQL contains 0 parameter markers, but 7 parameters were supplied'.
import pyodbc
import csv
cnxn = pyodbc.connect('driver', user='username', password='password', database='database')
cnxn.autocommit = True
cursor = cnxn.cursor()
csvfile = open('CSV File')
csv_data = csv.reader(csvfile)
SQL="insert into table([Col1],[Col2],[Col3],[Col4],[Col5],[Col6],[Col7]) values ('?','?','?','?','?','?','?')"
for row in csv_data:
cursor.execute(SQL, row)
time.sleep(1)
cnxn.commit()
cnxn.close()
2) In order to get rid of that error, I am defining the parameter markers by adding '=?' to each of the columns in the insert statement (see code below), however this then gives the following error: ProgrammingError: ('42000'"[42000] [Microsoft] [ODBC SQL Server Driver][SQL Server] Incorrect syntax near '=').
import pyodbc
import csv
cnxn = pyodbc.connect('driver', user='username', password='password', database='database')
cnxn.autocommit = True
cursor = cnxn.cursor()
csvfile = open('CSV File')
csv_data = csv.reader(csvfile)
SQL="insert into table([Col1]=?,[Col2]=?,[Col3]=?,[Col4]=?,[Col5]=?,[Col6]=?,[Col7]=?) values ('?','?','?','?','?','?','?')"
for row in csv_data:
cursor.execute(SQL, row)
time.sleep(1)
cnxn.commit()
cnxn.close()
This is the main error I am haveing trouble with, I have searched all over Stack Overflow and can't seem to find a solution. I know this error is probably very trivial, however I am new to Python and would greatly appreciate any advice or help.
Since SQL server can import your entire CSV file with a single statement this is a reinvention of the wheel.
BULK INSERT my_table FROM 'CSV_FILE'
WITH ( FIELDTERMINATOR=',', ROWTERMINATOR='\n');
If you want to persist with using python, just execute the above query with pyodbc!
If you would still prefer to execute thousands of statements instead of just one
SQL="insert into table([Col1],[Col2],[Col3],[Col4],[Col5],[Col6],[Col7]) values (?,?,?,?,?,?,?)"
note that the ' sorrounding the ? shouldn't be there.
# creating column list for insertion
colsInsert = "["+"],[".join([str(i) for i in mydata.columns.tolist()]) +']'
# Insert DataFrame recrds one by one.
for i,row in mydata.iterrows():
sql = "INSERT INTO Test (" +colsInsert + ") VALUES (" + "%?,"*(len(row)-1) + "%?)"
cursor.execute(sql, tuple(row))
# cursor.execute(sql, tuple(row))
# the connection is not autocommitted by default, so we must commit to save our changes
c.commit()
I am trying to write into my localhost MySQL database.
I have created a database named "test", a table called "price_update" and a row called "model"
When I run the script below I get no errors, however, I also get nothing written to my database.
I am not sure where to start looking for the problem. the row is varchar(10) and collation utf9_general_ci.
import MySQLdb
conn = MySQLdb.connect(host="127.0.0.1",user="someUser",passwd="somePassword",db="test")
query = "INSERT INTO price_update (model) values ('12345')"
x = conn.cursor()
x.execute(query)
row = x.fetchall()
You have to commit the changes:
conn.commit()
Also, I'd make your query safer:
query = "INSERT INTO price_update (model) values (%s)"
...
x.execute(query, ('12345',))