We Keep Coding

Inserting a Python List in Python into an SQL Table

I have code that reads from a socket and creates a list called i. The socket is read, the list is created from the socket, the list gets printed then deleted. This gets repeated in a while true loop. Instead of just printing the list, I'd like to insert it into a table in my DB. I already have the cursor and connection established in the code. I was messing around with some other stuff but keep getting errors. I would like to use REPLACE INTO instead of INSERT INTO. Thank you very much for the help.
This is an example of what the list will look like.
'Dec-11-2018,','12:28:43,','iPhone,','alpha,','lib,','lib,','(45.67.67)\n']
My table name is StudentPrototype and it has 7 columns
Columns - (Date,Time,Device,ID,AP,APGroup,MACAdd)
#!/bin/python
import socket
import os, os.path
import MySQLdb as mdb
con = mdb.connect('localhost', 'user', 'pass',
'StudentTacker');
cur = con.cursor()
cur.execute("SELECT VERSION()")
i = []
def ParseArray(l): #parses line in socke
i.append(l.split()[+0] + '-') # Gets Day
i.append(l.split()[+1] + '-') # Gets Month
i.append(l.split()[+3] + ',') # Gets Year
i.append(l.split()[+2] + ',') # Gets Time
i.append(l.split()[-2] + ',') # Gets Device
i.append(l.split()[+9] + ',') # Gets ID
i.append(l.split()[+18] + ',') # Gets AP
i.append(l.split()[+19] + ',') # Gets AP Group
i.append(l.split()[+16] + '\n') # Gets MAC
# This is where I want to REPLACE INTO my table called StudentTest using list i
print(i)
del i[:]
if os.path.exists("/-socket"):
os.remove("/-socket")
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sock.bind("/home/socket")
infile = sock.makefile('r')
while True:
l = sock.recv(4096).decode()
ParseArray(l)
Update: I tried another method that I found on this site for how to insert python lists in a db.
Here is my new code that I used inside my function:
def ParseArray(l): #parses line in socke
i.append(l.split()[+0] + '-') # Gets Day
i.append(l.split()[+1] + '-') # Gets Month
i.append(l.split()[+3] + ',') # Gets Year
i.append(l.split()[+2] + ',') # Gets Time
i.append(l.split()[-2] + ',') # Gets Device
i.append(l.split()[+9] + ',') # Gets FSU ID
i.append(l.split()[+18] + ',') # Gets AP
i.append(l.split()[+19] + ',') # Gets AP Group
i.append(l.split()[+16] + '\n') # Gets MAC
#insert line into db else by primary key mac adresss
#update line to db if mac adress doesn't exist
params = ['?' for item in i]
sql = 'REPLACE INTO SocketTest (month, day, year, time, device,
Id, ap, ApGroup, MacAdd) VALUES (%s); ' % ', '.join(params)
cur.execute(sql, i)
Using that I'm getting an error:
Traceback (most recent call last):
File "./UnixSocketReader9.py", line 55, in <module>
ParseArray(l)
File "./UnixSocketReader9.py", line 28, in ParseArray
cur.execute(sql, i)
File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line
187, in execute
query = query % tuple([db.literal(item) for item in args])
TypeError: not all arguments converted during string formatting

Reading .sql File in for Execution in Python (pymysql)

I'm attempting to create a multiscript tool, that will take an argument of a .sql file and execute it.
I've set up a simple test, just executing on one database, however the syntax is giving me issues every time.
DELIMITER $$
CREATE DEFINER=`a_user`#`%` PROCEDURE `a_procedure`(
IN DirectEmployeeID TEXT,
IN StartRange DATE,
IN EndRange DATE
)
BEGIN
SELECT aColumn
WHERE thisThing = 1;
END$$
DELIMITER ;
To be clear, this script has been tested, and works when passed like :
mysql -uuser -p -hhost -Pport databaseName < file.sql
and also works through mysql workbench.
I saw this type of solution on another site:
with conn.cursor() as cursor:
f = sys.argv[1]
file = open(f, 'r')
sql = " ".join(file.readlines())
cursor.execute(sql)
which gives me a MySQL syntax error:
pymysql.err.ProgrammingError: (1064, u"You have an error in your SQL syntax;
check the manual that corresponds to your MySQL server version for the right
syntax to use near 'DELIMITER $$\n CREATE DEFINER=`a_user`#`%` PROCEDURE
`MyCommissionsDirect`(\n \tIN ' at line 1")
as you can see, there are newline characters within the script that mysql isn't liking.
I then tried this:
with conn.cursor() as cursor:
f = sys.argv[1]
file = open(f, 'r')
sql = ''
line = file.readline()
while line:
sql += ' ' + line.strip('\n').strip('\t')
line = file.readline()
print sql
cursor.execute(sql)
and get another syntax issue, the print shows that this is all one line, which is not working in mysqlworkbench. doesn't even try to execute it, which is strange.
When I put the DELIMETER $$ on a separate line first, it executes in mysqlworkbench.
This is one of those situations where I feel like I may be making this more and more complicated. I'm very surprised pymysql doesn't have a way of simply executing a sql file directly. I'm weary of trying to do string manipulation and get this working for this particular file, because then the dream of making this tool ambiguous and reusable kind of goes out the door.
Am I going about this in the complete incorrect way?
Thanks!

Here is my solution for using an SQL file with PyMySQL. The files contain many requests ended by ; which is used to split requests in a list. So beware of the missing ; in the list.
I decided to add the missing ; not in the function to spar a for loop. Maybe there is a better way.
create-db-loff.sql :
DROP DATABASE IF EXISTS loff;
CREATE DATABASE loff CHARACTER SET 'utf8';
USE loff;
CREATE TABLE product(
`id` INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
`code` BIGINT UNSIGNED NOT NULL UNIQUE,
`name` VARCHAR(200),
`nutrition_grades` VARCHAR(1)
);
CREATE TABLE category(
`id`INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
`name` VARCHAR(200)
);
CREATE TABLE asso_prod_cat(
`category_id` INT UNSIGNED NOT NULL,
`product_id` INT UNSIGNED NOT NULL,
CONSTRAINT `fk_asso_prod_cat_category`
FOREIGN KEY(category_id)
REFERENCES category(id)
ON DELETE CASCADE,
CONSTRAINT `fk_asso_prod_cat_product`
FOREIGN KEY(product_id)
REFERENCES product(id)
ON DELETE CASCADE
);
db.py :
DB_CONFIG = {
'host': 'localhost',
'user': 'loff',
'pass': 'loff',
'db': 'loff',
'char': 'utf8',
'file': 'create-db-loff.sql'
}
def get_sql_from_file(filename=DB_CONFIG['file']):
"""
Get the SQL instruction from a file
:return: a list of each SQL query whithout the trailing ";"
"""
from os import path
# File did not exists
if path.isfile(filename) is False:
print("File load error : {}".format(filename))
return False
else:
with open(filename, "r") as sql_file:
# Split file in list
ret = sql_file.read().split(';')
# drop last empty entry
ret.pop()
return ret
request_list = self.get_sql_from_file()
if request_list is not False:
for idx, sql_request in enumerate(request_list):
self.message = self.MSG['request'].format(idx, sql_request)
cursor.execute(sql_request + ';')

DELIMITER is command used by a MySQL interpreter, such as the command line or Workbench, and not an actual MySQL command.
I ended up working in some logic in my Python application to disable execution of MySQL queries when DELIMITER has been defined, then to execute when DELIMITER has been defined again:
import MySQLdb
import re
file = 'somesql.sql'
conn = MySQLdb.Connection(mysqlserver, mysqluser, mysqlpass, mysqldb)
curs = conn.cursor()
ignorestatement = False # by default each time we get a ';' that's our cue to execute.
statement = ""
for line in open(file):
if line.startswith('DELIMITER'):
if not ignorestatement:
ignorestatement = True # disable executing when we get a ';'
continue
else:
ignorestatement = False # re-enable execution of sql queries on ';'
line = " ;" # Rewrite the DELIMITER command to allow the block of sql to execute
if re.match(r'--', line): # ignore sql comment lines
continue
if not re.search(r'[^-;]+;', line) or ignorestatement: # keep appending lines that don't end in ';' or DELIMITER has been called
statement = statement + line
else: # when you get a line ending in ';' then exec statement and reset for next statement providing the DELIMITER hasn't been set
statement = statement + line
# print "\n\n[DEBUG] Executing SQL statement:\n%s" % (statement)
try:
curs.execute(statement)
conn.commit()
statement = ""
except curs.Error, e:
print(file + " - Error applying (" + str(e) + ")\nTerminating.")
sys.exit(1)
It's a bit hacky, but seems to work well enough.

Most SQL files contain interpreter commands such as DELIMITER that make passing the commands through to pymysql somewhat difficult, this code snippet allows you to separate out the statements in the sql file into a list for sequential execution.
def parse_sql(filename):
data = open(filename, 'r').readlines()
stmts = []
DELIMITER = ';'
stmt = ''
for lineno, line in enumerate(data):
if not line.strip():
continue
if line.startswith('--'):
continue
if 'DELIMITER' in line:
DELIMITER = line.split()[1]
continue
if (DELIMITER not in line):
stmt += line.replace(DELIMITER, ';')
continue
if stmt:
stmt += line
stmts.append(stmt.strip())
stmt = ''
else:
stmts.append(line.strip())
return stmts
Usage example:
conn = pymysql.connect('test')
stmts = parse_sql('my_sql_file.sql')
with conn.cursor() as cursor:
for stmt in stmts:
cursor.execute(stmt)
conn.commit()

It's simple code
import pymysql
class ScriptRunner:
def __init__(self, connection, delimiter=";", autocommit=True):
self.connection = connection
self.delimiter = delimiter
self.autocommit = autocommit
def run_script(self, sql):
try:
script = ""
for line in sql.splitlines():
strip_line = line.strip()
if "DELIMITER $$" in strip_line:
self.delimiter = "$$"
continue
if "DELIMITER ;" in strip_line:
self.delimiter = ";"
continue
if strip_line and not strip_line.startswith("//") and not strip_line.startswith("--"):
script += line + "\n"
if strip_line.endswith(self.delimiter):
if self.delimiter == "$$":
script = script[:-1].rstrip("$") + ";"
cursor = self.connection.cursor()
print(script)
cursor.execute(script)
script = ""
if script.strip():
raise Exception("Line missing end-of-line terminator (" + self.delimiter + ") => " + script)
if not self.connection.get_autocommit():
self.connection.commit()
except Exception:
if not self.connection.get_autocommit():
self.connection.rollback()
raise
if __name__ == '__main__':
connection = pymysql.connect(host="127.0.0.1", user="root", password="root", db="test", autocommit=True)
sql = ""
ScriptRunner(connection).run_script(sql)

Can python sqlalchemy use a variable to construct an insert statement (for mysql database)

Thanks in advance for advice on this problem...
I am trying to create a python script to import a set of CSVs into a mysql database.
Each CSV filename matches the destination table. The first row of each CSV matches the fields of the table. Each CSV / table has a different number of fields, field names, etc.
The problem I am having is with this line (full code below)
ins = table_name.insert().values(temp_variable_name)
where I want to dynamically update the destination table (table_name) and the insert command (temp_variable_name).
So when reading the labels.csv file, this should produce
ins = labels.insert().values(id_label=d[0], label_name=d[1])
and when reading the company.csv file, this should produce
ins = company.insert().values(id_company=d[0], company_name=d[1], ticker=d[2])
The problem is if I generate a string,
temp_variable_name = 'id_company=d[0], company_name=d[1], ticker=d[2]'
I end up getting a 'str' object has no attribute 'items' error.
Is there any way to dynamically generate an insert command for an SQL statement?
Portion of the script below:
# files list contains a list of all of the files in the directory
# we read in CSVs, isolate the first row to determine table field names
# the rest of the data should then be imported into the table with the corresponding name as the CSV
for f in files:
if '.csv' in f :
# read in each CSV file
# these are a Class / Function I've set up to read files
x = Read_Files()
data = x.read_file_lines_strip(path, f)
temp = data[0].replace('"','') # get rid of quotation marks from the data
table_header_list = temp.split('|') # get the first row, which is the table field names
variable_name ='' # this is used to construct the insert into table string
for x in xrange (0, len(table_header_list)) :
if x == 0 :
variable_name = variable_name + table_header_list[0] + '=d[0]'
elif x == len(table_header_list) :
variable_name = variable_name + table_header_list[x] + '=d[' + str(x) + ']'
else :
variable_name = variable_name + ', ' + table_header_list[x] + '=d[' + str(x) + ']'
table_name = f.replace('.csv','') # remove the .csv from filename to isolate the file name, which is the same as table name
# data from file
for data_line in data[1:] :
data_line = data_line.replace('"', '') # remove quotation marks
d = data_line.split('|') # split the line which is delimited by a |
# used to construct the final insert string
for x in xrange(0, len(table_header_list)) :
if x == 0 :
temp_variable_name = variable_name.replace('d[0]', d[0])
else :
temp_variable_name = temp_variable_name.replace('d[' + str(x) + ']', d[x])
try:
# table name is the table to insert into, via the CSV filename
# temp_variable_name is the insert string, such as 'id_company=d[0], company_name=d[1], ticker=d[2]'
ins = table_name.insert().values(temp_variable_name)
result = conn.execute(ins)
except Exception, e :
print 'error : ' + str(e)

You can do this with Insert objects and the csv module which makes it easy with the DictReader class. Here's an example for the company table:
import csv
from sqlalchemy import create_engine
from sqlalchemy.sql import table, column
NULL_FIELD_VALUE = r'\N'
DB_CONNECT = 'sqlite:///company.db'
engine = create_engine(DB_CONNECT, echo=True)
conn = engine.connect()
with open('company.csv') as csvfile:
reader = csv.DictReader(csvfile, delimiter='|')
insert_table = table('company',
*[column(field) for field in reader.fieldnames])
insert_dict = [{k: None if v == NULL_FIELD_VALUE else v
for k,v in row.items()}
for row in reader]
conn.execute(insert_table.insert(), insert_dict)

TypeError: 'NoneType' object is not subscriptable [duplicate]

This question already has an answer here:
mysqldb .. 'NoneType' object is not subscriptable
(1 answer)
Closed 8 years ago.
The error: names = curfetchone()[0]
TypeError: 'NoneType' object is not subscriptable.
I tried checking the indentation but still there's an error. I read that maybe the variable names returns 'None' if there is no record of the filename in the database.
I use the same variable 'names' in other 'def' and it works fine. I'm sure it has to do with the 'None' value being returned.
global filename
global t
try:
con = sqlite3.connect('textdb.db')
cur = con.cursor()
filename = tkinter.simpledialog.askstring("Open file...", "Input filename to open:")
if filename != None:
cur.execute("SELECT file_name FROM file_info WHERE file_name = ?", (filename,))
names = cur.fetchone()[0]
same = str(names)
if filename == same:
cur.execute("SELECT file_content FROM file_info WHERE file_name = ?", (filename,))
content = cur.fetchone()[0]
t = str(content)
text.delete(0.0, END)
text.insert(0.0, t)
else:
result = tkinter.messagebox.askyesno(title="File doesn't exist", message="'"+(filename)+"' doesn't exist. Make a new file using '"+(filename)+"'?")
if result == True:
cur.execute("SELECT COUNT(*) FROM file_info")
count_row = cur.fetchone()
cntdata = count_row[0]
incr = (cntdata + 1)
t = str(text.get(0.0, END))
curtime = str(ctime())
cur.execute("INSERT OR IGNORE INTO file_info VALUES(?, ?, ?, ?, ?)", (incr, filename, t, curtime, curtime,))
con.commit()
except sqlite3.Error:
if con:
con.rollback()
finally:
if con:
con.close()

If, due to whatever reason (empty result set?) curfetchone() returns None, a [0] access is of course forbidden (as the error message clearly says).
So better do that in two steps and do
row = curfetchone()
if row is not None:
names = row[0]
# proceed
else:
# act appropriately

Python-xml parse using beautifulsoup4 and writing the output to mysql db - unicode error

I'm trying to parse an xml file using beautifulsoup4.
IDE : LICLIPSE
Python version: 2.7
xml encoding : utf-8
Sample xml file : http://pastebin.com/RhjvyKDN
Below is the code I used to parse the xml files and write the extracted information to a local mysql database.
from bs4 import BeautifulSoup
import pymysql
import os, os.path
#strips apostrophes from the text and then just adds them at the beginning and end for the query
def apostro(text):
text= text.replace("'","")
text= text.replace(",","")
text = "'"+text+"'"
return text
#sets up the MYSQL connection
conn = pymysql.connect(host='127.0.0.1', user='xxxx', passwd='xxxx', db='mysql', port= 3306 )
cur = conn.cursor()
#drop all of the previous values from the database
cur.execute("DELETE FROM db WHERE title is not null")
conn.commit()
#loop through all of the files
for root, _, files in os.walk("C:/usc/xml"):
for f in files:
#j is a counter for how many sections we have processed
j=0
#fullpath is the location of the file we're parsing
fullpath = os.path.join(root, f)
print(fullpath)
#open file using BeautifulSoup
soup = BeautifulSoup(open(""+fullpath+""), 'xml')
sec = soup.find_all("section", {"style" : "-uslm-lc:I80"})
t = soup.main.title
t_num = t.num['value']
#if not clauses are needed in case there is a blank, otherwise an error is thrown
if not t.heading.text:
t_head = ''
else:
t_head = t.heading.text.encode('ascii', 'ignore').encode("UTF-8")
for element in sec:
if not element.num['value']:
section = ''
else:
section = element.num['value'].encode('ascii', 'ignore').encode("UTF-8")
if not element.heading:
s_head = ''
else:
s_head = element.heading.text.encode('ascii', 'ignore').encode("UTF-8")
if not element.text:
s_text = ''
else:
s_text = element.text.encode('ascii', 'ignore').encode("UTF-8")
#inserttest is the sql command that 'cur' executes. counter is printed every time a section is written to let me know the program is still alive
inserttest = "insert into deadlaws.usc_new (title, t_head, section, s_head, s_text) values (" + t_num + "," + apostro(t_head) + "," + apostro(section) + "," + apostro(s_head) + "," + apostro(s_text) +")"
j=j+1
cur.execute( inserttest)
conn.commit()
print(fullpath + " " +str(j))
conn.commit()
cur.close()
conn.close()
Everything went well until I noticed that the program ignores the hyphens '-' in the section numbers which makes the entire activity wrong.
I know I have used 'ignore' in the encode statement, but a hyphen '-' is a legitimate character in ascii, right? Shouldn't it be writing the character to the db instead of ignoring it?
I did a lot of reading on SO and elsewhere.
I've tried including from_encoding="utf-8" in the soup statement, 'xmlrefreplace' in the encode() statement and other methods, which have resulted in the below output : it writes this a– (some special unicode character) instead of a hyphen '-' to the database.
Sample output:
The data is huge and I'm afraid there could be other characters like - that are being ignored by the program. It's ok if it ignores special characters from the t_head, s_head and s_text fields as they are text but not the section column.
Any help in resolving this issue would be greatly appreciated.

Don't encode, the MySQL library is perfectly capable of inserting Unicode text into the database directly. Use SQL parameters, not string interpolation, and specify the character set to use when connecting to the database:
conn = pymysql.connect(host='127.0.0.1', user='xxxx', passwd='xxxx',
db='mysql', port=3306,
charset='utf8')
Don't encode:
t_head = t.heading.text or ''
for element in sec:
if not element.num['value']:
section = ''
else:
section = element.num.get('value', '')
s_head = element.heading.text or ''
s_text = element.text or ''
inserttest = "insert into deadlaws.usc_new (title, t_head, section, s_head, s_text) values (?, ?, ?, ?)"
cur.execute(inserttest, (t_num, t_head, section, s_head, s_text))