We Keep Coding

Transfering Data in MS Access Using Python

I have an ever growing and changing database that reflects a permits passed by the State and EPA.
As the database changes and updates I need to transfer the relevant information.
The script does two things; first it checks which fields are the same and creates a list of fields and data that will be inserted into the new database. Second to insert the data into the new database.
Problem is I cannot get it to insert. I have matched everything like it says online in various ways but i get error ('42000', '[42000] [Microsoft][ODBC Microsoft Access Driver] Syntax error in INSERT INTO statement. (-3502) (SQLExecDirectW)').
I cannot figure out how to prevent it.
Code:
import pyodbc
importDatabase = r"J:\ENVIRO FIELD\AccessDatabases\MS4\MS4 Town Databases\~Template\MS4_Apocalypse Import DEV 1.accdb"
"Create the Import Database Connection"
connectionImport = pyodbc.connect(r'Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=%s;' %(importDatabase))
cursorImport = connectionImport.cursor()
"####---Outfall Section---####"
"Import the outfall names into the new database"
tbl = "tbl_Outfall_1_Profile"
exportList = []
importList = []
for row in cursorImport.columns(table = "tblExportMigration_Outfall_1_Profile"):
field = row.column_name
exportList.append(field)
for row in cursorImport.columns(table = "tbl_Outfall_1_Profile"):
field = row.column_name
importList.append(field)
matchingList = []
for field in exportList:
if field != "outfallID":
if field in importList:
matchingList.append(field)
else:
continue
sqlValue = ""
for field in matchingList:
sqlValue += "[%s], " %(field)
sqlValue = sqlValue[:-2]
sql = "SELECT %s from %s" %(sqlValue, "tblExportMigration_Outfall_1_Profile")
for rowA in cursorImport.execute(sql):
tupleList = list(rowA)
tupleList = ["" if i == None else i for i in tupleList]
tupleValues = tuple(tupleList)
sqlUpdate = """INSERT INTO tbl_Outfall_1_Profile (%s) Values %s;""" %(sqlValue, tupleValues)
cursorImport.execute(sqlUpdate)
cursorImport.close()
This is the sql string I create
"INSERT INTO tbl_Outfall_1_Profile ([profile_OutfallName], [profile_HistoricalName1], [profile_HistoricalName2], [profile_HistoricalName3], [profile_HistoricalName4]) Values ('756', '', '', '', '');"

Taking what #Gord Thompson said I was actually able to create a dynamic parameter flow
First created a module to create the ?
def Defining_Paramters(length):
parameterString = ""
for x in range(1,length):
parameterString += "?, "
parameterString += "?"
return parameterString
Then stuck it into the string for the sql update
sqlUpdate = sqlUpdate = "INSERT INTO %s (%s) Values (%s);" %(table, sqlFrameworkSubStr, parameters)
Run the cursor and commit it
cursorTo.execute(sqlUpdate, (dataTuple))
connectionTo.commit()
It would seem that you have to create the query in its entirety then have your data in tuple format for entry

This is the sql string [I think] I create
Try this:
sqlUpdate = """INSERT INTO tbl_Outfall_1_Profile (%s) Values (%s);""" %(sqlValue, tupleValues)
or perhaps:
sqlUpdate = "INSERT INTO tbl_Outfall_1_Profile (%s) Values (%s);" %(sqlValue, tupleValues)

Error: can only concatenate str (not "list") to str

I am trying to import txt file into sql, but i have an error:
TypeError: can only concatenate str (not "list") to str
My code:
import psycopg2
con = psycopg2.connect(
host = "",
database="",
user = "",
password = "")
cursor = con.cursor()
with open("pom1.txt") as infile:
for line in infile:
data = line.split()
print(data)
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES"
"(" + data + ");")
cursor.execute(query, *data)
con.commit()
Does anyone have an idea how can i solve it? :)

You don't put the actual values into the parameterized query; you put whatever placeholders are appropriate for your library.
data = line.split()
place_holders = ', '.join("%s" for _ in data) # Assuming %s is correct
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES"
"(" + place_holders + ");")
cursor.execute(query, *data)
cursor.execute takes care of inserting each value where a placeholder occurs, ensure things are properly quoted/escaped/etc.

There are several problems here. First, as the error says, you are trying to concatenate a List (which is data) directly to a string.
Second, you should not use + to concatenate your values and your query.
The doc says:
Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
You should only pass the values to the query via %s.
I'm not sure about the use of * in front of data in cursor.execute(query, *data).
Here is a code that should work, though I have nothing at hand for testing it right now:
import psycopg2
con = psycopg2.connect(
host = "",
database="",
user = "",
password = "")
cursor = con.cursor()
with open("pom1.txt") as infile:
for line in infile:
data = line.split()
print(data)
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES (%s, %s, %s, %s);")
cursor.execute(query, data)
con.commit()

Read from csv file and put to Mysql with different field type using python

i have table field of "datetime", "integer", "double", "string (varchar)" type in Mysql, how can i convert from string (from csv file) to each type of Mysql field using python ? example of my csv file :
EURUSD;M5;2011.01.05 02:10:00;1.33193;1.33193;1.33112;1.33135;0;-1;0;0;1.33215
My python code is like this :
import MySQLdb
# Open database connection
db = MySQLdb.connect("localhost","me","xxxxx","test" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
inputFile = "source.csv"
with open(inputFile, 'r') as fr:
data = fr.readlines()
for line in data:
words = line.strip().split(";")
getpair = words[0]
gettf = words[1]
gettime = (datetime) words[2]
getopen = (double) words[3]
gethigh = (double) words[4]
getlow = (double) words[5]
getclose = (double) words[6]
getcUpdown = (int) words[7]
getfractal = (int) words[8]
getzzId = (int) words[9]
getzzUpdown = (int) words[10]
getzzEndPrc = (double) words[11]
print(getpair,"=",gettf,"=",gettime,"=",getopen,"=",gethigh,"=",getlow,"=",getclose,"=",getcUpdown,"=",getfractal,"=",getzzId,"=",getzzUpdown,"=",getzzEndPrc)
# =====================================================================
# Prepare SQL query to INSERT a record into the database.
sql = '''INSERT INTO `mytable` (pair,timeframe,time,open,close,high,low,updown,fractal,zzid,zzupdown,zzlength) \
VALUES (getpair,gettf,gettime,getopen,getclose,gethigh,getlow,getcUpdown,getfractal,getzzId,getzzUpdown,getzzEndPrc)'''
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
fr.close()
Thanks
p

Type conversion in Python is done by:
int(words[7])
not
(int)words[7]
There is no double type in Python. Use float instead.
You can not convert directly to datetime. Use datetime.strptime() to parse the input string:
from datetime import datetime
gettime = datetime.strptime(words[2], '%Y.%m.%d %H:%M:%S')
You might not need to do this (conversion should be handled when using parameterised queries - see below), but you can convert that to a string format that MySQLDB supports. You can use strftime() for that:
gettime = gettime.strftime('%Y-%m-%d %H:%M:%S')
The sql query string needs to substitute the the actual values. Your
query is inserting the names of the variables, not the
values. You can (and should) use a parameterised query like this:
sql = 'INSERT INTO `mytable` (pair, timeframe, time, open, close, high, low, updown, fractal, zzid, zzupdown, zzlength) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)'
cursor.execute(sql, (getpair, gettf, gettime, getopen, getclose, gethigh, getlow, getcUpdown, getfractal, getzzId, getzzUpdown, getzzEndPrc))
cursor.execute(sql, ...) needs to be moved into the body of the for loop.

Chinese character doesn't support for Sqlite3 through CGI script

inserting Chinese character into Sqlite3 through Cgi script is not working for me. I can insert and select Chinese character from same database using Query browser tool but when I use python script for this, it's show error. This is the query i have used for create database
CREATE TABLE registrations (
m_username VARCHAR PRIMARY KEY
COLLATE 'BINARY',
m_identity VARCHAR,
m_updatetime DATETIME
);
and then this is the cgi script i have used for update and select values form the database
#! /Python26/python
dbFile = 'D:/sqlite/registrations'
import cgi
import sqlite3
import xml.sax.saxutils
query = cgi.parse()
db = sqlite3.connect(dbFile)
user = query.get('username', [None])[0]
identity = query.get('identity', [None])[0]
friends = query.get('friends', [])
print 'Content-type: text/plain\n\n<?xml version="1.0" encoding="utf-8"?>\n'
print "<result>"
if user:
try:
c = db.cursor()
c.execute("insert or replace into registrations values (?, ?, datetime('now'))", (user, identity))
print "\t<update>true</update>"
except:
print '\t<update>false</update>'
for f in friends:
print "\t<friend>\n\t\t<user>%s</user>" % (xml.sax.saxutils.escape(f), )
c = db.cursor()
c.execute("select m_username, m_identity from registrations where m_username = ? and m_updatetime > datetime('now', '-1 hour')", (f, ))
for result in c.fetchall():
eachIdent = result[1]
if not eachIdent:
eachIdent = ""
print "\t\t<identity>%s</identity>" % (xml.sax.saxutils.escape(eachIdent), )
if f != result[0]:
print "\t\t<registered>%s</registered>" % (xml.sax.saxutils.escape(result[0]), )
print "\t</friend>"
db.commit()
print "</result>"
I think, i need to set CHARSET as UTF-8 something, but I don't know how to do it. i was googled, but couldn't find good way to solve this issue. kindly some one help me please.

I have done this through the client side. I just used EncodeBase64 and encoded the chinese data and send to the db. I think, this one is not straight way. but I couldn't find any other way.

PyODBC - cursor record value rounding

I am working on a Python script (Python version 2.5.1 on Windows XP) that involves connecting to a Microsoft Access (.mdb) database to read values from a table. I'm getting some unexpected results with one record whereby the field of interest precision is getting rounded.
I know the Access table field of interest is a Double data type. But, the value that caused me to discover this in the table is 1107901035.43948. When I read the value in the Python code and print it out, it's showing 1107901035.44.
Is there a pyODBC connection parameter or other that must be set? I couldn't find anything in the documentation
Here's what my code looks like (the intention is to resolve unneeded records by identifying the record that has the greatest value for my field of interest):
conn = pyodbc.connect('DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=' + pGDB)
conn.autocommit = True
cursor = conn.cursor()
tableList = []
for x in cursor.tables():
val = str(x[2])
if val[0:3] <> "MSy":
if val[0:3] <> "GDB":
if val[-5:] <> "Index":
tableList.append(val)
for x in tableList:
try:
SQL = "SELECT * FROM %s" % (x)
cursor.execute(SQL)
rows = cursor.fetchall()
counter = 0
for row in rows:
counter +=1
if counter > 1:
print "Site %s is a multipart basin" % (x)
SQL = "SELECT MAX(Shape_Area) AS AREA FROM %s" % (x)
cursor.execute(SQL)
row = cursor.fetchone()
val = row.AREA
print str(val)
SQL = "DELETE * FROM %s WHERE Shape_Area < %s" % (x, val)
cursor.execute(SQL)
thanks,
Tom

Django uses the Jinja template, so you can use its round filer. It works as follows:
template.html
<p>{{ VALUE| round(2, 'floor') }}</p>
Check out Jinja's documentation on topic
The SQL round function can also do this job:
SQL = "SELECT ROUND(MAX(Shape_Area), 2) AS AREA FROM %s" % (x)