We Keep Coding

Transfering Data in MS Access Using Python

I have an ever growing and changing database that reflects a permits passed by the State and EPA.
As the database changes and updates I need to transfer the relevant information.
The script does two things; first it checks which fields are the same and creates a list of fields and data that will be inserted into the new database. Second to insert the data into the new database.
Problem is I cannot get it to insert. I have matched everything like it says online in various ways but i get error ('42000', '[42000] [Microsoft][ODBC Microsoft Access Driver] Syntax error in INSERT INTO statement. (-3502) (SQLExecDirectW)').
I cannot figure out how to prevent it.
Code:
import pyodbc
importDatabase = r"J:\ENVIRO FIELD\AccessDatabases\MS4\MS4 Town Databases\~Template\MS4_Apocalypse Import DEV 1.accdb"
"Create the Import Database Connection"
connectionImport = pyodbc.connect(r'Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=%s;' %(importDatabase))
cursorImport = connectionImport.cursor()
"####---Outfall Section---####"
"Import the outfall names into the new database"
tbl = "tbl_Outfall_1_Profile"
exportList = []
importList = []
for row in cursorImport.columns(table = "tblExportMigration_Outfall_1_Profile"):
field = row.column_name
exportList.append(field)
for row in cursorImport.columns(table = "tbl_Outfall_1_Profile"):
field = row.column_name
importList.append(field)
matchingList = []
for field in exportList:
if field != "outfallID":
if field in importList:
matchingList.append(field)
else:
continue
sqlValue = ""
for field in matchingList:
sqlValue += "[%s], " %(field)
sqlValue = sqlValue[:-2]
sql = "SELECT %s from %s" %(sqlValue, "tblExportMigration_Outfall_1_Profile")
for rowA in cursorImport.execute(sql):
tupleList = list(rowA)
tupleList = ["" if i == None else i for i in tupleList]
tupleValues = tuple(tupleList)
sqlUpdate = """INSERT INTO tbl_Outfall_1_Profile (%s) Values %s;""" %(sqlValue, tupleValues)
cursorImport.execute(sqlUpdate)
cursorImport.close()
This is the sql string I create
"INSERT INTO tbl_Outfall_1_Profile ([profile_OutfallName], [profile_HistoricalName1], [profile_HistoricalName2], [profile_HistoricalName3], [profile_HistoricalName4]) Values ('756', '', '', '', '');"

Taking what #Gord Thompson said I was actually able to create a dynamic parameter flow
First created a module to create the ?
def Defining_Paramters(length):
parameterString = ""
for x in range(1,length):
parameterString += "?, "
parameterString += "?"
return parameterString
Then stuck it into the string for the sql update
sqlUpdate = sqlUpdate = "INSERT INTO %s (%s) Values (%s);" %(table, sqlFrameworkSubStr, parameters)
Run the cursor and commit it
cursorTo.execute(sqlUpdate, (dataTuple))
connectionTo.commit()
It would seem that you have to create the query in its entirety then have your data in tuple format for entry

This is the sql string [I think] I create
Try this:
sqlUpdate = """INSERT INTO tbl_Outfall_1_Profile (%s) Values (%s);""" %(sqlValue, tupleValues)
or perhaps:
sqlUpdate = "INSERT INTO tbl_Outfall_1_Profile (%s) Values (%s);" %(sqlValue, tupleValues)

pyscopg2: Is it possible to dynamically add %s in loop

I am trying to create a method in python insert records into a table passing in a list of column names, and an associated list of records.
I was able to set it up where the column names populated dynamically via a for loop, but I can't figure out how to do the same thing with values because the psycopg2.executemany function relies on having %s's as placeholders.
Is it possible to have the number of %s's in the string populate dynamically via a loop? Is there another way to do this?
def load_table(dbname,table_name,fields,records):
try:
#Variable Qty Column Loop
sql_fields = []
for i in fields:
i = sql.Identifier(i)
sql_fields.append(i)
#Need similar loop to replace %s values
#Replace (%s,%s,%s) ???
#.....
#.....
sql_values = []
for i in fields:
sql_values.append('%s')
print(sql_values)
flist = sql.SQL(',').join(sql_fields)
connection, cursor = create_connection(dbname)
insert_query = sql.SQL('INSERT INTO {table_name} ({fields}) VALUES (%s,%s,%s)').format(
table_name = sql.Identifier(table_name),
fields = flist,
cursor.executemany(insert_query,records)
print('Records Loaded Successfully')
except (Exception,psycopg2.Error) as error:
print("Failed to insert record into table {error}".format(error = error))
finally:
# closing database connection.
if (connection):
close_connection(connection,cursor)

You can use sql.Placeholder, to populate the sql statement with the amount of %s-placeholders you need:
def load_table(dbname,table_name,fields,records):
con, cur = create_connection('foo')
query = sql.SQL("insert into {} ({}) values ({})").format(
sql.Identifier(table_name),
sql.SQL(', ').join(map(sql.Identifier, fields)),
sql.SQL(', ').join(sql.Placeholder() * len(fields)))
print(query.as_string(con))
if __name__ == '__main__':
dbname = '...'
table_name = 'messages'
fields = ['user_id', 'message_type', 'message_title']
records = [['12345', 'json', 'my first message'], ]
load_table(dbname,table_name,fields,records)
Output:
insert into "messages" ("user_id", "message_type", "message_title") values (%s, %s, %s)

How can I let the id be auto generated in my database?

Here is my code:
import sqlite3
def insert(fields=(), values=()):
connection = sqlite3.connect('database.db')
# g.db is the database connection
cur = connection.cursor()
query = 'INSERT INTO this_database (%s) VALUES (%s)' % (
', '.join(fields),
', '.join(['?'] * len(values))
)
cur.execute(query, values)
connection.commit()
id = cur.lastrowid
cur.close()
print (id)
test example:
insert(fields = ("id", "file_name", "url", "time", "type", "description"), values = (2, "file1", "wwww.test.com", "1", "photo", "my first database test"))
I don't want to give the id manually.
I want it to add it+1 automatically.
How can I do that?

You have an INTEGER PRIMARY KEY column, which, if you leave it out when inserting items, automatically increments:
INSERT INTO this_database(file_name, url, time, type, description)
VALUES (?,?,?,?,?)
Since id is omitted, every time you insert a value using the above statement, it's automatically assigned a number by sqlite.
The documentation explaining this.

Not able to insert multiple columns in once using executemany

I have two variables to insert in my table.
user_id - int
marks - float
and I am having this data for multiple users like this:
user_ids = (-,-,-,-,-,-,-) **TUPLE**
marks = (-,-,-,-,-,-,-,-) **TUPLE**
I want to insert this data into my table using executemany and I am executing this query in my flask snippet:
con = pymysql.connect(
host=host,
user=user,
password=password,
db=db,
charset=charset,
cursorclass=pymysql.cursors.DictCursor,
port=port,
)
cur = con.cursor()
percs = calcattnonull()
# percs contains list of dictionaries.
# {[<'user_id'>: <'marks'>], [<'user_id'>: <'marks'>]........}
id_ = []
perc_ = []
final = []
for perc in tqdm(percs):
id_.append(perc["user_id"])
perc_.append(perc["att_perc"])
id_ = tuple(id_)
perc_ = tuple(perc_)
final.append(id_)
final.append(perc_)
cur.executemany(
"UPDATE dream_offline_calculate SET (user_id,att_percentage) VALUES (?,?)",
final,
)
con.commit()
I am getting this error again and again:
TypeError: not all arguments converted during string formatting
Thanks in advance for helping me.

executemany takes an iterable of the same placeholders you would use when calling execute several times.
So if your original query would be
cur.execute(
"UPDATE dream_offline_calculate SET (user_id,att_percentage) VALUES (?,?)",
(user_id, att_perc),
)
the equivalent executemany would be
cur.executemany(
"UPDATE dream_offline_calculate SET (user_id,att_percentage) VALUES (?,?)",
[(user_id, att_perc)],
)
So that said, simply
cur.executemany(
"UPDATE dream_offline_calculate SET (user_id,att_percentage) VALUES (?,?)",
[(perc["user_id"], perc["att_perc"]) for perc in percs],
)
should do the trick.

Python: Set param for columns and values pypyodbc - executemany

I have this situation where I created a method that will insert rows in database. I provide to that method columns, values and table name.
COLUMNS = [['NAME','SURNAME','AGE'],['SURNAME','NAME','AGE']]
VALUES = [['John','Doe',56],['Doe','John',56]]
TABLE = 'people'
This is how I would like to pass but it doesn't work:
db = DB_CONN.MSSQL() #method for connecting to MS SQL or ORACLE etc.
cursor = db.cursor()
sql = "insert into %s (?) VALUES(?)" % TABLE
cursor.executemany([sql,[COLUMNS[0],VALUES[0]],[COLUMNS[1],VALUES[1]]])
db.commit()
This is how it will pass query but problem is that I must have predefined column names and that's not good because what if the other list has different column sort? Than the name will be in surname and surname in name.
db = DB_CONN.MSSQL() #method for connecting to MS SQL or ORACLE etc.
cursor = db.cursor()
sql = 'insert into %s (NAME,SURNAME,AGE) VALUES (?,?,?)'
cursor.executemany(sql,[['John','Doe',56],['Doe','John',56]])
db.commit()
I hope I explained it clearly enough.
Ps. COLUMNS and VALUES are extracted from json dictionary
[{'NAME':'John','SURNAME':'Doe','AGE':56...},{'SURNAME':'Doe','NAME':'John','AGE':77...}]
if that helps.
SOLUTION:
class INSERT(object):
def __init__(self):
self.BASE_COL = ''
def call(self):
GATHER_DATA = [{'NAME':'John','SURNAME':'Doe','AGE':56},{'SURNAME':'Doe','NAME':'John','AGE':77}]
self.BASE_COL = ''
TABLE = 'person'
#check dictionary keys
for DATA_EVAL in GATHER_DATA:
if self.BASE_COL == '': self.BASE_COL = DATA_EVAL.keys()
else:
if self.BASE_COL != DATA_EVAL.keys():
print ("columns in DATA_EVAL.keys() have different columns")
#send mail or insert to log or remove dict from list
exit(403)
#if everything goes well make an insert
columns = ','.join(self.BASE_COL)
sql = 'insert into %s (%s) VALUES (?,?,?)' % (TABLE, columns)
db = DB_CONN.MSSQL()
cursor = db.cursor()
cursor.executemany(sql, [DATA_EVAL.values() for DATA_EVAL in GATHER_DATA])
db.commit()
if __name__ == "__main__":
ins = INSERT()
ins.call()

You could take advantage of the non-random nature of key-value pair listing for python dictionaries.
You should check that all items in the json array of records have the same fields, otherwise you'll run into an exception in your query.
columns = ','.join(records[0].keys())
sql = 'insert into %s (%s) VALUES (?,?,?)' % (TABLE, columns)
cursor.executemany(sql,[record.values() for record in records])
References:
https://stackoverflow.com/a/835430/5189811