We Keep Coding

Not able to reach generated lists in other .py file

I need some help to solve the problem :
I have two files : "main.py" and let's say "script.py"
The "script.py" contains a func() that generates variable amount of lists (programm uses Database, so amount of lists and data inside is vary):
def func():
try:
connect = psycopg2.connect(database = 'car_rental',
user = 'postgres',
password = 'datapass')
curs = connect.cursor()
#iteration that creates the list contains all models available for every single brand with name '{category}_{brand.lower()}'
# (in this case: economy_{brand.lower()})
temp_var = ''
for brand in economy_brands_list:
curs.execute("""SELECT DISTINCT model
FROM fleet
WHERE category = %s and brand = %s""", ('economy', f'{brand}'))
expres = [x[0] for x in curs.fetchall()]
temp_var+=f"economy_{brand.lower()} = {expres}\n"
exec(temp_var)
finally:
curs.close()
connect.close()
In "main.py" i want to use the list(s) generated in func(). So i imported func() to 'main.py', call the func(), but it gives the error instead.NameError : name 'economy_{brand}' is not defined. From the 'script.py' (which contains func()) the function works and i'm able to print the lists generated. How to make 'main.py' to define the lists generated in func()?
Thank You in advance.

To expand on Tierry Lathuille's answer, you may want something like this:
def func():
economy_brands = {}
try:
connect = psycopg2.connect(database = 'car_rental',
user = 'postgres',
password = 'datapass')
curs = connect.cursor()
#iteration that creates the list contains all models available for every single brand with name '{category}_{brand.lower()}'
# (in this case: economy_{brand.lower()})
temp_var = ''
for brand in economy_brands_list:
curs.execute("""SELECT DISTINCT model
FROM fleet
WHERE category = %s and brand = %s""", ('economy', f'{brand}'))
expres = [x[0] for x in curs.fetchall()]
economy_brands[f"economy_{brand.lower()}"] = expres
finally:
curs.close()
connect.close()
return economy_brands

Use cursor.executemany instead of cursor.execute with tying panda header to sql variables

I have a python script that updates rows in an oracle sql table correctly, however I am using cursor.execute and try/except so if one update fails, it kills the whole run.
I want to be able to have it run through the whole update and just log the error and move onto the next one, which is where cursor.executemany comes in.
https://cx-oracle.readthedocs.io/en/latest/user_guide/batch_statement.html
Here is the script, it works great, except for the all or nothing error approach.
#oracle sql update statement for SHRTCKN
banner_shrtckn_update = """
UPDATE SATURN.SHRTCKN A
SET A.SHRTCKN_COURSE_COMMENT = :course_comment,
A.SHRTCKN_REPEAT_COURSE_IND = :repeat_ind,
A.SHRTCKN_ACTIVITY_DATE = SYSDATE,
A.SHRTCKN_USER_ID = 'STU00940',
A.SHRTCKN_DATA_ORIGIN = 'APPWORX'
WHERE A.SHRTCKN_PIDM = gb_common.f_get_pidm(:id) AND
A.SHRTCKN_TERM_CODE = :term_code AND
A.SHRTCKN_SEQ_NO = :seqno AND
A.SHRTCKN_CRN = :crn AND
A.SHRTCKN_SUBJ_CODE = :subj_code AND
A.SHRTCKN_CRSE_NUMB = :crse_numb
"""
def main():
# get time of run and current year
now = datetime.datetime.now()
year = str(now.year)+"40"
# configure connection strings for banner PROD
db_pass = os.environ['DB_PASSWORD']
dsn = cx_Oracle.makedsn(host='FAKE', port='1521', service_name='TEST.FAKE.BLAH')
try: # initiate banner db connection -- PROD
banner_cnxn = cx_Oracle.connect(user=config.db_test['user'], password = db_pass, dsn=dsn)
writeLog("---- Oracle Connection Made ----")
insertCount = 0
for index, row in df.iterrows():
shrtcknupdate(row,banner_cnxn)
insertCount = insertCount + 1
banner_cnxn.commit()
banner_cnxn.close()
writeLog(str(insertCount)+" rows updated")
except Exception as e:
print("Error: "+str(e))
writeLog("Error: "+str(e))
def writeLog(content):
print(content)
log.write(str(datetime.date.today())+" "+content+"\n")
#define the variable connection between panda/csv and SHRTCKN table
def shrtcknupdate(row, connection):
sql = banner_shrtckn_update
variables = {
'id' : row.Bear_Nbr,
'term_code' : row.Past_Term,
'seqno' : row.Seq_No,
'crn' : row.Past_CRN,
'subj_code' : row.Past_Prefix,
'crse_numb' : row.Past_Number,
'course_comment' : row.Past_Course_Comment,
'repeat_ind' : row.Past_Repeat_Ind
}
cursor = connection.cursor()
cursor.execute(sql, variables)
if __name__ == "__main__":
writeLog("-------- Process Start --------")
main()
writeLog("-------- Process End --------")
The executemany option, I can turn on batcherrors=True
and it will do exactly what I need.
The problem I am running into, is if I get rid of the for loop that runs through the excel/panda dataframe to update the oracle sql rows, which is not needed when doing the update in batch, then how do I attach the column headers to the sql update statement.
If I leave in the for loop, I get this error when using executemany:
Error: parameters should be a list of sequences/dictionaries or an integer specifying the number of times to execute the statement

For named binds, you need to provide a list of dictionaries. This list can be obtained by to_dict(orient='records'):
‘records’ : list like [{column -> value}, … , {column -> value}]
banner_shrtckn_update = """
UPDATE SATURN.SHRTCKN A
SET A.SHRTCKN_COURSE_COMMENT = :Past_Course_Comment,
A.SHRTCKN_REPEAT_COURSE_IND = :Past_Repeat_Ind,
A.SHRTCKN_ACTIVITY_DATE = SYSDATE,
A.SHRTCKN_USER_ID = 'STU00940',
A.SHRTCKN_DATA_ORIGIN = 'APPWORX'
WHERE A.SHRTCKN_PIDM = gb_common.f_get_pidm(:Bear_Nbr) AND
A.SHRTCKN_TERM_CODE = :Past_Term AND
A.SHRTCKN_SEQ_NO = :Seq_No AND
A.SHRTCKN_CRN = :Past_CRN AND
A.SHRTCKN_SUBJ_CODE = :Past_Prefix AND
A.SHRTCKN_CRSE_NUMB = :Past_Number
"""
def main():
# get time of run and current year
now = datetime.datetime.now()
year = str(now.year)+"40"
# configure connection strings for banner PROD
db_pass = os.environ['DB_PASSWORD']
dsn = cx_Oracle.makedsn(host='FAKE', port='1521', service_name='TEST.FAKE.BLAH')
try: # initiate banner db connection -- PROD
banner_cnxn = cx_Oracle.connect(user=config.db_test['user'], password = db_pass, dsn=dsn)
writeLog("---- Oracle Connection Made ----")
# batch execute banner_shrtckn_update
cursor = banner_cnxn.cursor()
data = df[['Bear_Nbr', 'Past_Term', 'Seq_No', 'Past_CRN', 'Past_Prefix', 'Past_Number', 'Past_Course_Comment', 'Past_Repeat_Ind']].to_dict('records')
cursor.executemany(banner_shrtckn_update, data, batcherrors=True)
for error in cursor.getbatcherrors():
writeLog(f"Error {error.message} at row offset {error.offset}")
banner_cnxn.commit()
banner_cnxn.close()
except Exception as e:
print("Error: "+str(e))
writeLog("Error: "+str(e))
This isn't described in detail in the documenation, but you can find an example for named binds in python-cx_Oracle/samples/bind_insert.py.
(Please note that I adjusted the variable names in your sql statement to the dataframe column names to avoid renaming of the columns at creating data.)

Strange escape character added to record's message in Python logger

I am facing a weird issue.
The following code
logger.info("Setting coordinates to [lat: " + str(self.curr_coord.latitude())
+ ", lng: " + str(self.curr_coord.longitude())
+ "]")
The values returned by the functions mentioned above are simple double.
produces a string, which contains escape character after str(self.curr_coord.latitude()). In my custom logging handler I am connecting to a SQLite3 database file. In addition I have the ability to dump the log records into a CSV file.
The code above produces an entry that is exported the following entry
2021-10-26T14:47:39.528605+02:00,0,geointelkit,"Setting coordinates to [lat: 48.9475, lng: 8.4106]"
My CSV writer looks like this:
with open(db_dump_filename, "w", newline="") as csv_dump_file:
cols = ["Timestamp", "Level", "Source", "Message"]
db_exporter = csv.writer(csv_dump_file)
db_exporter.writerow(cols)
db_exporter.writerows(entries)
All of my other entries do not have quotation marks. At first I thought it may be the length of the message and some weird stuff SQLite is doing with it. However after adding an even longer string I didn't get the same problem.
The next thing I did was to customize the CSV writer:
with open(db_dump_filename, "w", newline="") as csv_dump_file:
cols = ["Timestamp", "Level", "Source", "Message"]
db_exporter = csv.writer(csv_dump_file,
quoting=csv.QUOTE_NONE,
delimiter=",",
escapechar="#")
db_exporter.writerow(cols)
db_exporter.writerows(entries)
The modified writer's output for that entry was
2021-10-26T14:47:39.528605+02:00,0,geointelkit,Setting coordinates to [lat: 48.9475#, lng: 8.4106]
Notice the # after 48.9475. While I do some formatting in my custom handler in regards to the records it handles it is only related to the timestamp (first column) and value after it (second column), which is simply mapping the logger level numeric values (20, 30, 40...) to the ones I use in my application. The record message is not touched in any way. Here is the formatter I'm using in my custom logger class:
self.formatter = logging.Formatter(fmt="%(asctime)s %(levelno)d %(name)s %(message)s",
datefmt="%Y-%m-%dT%H:%M:%S")
The custom handler and custom logger are shown below. LogLevel is a simple class that includes (beside my different levels of logging) functions for mapping a log level to different representation. I didn't include it since it's not important.
LogEntry = namedtuple("LogEntry", "dtime lvl src msg")
class LogHandlerSQLite(logging.Handler):
db_init_script = """
CREATE TABLE IF NOT EXISTS logs(
TimeStamp TEXT,
Level INT,
Source TEXT,
Message TEXT
);
"""
db_clear_script = """
DROP TABLE IF EXISTS logs;
"""
db_insert_script = """
INSERT INTO logs(
Timestamp,
Level,
Source,
Message
)
VALUES (
'%(tstamp)s',
%(levelno)d,
'%(name)s',
'%(message)s'
);
"""
db_get_all = """
SELECT *
FROM logs
"""
def __init__(self, db_path="logs.db"):
logging.Handler.__init__(self)
self.db = db_path
conn = sq3.connect(self.db)
conn.execute(LogHandlerSQLite.db_init_script)
conn.commit()
conn.close()
def dump(self):
logger = logging.getLogger("geointelkit")
logger.info("Dumping log records to CSV file")
conn = sq3.connect(self.db)
cur = conn.cursor()
cur.execute(self.db_get_all)
entries = cur.fetchall()
conn.close()
db_dump_filename = self.rename_db_file(add_extension=False) + ".csv"
with open(db_dump_filename, "w", newline="") as csv_dump_file:
cols = ["Timestamp", "Level", "Source", "Message"]
db_exporter = csv.writer(csv_dump_file,
quoting=csv.QUOTE_NONE,
delimiter=",",
escapechar="#")
db_exporter.writerow(cols)
db_exporter.writerows(entries)
def clear_entries(self):
conn = sq3.connect(self.db)
# Drop current logs table
conn.execute(LogHandlerSQLite.db_clear_script)
# Shrink size of DB on the filesystem as much as possible
conn.execute("VACUUM")
# Create an empty one
conn.execute(LogHandlerSQLite.db_init_script)
conn.commit()
conn.close()
def format_time(self, record):
# FIXME Timestamp includes microseconds
record.tstamp = datetime.datetime.fromtimestamp(record.created, datetime.timezone.utc).astimezone().isoformat()
def emit(self, record):
"""
Emits a log entry after processing it. The processing includes
* formatting the entry's structure
* formatting the datetime component
* adding exception information (in case the log entry was emitted due to exception)
* mapping of the log level (Python) to custom level that allows it to be used in the model and log console
In addition the entry is inserted into a database, which can then be viewed by and processed with external tools
Args:
record: Log entry
Returns:
"""
# Format log entry
self.format(record)
self.format_time(record)
if record.exc_info: # for exceptions
record.exc_text = logging._defaultFormatter.formatException(record.exc_info)
else:
record.exc_text = ""
# Map Python logging module's levels to the custom ones
if record.levelno == 10:
record.levelno = LogEntriesModel.LoggingLevel.DEBUG # 3
elif record.levelno == 20:
record.levelno = LogEntriesModel.LoggingLevel.INFO # 0
elif record.levelno == 30:
record.levelno = LogEntriesModel.LoggingLevel.WARN # 1
elif record.levelno == 40:
record.levelno = LogEntriesModel.LoggingLevel.ERROR # 2
# Insert the log record
sql = LogHandlerSQLite.db_insert_script % record.__dict__
conn = sq3.connect(self.db)
conn.execute(sql)
conn.commit()
conn.close()
def rename_db_file(self, add_extension=False):
db_old_name = os.path.splitext(self.db)[0]
db_new_name = db_old_name + "_" \
+ datetime.datetime.now().replace(microsecond=0).isoformat().replace(":", "-").replace(" ", "_")
if add_extension:
db_new_name = db_new_name + os.path.splitext(self.db)[1]
return db_new_name
def close(self):
"""
Clean-up procedure called during shutdown of the logger that uses the handler
The previously created log database is renamed by adding the current system
time stamp. This is done to ensure that the database can be viewed later (e.g.
bug report) and not overwritten when the logging is setup again
Returns:
"""
db_new_name = self.rename_db_file(add_extension=True)
os.rename(self.db, db_new_name)
super().close()
class SQLiteLogger(logging.Logger):
def __init__(self, name="geointelkit", level=logging.DEBUG, db_path="logs.db"):
logging.Logger.__init__(self, name, level)
self.formatter = logging.Formatter(fmt="%(asctime)s %(levelno)d %(name)s %(message)s",
datefmt="%Y-%m-%dT%H:%M:%S")
self.handler = LogHandlerSQLite(db_path=db_path)
self.handler.setFormatter(self.formatter)
self.addHandler(self.handler)
def clear_entries(self):
self.handler.clear_entries()
def dump(self):
self.handler.dump()
def setup_logger():
logging.setLoggerClass(SQLiteLogger)
Last but not least changing the escapechar to a blank space " " resulted in double blank spaces in ALL messages even those that are just a simple short string. Here are some examples including the problematic log record from above:
2021-10-26T15:12:09.467067+02:00,3,geointelkit,Adding OpenStreetMaps view
2021-10-26T15:12:09.536681+02:00,0,geointelkit,Setting coordinates to [lat: 48.9475 , lng: 8.4106]

How to use Types and Methods with SUDS

My code connects to a Autotask service through SUDS. I get a list of Methods, one of which I have to call. I'm new to this, so I'm not sure on how to correctly do this. This is my code, including help by Bodsda:
import os, sys
import xml.etree.ElementTree as ET
from suds.client import Client
from suds.sax.element import Element
class Init():
def __init__(self):
#Search the app.config file for all data to be used
script_dir = os.path.dirname(__file__)
file_path = "app.config"
abs_file_path = os.path.join(script_dir, file_path)
tree = ET.parse(abs_file_path)
root = tree.getroot()
sites = root.iter('AutotaskUpdateTicketEstimatedHours.My.MySettings')
self.data = {}
for site in sites:
apps = site.findall('setting')
for app in apps:
self.data[app.get('name')] = app.find('value').text
class ConnectATWS():
def __init__(self):
#Connect to server with the credentials
app_config = Init()
self.username = app_config.data["Username"]
self.password = app_config.data["Password"]
self.login_id = app_config.data["LoginID"]
strQuery = """
<queryxml>
<entity>Ticket</entity>
<query>
<condition>
<field>Id
<expression op="GreaterThan">0</expression>
</field>
</condition>
<condition>
<field>Status
<expression op="NotEqual">5</expression>
</field>
</condition>
<condition>
<field>EstimatedHours
<expression op="IsNull"></expression>
</field>
</condition>
</query>
</queryxml>"""
new_url = 'https://webservices5.autotask.net/atservices/1.5/atws.wsdl'
client = Client(new_url, username=self.login_id, password=self.password)
response = client.service.query(strQuery)
if response.ReturnCode != 1:
print "Error code: %s" % response.ReturnCode
print "Error response: %s" % response.Errors
sys.exit(1)
else:
print "Query successful..."
print "============================="
print response.EntityResults
if __name__ == '__main__':
handler = ConnectATWS()
This is my response:
(ATWSResponse){
ReturnCode = 1
EntityResults = ""
EntityResultType = "ticket"
Errors = ""
EntityReturnInfoResults = ""
}
This is the list of Methods and Types I get through the client:
Suds ( https://fedorahosted.org/suds/ ) version: 0.4 GA build: R699-20100913
Service ( ATWS ) tns="http://autotask.net/ATWS/v1_5/"
Prefixes (1)
ns0 = "http://autotask.net/ATWS/v1_5/"
Ports (1):
(ATWSSoap)
Methods (14):
CreateAttachment(Attachment attachment, )
DeleteAttachment(xs:long attachmentId, )
GetAttachment(xs:long attachmentId, )
GetFieldInfo(xs:string psObjectType, )
GetInvoiceMarkup(xs:int InvoiceId, xs:string Format, )
GetWsdlVersion()
create(ArrayOfEntity Entities, )
delete(ArrayOfEntity Entities, )
getEntityInfo()
getThresholdAndUsageInfo()
getUDFInfo(xs:string psTable, )
getZoneInfo(xs:string UserName, )
query(xs:string sXML, )
update(ArrayOfEntity Entities, )
Types (108):
ATWSError
ATWSResponse
ATWSZoneInfo
Account
AccountLocation
AccountNote
AccountTeam
AccountToDo
ActionType
AdditionalInvoiceFieldValue
AllocationCode
Appointment
ArrayOfATWSError
ArrayOfEntity
ArrayOfEntityInfo
ArrayOfEntityReturnInfo
ArrayOfField
ArrayOfPickListValue
ArrayOfUserDefinedField
Attachment
AttachmentInfo
AutotaskIntegrations
BillingItem
BillingItemApprovalLevel
ChangeRequestLink
ClientPortalUser
Contact
Contract
ContractBlock
ContractCost
ContractFactor
ContractMilestone
ContractNote
ContractRate
ContractRetainer
ContractService
ContractServiceAdjustment
ContractServiceBundle
ContractServiceBundleAdjustment
ContractServiceBundleUnit
ContractServiceUnit
ContractTicketPurchase
Country
Department
Entity
EntityDuplicateStatus
EntityInfo
EntityReturnInfo
EntityReturnInfoDatabaseAction
ExpenseItem
ExpenseReport
Field
InstalledProduct
InstalledProductType
InstalledProductTypeUdfAssociation
InternalLocation
InventoryItem
InventoryItemSerialNumber
InventoryLocation
InventoryTransfer
Invoice
InvoiceTemplate
Opportunity
PaymentTerm
Phase
PickListValue
Product
ProductVendor
Project
ProjectCost
ProjectNote
PurchaseOrder
PurchaseOrderItem
PurchaseOrderReceive
Quote
QuoteItem
QuoteLocation
Resource
ResourceRole
ResourceSkill
Role
SalesOrder
Service
ServiceBundle
ServiceBundleService
ServiceCall
ServiceCallTask
ServiceCallTaskResource
ServiceCallTicket
ServiceCallTicketResource
ShippingType
Skill
Task
TaskNote
TaskPredecessor
TaskSecondaryResource
Tax
TaxCategory
TaxRegion
Ticket
TicketChangeRequestApproval
TicketCost
TicketNote
TicketSecondaryResource
TimeEntry
UserDefinedField
UserDefinedFieldDefinition
UserDefinedFieldListItem
I need to use the ATWSResponse type but I don't really understand how to do this. Furthermore, I'm attempting to simulate this vb.net code part which does what I'm trying to do in Python:
Sub ProcessTicket()
Dim boolQueryFinished = False
Dim strCurrentID As String = "0"
Dim strQuery As String
Dim strCriteria As String = ""
Dim TicketArray(0) As Ticket
While (Not (boolQueryFinished))
If LCase(Trim(varIgnoreTicketStatus)) = "true" Then
WriteToLog("Updating All Tickets Where EstimatedHours Is Null")
Else
WriteToLog("Updating Ticket Where Status <> Complete And EstimatedHours Is Null")
' 5 - Complete
strCriteria = "<condition><field>Status<expression op=""NotEqual"">5</expression></field></condition>"
End If
strQuery = "<queryxml><entity>Ticket</entity><query>" & _
"<condition><field>id<expression op=""greaterthan"">" & strCurrentID & "</expression></field></condition>" & strCriteria & _
"<condition><field>EstimatedHours<expression op=""isnull""></expression></field></condition>" & _
"</query></queryxml>"
Dim r As ATWSResponse
Dim strLog As String
r = ATWSService.query(strQuery)
WriteToLog("Found " & r.EntityResults.Length & " Tickets begining at " & strCurrentID)
If r.EntityResults.Length > 0 Then
For Each ent As Entity In r.EntityResults
CType(ent, Ticket).EstimatedHours = EstHoursDefault
TicketArray(0) = CType(ent, Ticket)
Dim sResponse As ATWSResponse
Dim entityArray() As Entity = CType(TicketArray, Entity())
sResponse = ATWSService.update(entityArray)
My question is how can I successfully query with the QueryXML with my Python code and return entities like in the vb.net code?

First of all, the url being used to connect to the api was not the correct one. I needed to use https://webservices5.autotask.net/atservices/1.5/atws.wsdl.
Once the client received a response, I had to match the ReturnCode in order to run the code I wanted. The return code could not be 1 (which is an error in the query), therefore I used:
if response.ReturnCode != 1:
print "Error code: %s" % response.ReturnCode
print "Error response: %s" % response.Errors
sys.exit(1)
else:
print "Query successful..."
print "============================="
response = response.EntityResults[0]
In order to store my returned data (entities), which were Tickets, I had to do the following loops:
entities = []
for entity in response:
entities.append(dict(entity))
for entity in entities:
self.query_data.append(entity["Title"] + " Estimated Hours " + str(entity["EstimatedHours"]))
This way I could store the list of tuples in entities list in order to then access each entity themselves. With this done, all I had to do was append the results to a list self.query_data.
Note: Credit to Bodsda for his help in figuring out how I was querying the wrong address with my XML, plus help with the loops.

I am not sure, but besides the answer made by cpburnz, i noticed that in the vb code ,which seems to be working, the python variable r should be created as ATWSResponse, like you did with client where in vb is an object called ATWSService.

Error message "MemoryError" in Python

Here's my problem: I'm trying to parse a big text file (about 15,000 KB) and write it to a MySQL database. I'm using Python 2.6, and the script parses about half the file and adds it to the database before freezing up. Sometimes it displays the text:
MemoryError.
Other times it simply freezes. I figured I could avoid this problem by using generator's wherever possible, but I was apparently wrong.
What am I doing wrong?
When I press Ctrl + C to keyboard interrupt, it shows this error message:
...
sucessfully added vote # 2281
sucessfully added vote # 2282
sucessfully added vote # 2283
sucessfully added vote # 2284
floorvotes_db.py:35: Warning: Data truncated for column 'vote_value' at row 1
r['bill ID'] , r['last name'], r['vote'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "floorvotes_db.py", line 67, in addAllFiles
addFile(file)
File "floorvotes_db.py", line 61, in addFile
add(record)
File "floorvotes_db.py", line 35, in add
r['bill ID'] , r['last name'], r['vote'])
File "build/bdist.linux-i686/egg/MySQLdb/cursors.py", line 166, in execute
File "build/bdist.linux-i686/egg/MySQLdb/connections.py", line 35, in defaulte rrorhandler
KeyboardInterrupt
import os, re, datetime, string
# Data
DIR = '/mydir'
tfn = r'C:\Documents and Settings\Owner\Desktop\data.txt'
rgxs = {
'bill number': {
'rgx': r'(A|S)[0-9]+-?[A-Za-z]* {50}'}
}
# Compile rgxs for speediness
for rgx in rgxs: rgxs[rgx]['rgx'] = re.compile(rgxs[rgx]['rgx'])
splitter = rgxs['bill number']['rgx']
# Guts
class floor_vote_file:
def __init__(self, fn):
self.iterdata = (str for str in
splitter.split(open(fn).read())
if str and str <> 'A' and str <> 'S')
def iterVotes(self):
for record in self.data:
if record: yield billvote(record)
class billvote(object):
def __init__(self, section):
self.data = [line.strip() for line
in section.splitlines()]
self.summary = self.data[1].split()
self.vtlines = self.data[2:]
self.date = self.date()
self.year = self.year()
self.votes = self.parse_votes()
self.record = self.record()
# Parse summary date
def date(self):
d = [int(str) for str in self.summary[0].split('/')]
return datetime.date(d[2],d[0],d[1]).toordinal()
def year(self):
return datetime.date.fromordinal(self.date).year
def session(self):
"""
arg: 2-digit year int
returns: 4-digit session
"""
def odd():
return divmod(self.year, 2)[1] == 1
if odd():
return str(string.zfill(self.year, 2)) + \
str(string.zfill(self.year + 1, 2))
else:
return str(string.zfill(self.year - 1, 2))+ \
str(string.zfill(self.year, 2))
def house(self):
if self.summary[2] == 'Assembly': return 1
if self.summary[2] == 'Senate' : return 2
def splt_v_line(self, line):
return [string for string in line.split(' ')
if string <> '']
def splt_v(self, line):
return line.split()
def prse_v(self, item):
"""takes split_vote item"""
return {
'vote' : unicode(item[0]),
'last name': unicode(' '.join(item[1:]))
}
# Parse votes - main
def parse_votes(self):
nested = [[self.prse_v(self.splt_v(vote))
for vote in self.splt_v_line(line)]
for line in self.vtlines]
flattened = []
for lst in nested:
for dct in lst:
flattened.append(dct)
return flattened
# Useful data objects
def record(self):
return {
'date' : unicode(self.date),
'year' : unicode(self.year),
'session' : unicode(self.session()),
'house' : unicode(self.house()),
'bill ID' : unicode(self.summary[1]),
'ayes' : unicode(self.summary[5]),
'nays' : unicode(self.summary[7]),
}
def iterRecords(self):
for vote in self.votes:
r = self.record.copy()
r['vote'] = vote['vote']
r['last name'] = vote['last name']
yield r
test = floor_vote_file(tfn)
import MySQLdb as dbapi2
import floorvotes_parse as v
import os
# Initial database crap
db = dbapi2.connect(db=r"db",
user="user",
passwd="XXXXX")
cur = db.cursor()
if db and cur: print "\nConnected to db.\n"
def commit(): db.commit()
def ext():
cur.close()
db.close()
print "\nConnection closed.\n"
# DATA
DIR = '/mydir'
files = [DIR+fn for fn in os.listdir(DIR)
if fn.startswith('fvote')]
# Add stuff
def add(r):
"""add a record"""
cur.execute(
u'''INSERT INTO ny_votes (vote_house, vote_date, vote_year, bill_id,
member_lastname, vote_value) VALUES
(%s , %s , %s ,
%s , %s , %s )''',
(r['house'] , r['date'] , r['year'],
r['bill ID'] , r['last name'], r['vote'])
)
#print "added", r['year'], r['bill ID']
def crt():
"""create table"""
SQL = """
CREATE TABLE ny_votes (openleg_id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
vote_house int(1), vote_date int(5), vote_year int(2), bill_id varchar(8),
member_lastname varchar(50), vote_value varchar(10));
"""
cur.execute(SQL)
print "\nCreate ny_votes.\n"
def rst():
SQL = """DROP TABLE ny_votes"""
cur.execute(SQL)
print "\nDropped ny_votes.\n"
crt()
def addFile(fn):
"""parse and add all records in a file"""
n = 0
for votes in v.floor_vote_file(fn).iterVotes():
for record in votes.iterRecords():
add(record)
n += 1
print 'sucessfully added vote # ' + str(n)
def addAllFiles():
for file in files:
addFile(file)
if __name__=='__main__':
rst()
addAllFiles()

Generators are a good idea, but you seem to miss the biggest problem:
(str for str in splitter.split(open(fn).read()) if str and str <> 'A' and str <> 'S')
You're reading the whole file in at once even if you only need to work with bits at a time. You're code is too complicated for me to fix, but you should be able to use file's iterator for your task:
(line for line in open(fn))

I noticed that you use a lot of slit() calls. This is memory consuming, according to http://mail.python.org/pipermail/python-bugs-list/2006-January/031571.html . You can start investigating this.

Try to comment out add(record) to see if the problem is in your code or on the database side. All the records are added in one transaction (if supported) and maybe this leads to a problem if it get too many records. If commenting out add(record) helps, you could try to call commit() from time to time.

This isn't a Python memory issue, but perhaps it's worth thinking about. The previous answers make me think you'll sort that issue out quickly.
I wonder about the rollback logs in MySQL. If a single transaction is too large, perhaps you can checkpoint chunks. Commit each chunk separately instead of trying to rollback a 15MB file's worth.