We Keep Coding

Python nested loop not working as intended

I'm working on an assignment for school where I have a text file: data.txt which looks like this:(instead of 'name' there are actual names I just replaced them here)
10001-31021 'name' 2015.12.30. 524432
10001-31121 'name' 2016.03.21. 765432
10012-34321 'name' 2016.02.20. 231231
10201-11021 'name' 2016.01.10. 2310456
And I have an update.txt which looks like this:
2016.03.22.
10001-31021 'name' +20000
10012-34321 'name' +35432
10012-34321 'name' -10000
10120-00123 'name' +120334
10001-31021 'name' +5000
10210-41011 'name' -6000
10201-11021 'name' +100210
12345-32100 'name' +123456
And I have to make a newdata.txt file according to the changes to the last column that update.txt includes.
This is my code so far:
adat = open("data.txt", "r")
newdata = open("newdata.txt", "w")
update = open("update", "r")
date = update.readline().decode("utf-8-sig").encode("utf-8").splitlines()
num_lines = sum(1 for line in open('update'))
elsociklus = 0
masodikciklus = 0
for num_lines in update:
updateData = re.search("(.{11}\t)(\D+\t)([+-]\d+)", num_lines)
elsociklus = elsociklus + 1
print("elsociklus: " + str(elsociklus))
for j in adat:
data = re.search("(.{11}\t)(\D+\t)(\d{4}\.\d{2}\.\d{2}\.\t)(\d+)", j)
masodikciklus = masodikciklus + 1
print("masodikciklus: " + str(masodikciklus))
if data != None:
if updateData.group(1) == data.group(1):
print("regi: " + data.group(0))
print("update: " + updateData.group(0))
print("uj: " + data.group(1) + data.group(2) + date[0] + "\t" + str(int(data.group(4)) + int(updateData.group(3))))
newdata.write(data.group(1) + data.group(2) + date[0] + "\t" + str(int(data.group(4)) + int(updateData.group(3))))
newdata.write("\n")
else:
print("nincs valtozas: " + data.group(0))
adat.close()
newdata.close()
update.close()
My problem is with the nested loop. I just can't figure it out why it isn't entering the inner loop for the second time. It works perfectly on the first iteration but when entering the 2nd one in the outer loop it just ignores the inner loop.
Thank you in advance for your help.

Thanks to codingCat for the answer. I fixed the problem by returning my file pointer to the beginning of my file in the inner loop

What can I do for my program not to throw a KeyError for a Header not existing in a .fits file?

I am making a program that searches the computer for .fts and .fits files in which it opens the file and retrieves info that corresponds to a specific keyword in the header and renames the file to that keyword.
I am having a problem where i keep receiving a KeyError becasue a header keyword I am searching for is not found in the file. Is there a way around this? I want to be able to search various keywords and do something even if that key word does not exist in the file.
Here is code:
from astropy.io import fits
import os
for i in os.listdir(os.getcwd()):
if i.endswith(".fits") or i.endswith(".fts"):
hdulist = fits.open(i)
DATEOBS_header = hdulist[0].header['DATE-OBS']
EXPTIME_header = int(round(hdulist[0].header['EXPTIME']))
CCDTEMP_header = int(round(hdulist[0].header['CCD-TEMP']))
XBINNING_header = hdulist[0].header['XBINNING']
FILTER_header = hdulist[0].header['FILTER']
IMAGETYP_header = hdulist[0].header['IMAGETYP']
OBJECT_header = hdulist[0].header['OBJECT']
DATEandTIME = DATEOBS_header[0:]
YEAR = DATEandTIME[0:4]
MONTH = DATEandTIME[5:7]
DAY = DATEandTIME[8:10]
#TIME = DATEOBS_header[11:]
HOUR = DATEandTIME[11:13]
MINUTE = DATEandTIME[14:16]
SECONDS = DATEandTIME[17:]
DATE = str(YEAR) + str(MONTH) + str(DAY) + 'at' + str(HOUR) + str(MINUTE) + str(SECONDS)
if IMAGETYP_header == 'Light Frame':
newname = str(OBJECT_header) + '_' + str(DATE) + '_' + str(CCDTEMP_header) + 'temp_' + str(XBINNING_header) + 'bin_' + str(EXPTIME_header) + 'exptime_' + str(FILTER_header) + '.fits'
if IMAGETYP_header == 'Dark Frame':
newname = 'Dark_' + str(DATE) + 'at' + str(TIME) + '_' + str(CCDTEMP_header) + 'temp_' + str(XBINNING_header) + 'bin_' + str(EXPTIME_header) + 'exptime' + '.fits'
if IMAGETYP_header == 'Flat Field':
newname = 'Flat_' + str(DATE) + 'at' + str(TIME) + '_' + str(CCDTEMP_header) + 'temp_' + str(XBINNING_header) + 'bin_' + str(EXPTIME_header) + 'exptime_' + str(FILTER_header) + '.fits'
prevname = i
os.rename(prevname, newname)
hdulist.close()
continue
else:
continue
This is the Error I get:
Traceback (most recent call last):
File "glo1.py", line 9, in <module>
DATEOBS_header = hdulist[0].header['DATE-OBS']
File "/home/luisgeesb/.local/lib/python2.7/site-packages/astropy/io/fits/header.py", line 151, in __getitem__
card = self._cards[self._cardindex(key)]
File "/home/luisgeesb/.local/lib/python2.7/site-packages/astropy/io/fits/header.py", line 1723, in _cardindex
raise KeyError("Keyword %r not found." % keyword)
KeyError: "Keyword 'DATE-OBS' not found."

To prevent these kinds of exceptions from stopping your program, you can either catch them, like this:
try:
DATEOBS_header = hdulist[0].header['DATE-OBS']
except KeyError:
DATEOBS_header = None
Or, use the .get() method of dictionaries, which checks if a key exists and if it doesn't returns a default value, instead of raising an exception. The default value returned is None.
If you do this, you will also need to set some sensible defaults, or catch those cases where you are casting the values (since you cannot cast None).
Finally, whenever you are reading from files - you should always assume the data is malformed/junk and do a bit of defensive programming. In your code, you are assuming that the values returned for CCDTEMP is a number, but what if the file is corrupted or has a blank? Your application doesn't handle this case.
Here is some code that attempts to catch as many errors as possible:
DATEOBS_header = hdulist[0].header.get('DATE-OBS')
XBINNING_header = hdulist[0].header.get('XBINNING')
FILTER_header = hdulist[0].header.get('FILTER')
IMAGETYP_header = hdulist[0].header.get('IMAGETYP')
OBJECT_header = hdulist[0].header.get('OBJECT')
# For these two, you need to either set a default
# Here I am setting the default to 0, ------------v
EXPTIME_header = int(round(hdulist[0].header.get('EXPTIME', 0)))
# Or you need to check for a value :
ccdtemp_value = hdulist[0].header.get('CCD-TEMP')
try:
ccdtemp_value = int(round(ccdtemp_value))
except ValueError:
# This means, the value was either None (header does not exist)
# or it was something that can't be converted to a number
# since it cannot be converted to a number, we do not know
# if the value is None or something like an empty string,
# so we explicitly set the value to None
ccdtemp_value = None
CCDTEMP_header = ccdtemp_value

Assuming hdulist[0].header gives you a dict instance, you can do something like
DATEOBS_header = hdulist[0].header.get('DATE-OBS')
Which would return a None if the key 'DATE-OBS' does not exist.
See https://docs.python.org/2/library/stdtypes.html#dict.get for more details.

python wont write even when I use f.close

I'm trying to write some code that outputs some text to a list. output is a variable that is a string which is the name of the file to be written. However whenever I look at the file nothing is written.
with open(output, 'w') as f:
f.write("Negative numbers mean the empty space was moved to the left and positive numbers means it was moved to the right" + '\n')
if A == True:
the_h = node.h
elif A== False:
the_h = 0
f.write("Start " + str(node.cargo) + " " + str(node.f) +" " +str(the_h)+" " + '\n')
if flag == 0:
flag = len(final_solution)
for i in range (1,flag):
node = final_solution[i]
f.write(str(node.e_point - node.parent.e_point) + str(node.cargo) + " " + str(node.f) +'\n')
f.close()

Program looks ok, check if the output is set ok, I set as a dummy filename, it worked, presuming code within the block after open has no compiler/interpreter error. The output file should be in the same directory where the source is.
output = "aa.txt"
with open(output, 'w') as f:
f.write("Negative numbers mean the empty space was moved to the left and positive numbers means it was moved to the right" + '\n')
if A == True:
the_h = node.h
elif A== False:
the_h = 0
f.write("Start " + str(node.cargo) + " " + str(node.f) +" " +str(the_h)+" " + '\n')
if flag == 0:
flag = len(final_solution)
for i in range (1,flag):
node = final_solution[i]
f.write(str(node.e_point - node.parent.e_point) + str(node.cargo) + " " + str(node.f) +'\n')
f.close()

You should not add f.close(), as the with statement will do it for you. Also ensure you don't reopen the file elsewhere with open(output, 'w') as that will erase the file.

Python Nested Loops - continue iterates first loop

Brand new to programming but very enjoyable challenge.
Here's a question which I suspect may be caused by a misunderstanding of python loops.
System info: Using notepad++ and IDLE python 3.4.3 on Win 7 32-bit
My solution is to open 1 database, use it to look for a correct master entry from database 2, pulls a index number (task_no), then write a 3rd file identical to the first database, this time with the correct index number.
My problem is that it performs 1st and 2nd loop correctly, then on the 2nd iteration of loop 1, tries to perform a block in loop 2 while iterating through the rows of loop 1, not the task_rows of loop 2.
footnote: Both files are quite large (several MB) so I'm note sure if storing them in memory is a good idea.
This was a relevant question that I found closest to this problem:
python nested loop using loops and files
What I got out of it was that I tried moving the file opening within the 1st loop, but the problem persists. Something to do with how I'm using CSV reader?
I also have the sinking suspicion that there may be a root cause in problem solving so I am welcome to suggestions for alternative ways to solve the problem.
Thanks in advance!
The gist:
for row in readerCurrentFile: #LOOP 1
# iterates through readerCurrentFile to define search variables
[...]
for task_row in readerTaskHeader: #LOOP 2
# searches each row iteratively through readerTaskHeader
# Match compid
#if no match, continue <<<- This is where it goes back to 1st loop
[...]
# Match task frequency
#if no match, continue
[...]
# once both of the above matches check out, will grab data (task_no from task_row[0]
task_no = ""
task_no = task_row[0]
if task_row:
break
[...]
# writes PM code
print("Successful write of PM schedule row")
print(compid + " " + dict_freq_names[str(pmfreqx) + str(pmfreq)] + ": " + pmid + " " + task_no)
The entire code:
import csv
import re
#Writes schedule
csvNewPMSchedule = open('new_pm_schedule.csv', 'a', newline='')
writerNewPMSchedule = csv.writer(csvNewPMSchedule)
# Dictionaries of PM Frequency
def re_compile_dict(d,f):
for k in d:
d[k] = re.compile(d[k], flags=f)
dict_month = {60:'Quin',36:'Trien',24:'Bi-An',12:'Annual(?<!Bi-)(?<!Semi-)',6:'Semi-An',3:'Quart',2:'Bi-Month',1:'Month(?<!Bi-)'}
dict_week = {2:'Bi-Week',1:'Week(?<!Bi-)'}
dict_freq_names = {'60Months':'Quintennial','36Months':'Triennial','24Months':'Bi-Annual','12Months':'Annual','6Months':'Semi-Annual','3Months':'Quarterly','2Months':'Bi-Monthly','1Months':'Monthly','2Weeks':'Bi-Weekly','1Weeks':'Weekly'}
re_compile_dict(dict_month,re.IGNORECASE)
re_compile_dict(dict_week, re.IGNORECASE)
# Unique Task Counter
task_num = 0
total_lines = 0
#Error catcher
error_in_row = []
#Blank out all rows
pmid = 0
compid = 0
comp_desc = 0
pmfreqx = 0
pmfreq = 0
pmfreqtype = 0
# PM Schedule Draft (as provided by eMaint)
currentFile = open('pm_schedule.csv', encoding='windows-1252')
readerCurrentFile = csv.reader(currentFile)
# Loop 1
for row in readerCurrentFile:
if row[0] == "pmid":
continue
#defines row items
pmid = row[0]
compid = row[1]
comp_desc = row[2]
#quantity of pm frequency
pmfreqx_temp = row[3]
#unit of pm frequency, choices are: Months, Weeks
pmfreq = row[4]
#pmfreqtype is currently only static not sure what other options we have
pmfreqtype = row[5]
#pmnextdate is the next scheduled due date from this one. we probably need logic later that closes out any past due date
pmnextdate = row[6]
# Task Number This is what we want to change
# pass
# We want to change this to task header's task_desc
sched_task_desc = row[8]
#last done date
last_pm_date = row[9]
#
#determines frequency search criteria
#
try:
pmfreqx = int(pmfreqx_temp)
except (TypeError, ValueError):
print("Invalid PM frequency data, Skipping row " + pmid)
error_in_row.append(pmid)
continue
#
#defines frequency search variable
#
freq_search_var = ""
if pmfreq == "Weeks":
freq_search_var = dict_week[pmfreqx]
elif pmfreq == "Months":
freq_search_var = dict_month[pmfreqx]
if not freq_search_var:
print("Error in assigning frequency" + compid + " " + str(pmfreqx) + " " + pmfreq)
error_in_row.append(pmid)
continue
#defines Equipment ID Search Variable
print(compid + " frequency found: " + str(pmfreqx) + " " + str(pmfreq))
compid_search_var = re.compile(compid,re.IGNORECASE)
#
# Matching function - search taskHeader for data
#
#PM Task Header Reference
taskHeader = open('taskheader.csv', encoding='windows-1252')
readerTaskHeader = csv.reader(taskHeader)
for task_row in readerTaskHeader:
# task_row[0]: taskHeader pm number
# task_row[1]: "taskHeader task_desc
# task_row[2]: taskHeader_task_notes
#
# search for compid
compid_match = ""
compid_match = compid_search_var.search(task_row[1])
if not compid_match:
print(task_row[1] + " does not match ID for " + compid + ", trying next row.") #debug 2
continue # <<< STOPS ITERATING RIGHT OVER HERE
print("Found compid " + task_row[1]) # debug line
#
freq_match = ""
freq_match = freq_search_var.search(task_row[1])
if not freq_match:
print(task_row[1] + " does not match freq for " + compid + " " + dict_freq_names[str(pmfreqx) + str(pmfreq)] + ", trying next row.") #debug line
continue
print("Frequency Match: " + compid + " " + dict_freq_names[str(pmfreqx) + str(pmfreq)]) # freq debug line
#
task_no = ""
print("Assigning Task Number to " + task_row[0])
task_no = task_row[0]
if task_row:
break
#
#error check
#
if not task_no:
print("ERROR IN SEARCH " + compid + " " + pmid)
error_in_row.append(pmid)
continue
#
# Writes Rows
#
writerNewPMSchedule.writerow([pmid,compid,comp_desc,pmfreqx,pmfreq,pmfreqtype,pmnextdate,task_no,sched_task_desc,last_pm_date])
print("Successful write of PM schedule row")
print(compid + " " + dict_freq_names[str(pmfreqx) + str(pmfreq)] + ": " + pmid + " " + task_no)
print("==============")
# Error reporting lined out for now
# for row in error_in_row:
# writerNewPMSchedule.writerow(["Error in row:",str(error_in_row[row])])
# print("Error in row: " + str(error_in_row[row]))
print("Finished")

Python's csv reader keep reading the same file

i'm having a problem with the python's csv reader. The problem is that i want to open and read different csv files, but he keeps on reading always the same one.
from csv import reader
alphabet = ["a", "b", "c"]
for letter in alphabet:
csv_file = open('/home/desktop/csv/' + letter + '.csv', 'r')
csv_data = reader(csv_file)
The problem is that he seems to open the other files, but he keep on reading always the first file.
Is there a way to "clean" the reader or make him read another file? I even tried to close the csv file, but it didn't work.
The full code is this
from csv import reader
#Orario
orario_csv_file = '/home/andrea/Scrivania/orario.csv'
orario_csv = open(orario_csv_file)
orario_data = reader(orario_csv)
orario = []
#Corsi
corsi = ["EDILIZIA", "EDILE-ARCHIT", "ELETTRONICA", "TECNOLOGIE_DI_INTERNET", "INFORMATICA", "GESTIONALE", "ENERGETICA", "MECCANICA", "CIVILE_ED_AMBIENTALE", "MEDICA", "ENGINEERING_SCIENCES"]
giorni = ["Lun", "Mar", "Mer", "Gio", "Ven"]
for row in orario_data:
orario.append(row)
for corso in corsi:
nome_corso_file = '/home/andrea/Scrivania/xml/' + corso + '.xml'
nome_corso_xml = open(nome_corso_file, 'wt')
nome_corso_xml.write('<?xml version="1.0"?>' + "\n")
nome_corso_xml.write('<orario>' + "\n")
nome_csv = corso + '_csv'
nome_csv = '/home/andrea/Scrivania/csv/' + corso + '.csv'
nome_corso_csv = open(nome_csv, 'rt')
corso_data = reader(nome_corso_csv)
nome_corso_xml.write(' <corso name="' + corso + '">' + "\n")
for a in range(0, 3):
nome_corso_xml.write(' <anno num="' + str(a+1) + '">' + "\n")
for j in range(1, 6):
nome_corso_xml.write(' <giorno name="' + orario[2][j] + '">' + "\n")
for i in range(3, 12):
lez = orario[i + a*12][j]
if lez == "":
nome_corso_xml.write(' <lezione>' + "-" + '</lezione>' + "\n")
else:
for riga in corso_data:
if riga[0] == lez:
if riga[2] == "":
nome_corso_xml.write(' <lezione name="' + lez + '">' + riga[1] + '</lezione>' + "\n")
else:
for g in range(0, len(riga)):
if riga[g].lower() == orario[2][j].lower():
nome_corso_xml.write(' <lezione name="' + lez + '">' + riga[g+1] + '</lezione>' + "\n")
nome_corso_csv.seek(0)
nome_corso_xml.write(' </giorno>' + "\n")
nome_corso_xml.write(' </anno>' + "\n")
nome_corso_xml.write(' </corso>' + "\n")
nome_corso_xml.write('</orario>' + "\n")
nome_corso_xml.close()
He open the "EDILIZIA.csv" and compile the "EDILIZIA.xml", then he should open the "EDILE-ARCHIT.csv" and compile its xml, but when he read, he keeps on reading from "EDILIZIA.csv"
Here's the .csv files that you need.
http://pastebin.com/kJhL8HpK
If you try to make it read first EDILIZIA.csv and then EDILE-ARCHIT.csv he'll keep on using always the EDILIZIA.csv to compile the xml, but he should firt open EDILIZIA.csv, compile the EDILIZIA.xml, then read the EDILE-ARCHIT.csv and compile the EDILE-ARCHIT.xml.
If you take a look at the final xmls, you'll see that the EDILE-ARCHIT.xml will only display the common subjects of EDILIZIA.csv and EDILE-ARCHIT.csv

It took a long time to figure out what you are doing here. To tell the truth your code is a mess - there are many unused variables and lines that make no sense at all. Anyway, your code reads the appropriate csv file each time, thus the error is not where you thought it was.
If I am right, orario.csv contains the timetable of each course (stored in corsi list) for three semesters or years, and the corso.csv files contain the room where subjects are held. So you want to merge the information into an XML file.
You only forgot one thing: to proceed in orario.csv. Your code wants to merge the very first three anno with the current corso. To fix it, you have to make two changes.
First in this for loop header:
for corso in corsi:
Modify to:
for num, corso in enumerate(corsi):
And when you assign lez:
lez = orario[i + a*12][j]
Modify to:
lez = orario[i + a*12*(num+1)][j]
Now it should work.
This code produces exactly the same result, but it uses Python's XML module to build the output file:
from csv import reader
import xml.etree.cElementTree as ET
import xml.dom.minidom as DOM
corsi = ["EDILIZIA", "EDILE-ARCHIT", "ELETTRONICA", "TECNOLOGIE_DI_INTERNET", "INFORMATICA", "GESTIONALE", "ENERGETICA", "MECCANICA", "CIVILE_ED_AMBIENTALE", "MEDICA", "ENGINEERING_SCIENCES"]
with open('orario.csv', 'r') as orario_csv:
orario = reader(orario_csv)
orario_data = [ row for row in orario ]
for num, corso in enumerate(corsi):
with open(corso + '.csv', 'r') as corso_csv:
corso_raw = reader(corso_csv)
corso_data = [ row for row in corso_raw ]
root_elem = ET.Element('orario')
corso_elem = ET.SubElement(root_elem, 'corso')
corso_elem.set('name', corso)
for anno in range(0, 3):
anno_elem = ET.SubElement(corso_elem, 'anno')
anno_elem.set('num', str(anno + 1))
for giorno in range(1, 6):
giorno_elem = ET.SubElement(anno_elem, 'giorno')
giorno_elem.set('name', orario_data[2][giorno])
for lezione in range(3, 12):
lez = orario_data[lezione + anno * 12 * (num + 1)][giorno]
if lez == '':
lezione_elem = ET.SubElement(giorno_elem, 'lezione')
lezione_elem.text = '-'
else:
for riga in corso_data:
if riga[0] == lez:
if riga[2] == '':
lezione_elem = ET.SubElement(giorno_elem, 'lezione')
lezione_elem.set('name', lez)
lezione_elem.text = riga[1]
else:
for g in range(0, len(riga)):
if riga[g].lower() == orario_data[2][giorno].lower():
lezione_elem = ET.SubElement(giorno_elem, 'lezione')
lezione_elem.set('name', lez)
lezione_elem.text = riga[g + 1]
with open(corso + '_new.xml', 'w') as corso_xml:
xml_data = DOM.parseString(ET.tostring(root_elem, method = 'xml')).toprettyxml(indent = ' ')
corso_xml.write(xml_data)
Cheers.

I think I may have spotted the cause of your problem.
The second item in your corsi list ends with a full stop. This means that you will be looking for the file "EDILE-ARCHIT..csv", which is almost does not exist. When you try and open the file, the open() call will throw an exception, and your program will terminate.
Try removing the trailing full stop, and running it again.