I've done some research and from what I can tell this normally happens when a file is closed before it's finished being used?
But that doesn't make sense for what is happening here.
Here is my code:
import csv
dicto = {}
name = ""
with open(input("enter filepath here: "), "r") as mainfile:
reader = csv.reader(mainfile)
for row in reader:
name = row[8].lstrip("'")
name = name.lstrip("\xa0")
name1 = name
name = name.upper()
if not name[:3] in dicto:
dicto[name[:3]] = [name[:3]+".js", 0]
with open(dicto[name[:3]][0], "w") as file1: #here is the problem line
file1.write("tags=[")
else:
dicto[name[:3]][1] += 1
if name[:1] == "#":
print(name)
with open(dicto[name[:3]][0], "a") as file2:
if dicto[name[:3]][1]>0:
file2.write('various spam')
else:
file2.write('various eggs')
for key in dicto.keys():
with open(dicto[key][0], "a") as file3:
file3.write("\n];")
I'm running through a large database and splitting it up into JS files which are named after the first three letters of the data's label. It seems to run ok at first (there are 44k entries to go through so it takes a few seconds to finish). Overall I currently have 309 files generated, although none are complete. However Once it gets to the combo "CON" an error occurs:
Traceback (most recent call last):
File "C:\Users\SarbickiN\Documents\Codes\Python\schools\schools.py", line 16, in <module>
with open(dicto[name[:3]][0], "w") as file1:
OSError: [Errno 9] Bad file descriptor: 'CON.js'
and this shuts down the programme. Is there any reason why this would have happened? I've made a comment next to the line causing the issue.
Edit: solution (or lack thereof)
CON is a reserved name for files in windows along with a few others so would need to be replaced with something else. Check here for more details.
Self answering
CON is a reserved name for files in windows along with a few others so would need to be replaced with something else. Check here for more details.
Related
Kida lost here. I am trying to get a consolidated csv and I keep getting this error:
File "consolidate.py", line 26, in csv_merge.write(line)
ValueError: I/O operation on closed file.
I tried moving indentation:
csv_header='name,location,age,phonenumber'
csv_out = 'consolidated.csv'
csv_d
dir_tree = os.walk(csv_dir)
for dirpath, dirnames, filenames in dir_tree:
pass
csv_list = []
for file in filenames:
if file.endswith('.csv'):
csv_list.append(file)
csv_merge = open(csv_out, 'w')
csv_merge.write(csv_header)
csv_merge.write('\n')ir = os.getcwd()
for file in csv_list:
csv_in = open(file)
for line in csv_in:
if line.startswith(csv_header):
continue
csv_merge.write(line)
csv_in.close()
csv_merge.close()
print('Verify consolidated CSV file : ' + csv_out)
But this didn't work. How can I resolve this error?
You never open csv_merge properly and even if you do you still close csv_merge after the first item in csv_list is written.
Why are you even using things like csv_merge.close()?
The convention is to use things like with open(csv_merge, 'w') as f:, that way the file always gets closed, even if the loop or script fails to execute properly.
After looking for a large amount of time, I still can't seem to find an answer to my problem (I am new to python).
Here is what I'm trying to do :
Prompt the user to insert an nlps version and ncs version (both are just server builds)
Get all the filenames ending in .properties in a specified folder
Read those files to find the old nlps and ncs version
Replace, in the same .properties files, the old nlps and ncs versions by the ones given by the user
Here is my code so far :
import glob, os
import fileinput
nlpsversion = str(input("NLPS Version : "))
ncsversion = str(input("NCS Version : "))
directory = "C:/Users/x/Documents/Python_Test"
def getfilenames():
filenames = []
os.chdir(directory)
for file in glob.glob("*.properties"):
filenames.append(file)
return filenames
properties_files = getfilenames()
def replaceversions():
nlpskeyword = "NlpsVersion"
ncskeyword = "NcsVersion"
for i in properties_files:
searchfile = open(i, "r")
for line in searchfile:
if line.startswith(nlpskeyword):
old_nlpsversion = str(line.split("=")[1])
if line.startswith(ncskeyword):
old_ncsversion = str(line.split("=")[1])
for line in fileinput.FileInput(i,inplace=1):
print(line.replace(old_nlpsversion, nlpsVersion))
replaceversions()
In the .properties files, the versions would be written like :
NlpsVersion=6.3.107.3
NcsVersion=6.4.000.29
I am able to get old_nlpsversion and old_ncsversion to be 6.3.107.3 and 6.4.000.29. The problem occurs when I try to replace the old versions with the ones the user inputed. I get the following error :
C:\Users\X\Documents\Python_Test>python replace.py
NLPS Version : 15
NCS Version : 16
Traceback (most recent call last):
File "replace.py", line 43, in <module>
replaceversions()
File "replace.py", line 35, in replaceversions
for line in fileinput.FileInput(i,inplace=1):
File "C:\Users\X\AppData\Local\Programs\Python\Python36-
32\lib\fileinput.py", line 250, in __next__
line = self._readline()
File "C:\Users\X\AppData\Local\Programs\Python\Python36-
32\lib\fileinput.py", line 337, in _readline
os.rename(self._filename, self._backupfilename)
PermissionError: [WinError 32] The process cannot access the file because it
is being used by another process: 'test.properties' -> 'test.properties.bak'
It may be that my own process is the one using the file, but I can't figure out how to replace, in the same file, the versions without error. I've tried figuring it out myself, there are a lot of threads/resources out there on replacing strings in files, and i tried all of them but none of them really worked for me (as I said, I'm new to Python so excuse my lack of knowledge).
Any suggestions/help is very welcome,
You are not releasing the file. You open it readonly and then attempt to write to it while it is still open. A better construct is to use the with statement. And you are playing fast and loose with your variable scope. Also watch your case with variable names. Fileinput maybe a bit of overkill for what you are trying to do.
import glob, os
import fileinput
def getfilenames(directory):
filenames = []
os.chdir(directory)
for file in glob.glob("*.properties"):
filenames.append(file)
return filenames
def replaceversions(properties_files,nlpsversion,ncsversion):
nlpskeyword = "NlpsVersion"
ncskeyword = "NcsVersion"
for i in properties_files:
with open(i, "r") as searchfile:
lines = []
for line in searchfile: #read everyline
if line.startswith(nlpskeyword): #update the nlpsversion
old_nlpsversion = str(line.split("=")[1].strip())
line = line.replace(old_nlpsversion, nlpsversion)
if line.startswith(ncskeyword): #update the ncsversion
old_ncsversion = str(line.split("=")[1].strip())
line = line.replace(old_ncsversion, ncsversion)
lines.append(line) #store changed and unchanged lines
#At the end of the with loop, python closes the file
#Now write the modified information back to the file.
with open(i, "w") as outfile: #file opened for writing
for line in lines:
outfile.write(line+"\n")
#At the end of the with loop, python closes the file
if __name__ == '__main__':
nlpsversion = str(input("NLPS Version : "))
ncsversion = str(input("NCS Version : "))
directory = "C:/Users/x/Documents/Python_Test"
properties_files = getfilenames(directory)
replaceversions(properties_files,nlpsversion,ncsversion)
I wrote a code which query a mongo database and write results in a file.
My code create the file and start to write in it succesfully. But after multiple iterations (not sure if the number of iteration is fix or not) I got a PermissionError.
I've search about it but I only found answers about people who got the error at first attempt because they don't have permission. I will precise that I am not doing anything on my computer during the execution so I really don't understand how it can happen.
Here is parts of the code:
def query(self, query_date_part, query_actKey_part, filepath):
empty = True
print("0.0 %")
for i in range(len(query_date_part)):
query = {"dt": query_date_part[i], "actKey": query_actKey_part}
cursor = self.collection.find(query)
while cursor.alive:
try:
if empty:
with open(filepath, 'w') as fp:
json.dump(cursor.next(), fp, default=json_util.default)
empty = False
else:
append_to_json(filepath, cursor.next())
except StopIteration:
print("Stop Iteration")
print(str(round(float(i+1) / len(query_date_part) * 100, ndigits=2)) + " %")
return 0
def append_to_json(filepath, data):
"""
Append data in JSON format to the end of a JSON file.
NOTE: Assumes file contains a JSON object (like a Python dict) ending in '}'.
:param filepath: path to file
:param data: dict to append
"""
# construct JSON fragment as new file ending
new_ending = ", " + json.dumps(data, default=json_util.default)[1:-1] + "}\n"
# edit the file in situ - first open it in read/write mode
with open(filepath, 'r+') as f:
f.seek(0, 2) # move to end of file
index = f.tell() # find index of last byte
# walking back from the end of file, find the index
# of the original JSON's closing '}'
while not f.read().startswith('}'):
index -= 1
if index == 0:
raise ValueError("can't find JSON object in {!r}".format(filepath))
f.seek(index)
# starting at the original ending } position, write out
# the new ending
f.seek(index)
f.write(new_ending)`
Part of the output:
6.75 %
Stop Iteration
6.76 %
Traceback (most recent call last):
File "C:/Users/username/PycharmProjects/mongodbtk/mquerytk.py", line 237, in <module>
mdbc.query(split_date(2017,5,6,1,0,2017,5,16,10,0,step=2), {"$in": ["aFeature"]}, 'test.json')
File "C:/Users/username/PycharmProjects/mongodbtk/mquerytk.py", line 141, in query
append_to_json(filepath, cursor.next())
File "C:/Users/username/PycharmProjects/mongodbtk/mquerytk.py", line 212, in append_to_json
with open(filepath, 'r+') as f:
PermissionError: [Errno 13] Permission denied: 'test.json'
Process finished with exit code 1
Note: The size of the file increase during the execution. When it crash it is about 300 Mo, I still have a lot of space on my hard drive but maybe the size of the file can be an issue ?
Config: I use Windows 7, Python 3.6 and my IDE is PyCharm Community Edition 2016.3.2
I had the same issue, and after testing it out, it seems like there might be some "bug" when trying to write to the same file too "often", multiple times in each sec. I'll provide a very small code snippet what you can test with:
import csv
text = "dfkjghdfkljghflkjghjkdfdfgsktjgrhsleiuthsl uirghuircbl iawehcg uygbc sgygerh"
FIELD_NAMES = ['asd', 'qwe']
with open('test.txt', 'w', newline='') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=FIELD_NAMES)
writer.writeheader()
max = 10000
i = 0
while i <= max:
print(str(i))
with open('test.txt', 'a', newline='') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=FIELD_NAMES)
rowData = {'asd': text, 'qwe': text}
writer.writerow(rowData)
i += 1
The code I think is pretty self explanatory. I got the error very randomly, sometimes it happens after the ~75th iteration, sometimes it can get to even ~750, but it looks like the code can't reach the limit. So I recommend you to try to write more data a few times rather than few data very often. I hope it helps.
Try this
except StopIteration:
if not fp.closed: fp.close()
print("Stop Iteration")
It appears to be a Race Condition. Creating a helper function which does same task as a function raising a race condition, worked out for me.
For example, if a code block or function A() is creating a race condition, then we can handle this exception by calling again another helper function A_helper() which also does the same thing.
A python code example:
try:
A()
except PermissionError:
# Do something
else:
A_helper()
Your file is located somewhere where your program can't access it. Try moving it to a different directory. Or it might be you are entering the wrong filepath. I hope this Works for you!
What I would like the final code to execute is read a string of names in a text document named, 'names.txt'. Then tell the program to calculate how many names there are in that file and display the amount of names. The code I have so far was meant to display the sum of the numbers in a text file, but it was close enough to the program I need now that I think I may be able to rework it to gather the amount of strings/names and display that instead of the sum.
Here is the code so far:
def main():
#initialize an accumulator.
total = 0.0
try:
# Open the file.
myfile = open('names.txt', 'r')
# Read and display the file's contents.
for line in myfile:
amount = float(line)
total += amount
# Close the file.
myfile.close()
except IOError:
print('An error occured trying to read the file.')
except ValueError:
print('Non-numeric data found in the file.')
except:
print('An error occured.')
# Call the main function.
main()
I am still really new to Python programming so please don't be too harsh on me. If anyone can figure out how to rework this to display the amount of numbers/names instead of the sum of numbers. I would greatly appreciate it. If this program cannot be reworked, I would be happy to settle for a new solution.
Edit: This it an example of what the 'names.txt' will look like:
john
mary
paul
ann
If you just want to count the lines in the file
# Open the file.
myfile = open('names.txt', 'r')
#Count the lines in the file
totalLines = len(myfile.readlines()):
# Close the file.
myfile.close()
fh = open("file","r")
print "%d lines"%len(fh.readlines())
fh.close()
or you could do
fh=open("file","r")
print "%d words"%len(fh.read().split())
fh.close()
All this is readily available information that is not hard to find if you put forth some effort...just getting the answers usually results in flunked classes...
Considering the names in your text files are delimited by line.
myfile = open('names.txt', 'r')
lstLines = myfile.read().split('\n')
dict((name,lstLines.count(name)) for name in lstLines)
This creates a dictionary of each name having its number of occurrence.
To search for the occurrence of perticular name such as 'name1' in the list
lstLines.count('name1')
Assuming names are splitted using whitespaces :
def main():
#initialize an accumulator.
total = 0.0
try:
# Open the file.
myfile = open('names.txt', 'r')
# Read and display the file's contents.
for line in myfile:
words = line.split()
total += len(words)
# Close the file.
myfile.close()
except IOError:
print('An error occured trying to read the file.')
except ValueError:
print('Non-numeric data found in the file.')
except:
print('An error occured.')
# Call the main function.
main()
Use with statement to open a file. It will close the file properly even if an exception occurred. You can omit the file mode, it is default.
If each name is on its own line and there are no duplicates:
with open('names.txt') as f:
number_of_nonblank_lines = sum(1 for line in f if line.strip())
name_count = number_of_nonblank_lines
The task is very simple. Start with a new code to avoid accumulating unused/invalid for the problem code.
If all you need is to count lines in a file (like wc -l command) then you could use .count('\n') method:
#!/usr/bin/env python
import sys
from functools import partial
read_chunk = partial(sys.stdin.read, 1 << 15) # or any text file instead of stdin
print(sum(chunk.count('\n') for chunk in iter(read_chunk, '')))
See also, Why is reading lines from stdin much slower in C++ than Python?
I am new to python and I have a question about a piece of python code that creates a cleaned up output file from a model output file. This code was written for a Mac user, but now I want to run it in Windows. But it gives an error message. Could you help me in converting this code so I can use it in Windows? Thanks.
import sys
if len(sys.argv) > 1:
fileName = sys.argv[1]
else:
print "selected_um_b.out" #insert file name here
sys.exit()
f = open(fileName)
counter = 0
fw = open(fileName+".cleaned", 'w')
for line in f:
line = line.strip()
counter = counter + 1
if counter <= 4:
fw.write(line+"\n");
continue
values = line.split("\t")
if (values[4].strip() == "-99" or values[5].strip() == "0"): continue
fw.write("\t".join(values)+"\n")
f.close()
Update
The error message is:
Traceback (most recent call last): File
"C:\trial_batch\clean_output.py", line 7, in sys.exit()
SystemExit
The program expects a filename on the command line when you execute it. It appears you did not provide one, so the program exited (the sys.exit() call terminates the program).
How are you trying to use it? If you just want to convert one file, put the file and the Python script into the same directory. Replace lines 3 through 7 with filename = "yourfilename.typ" (do not indent the line); it will read the file ("yourfilename.typ" in my example) and write an output file with 'cleaned' in the filename.