I am writing a program which stores some JSON-encoded data in a file, but sometimes the resulting file is blank (because there wasn't found any new data). When the program finds data and stores it, I do this:
with open('data.tmp') as f:
data = json.load(f)
os.remove('data.tmp')
Of course, if the file is blank this will raise an exception, which I can catch but does not let me to remove the file. I have tried:
try:
with open('data.tmp') as f:
data = json.load(f)
except:
os.remove('data.tmp')
And I get this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "MyScript.py", line 50, in run
os.remove('data.tmp')
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process
How could I delete the file when the exception occurs?
How about separating out file reading and json loading? json.loads behaves exactly same as json.load but uses a string.
with open('data.tmp') as f:
dataread = f.read()
os.remove('data.tmp')
#handle exceptions as needed here...
data = json.loads(dataread)
I am late to the party. But the json dump and load modules seem to keep using files even after writing or reading data from them. What you can do is use dumps or loads modules to get the string representation and then use normal file.write() or file.read() on the result.
For example:
with open('file_path.json'), 'w') as file:
file.write(json.dumps(json_data))
os.remove('file_path.json')
Not the best alternative but it saves me a lot especially when using temp dir.
you need to edit the remove part, so it handles the non-existing case gracefully.
import os
try:
fn = 'data.tmp'
with open(fn) as f:
data = json.load(f)
except:
try:
if os.stat(fn).st_size > 0:
os.remove(fn) if os.path.exists(fn) else None
except OSError as e: # this would be "except OSError, e:" before Python 2.6
if e.errno != errno.ENOENT:
raise
see also Most pythonic way to delete a file which may not exist
you could extract the silent removal in a separate function.
also, from the same other SO question:
# python3.4 and above
import contextlib, os
try:
fn = 'data.tmp'
with open(fn) as f:
data = json.load(f)
except:
with contextlib.suppress(FileNotFoundError):
if os.stat(fn).st_size > 0:
os.remove(fn)
I personally like the latter approach better - it's explicit.
Related
I have a script which wants to load integers from a text file. If the file does not exist I want the user to be able to browse for a different file (or the same file in a different location, I have UI implementation for that).
What I don't get is what the purpose of Exception handling, or catching exceptions is. From what I have read it seems to be something you can use to log errors, but if an input is needed catching the exception won't fix that. I am wondering if a while loop in the except block is the approach to use (or don't use the try/except for loading a file)?
with open(myfile, 'r') as f:
try:
with open(myfile, 'r') as f:
contents = f.read()
print("From text file : ", contents)
except FileNotFoundError as Ex:
print(Ex)
You need to use to while loop and use a variable to verify in the file is found or not, if not found, set in the input the name of the file and read again and so on:
filenotfound = True
file_path = myfile
while filenotfound:
try:
with open(file_path, 'r') as f:
contents = f.read()
print("From text file : ", contents)
filenotfound = False
except FileNotFoundError as Ex:
file_path = str(input())
filenotfound = True
Consider the following code:
with open('file.txt', 'r') as f:
for line in f:
print(line)
In Python 3, the interpreter tries to decode the strings it reads, which might lead to exceptions like UnicodeDecodeError. These can of course be caught with a try ... except block around the whole loop, but I would like to handle them on a per-line basis.
Question: Is there a way to directly catch and handle exceptions for each line that is read? Hopefully without changing the simple syntax of iterating over the file too much?
The Pythonic way is probably to register an error handler with codecs.register_error_handler('special', handler) and declare it in the open function:
with open('file.txt', 'r', error='special') as f:
...
That way if there is an offending line, the handler will the called with the UnicodeDecodeError, and will be able to return a replacement string or re-raise the error.
If you want a more evident processing, an alternate way would be to open the file in binary mode and explicitely decode each line:
with open('file.txt', 'rb') as f:
for bline in f:
try:
line = bline.decode()
print(line)
except UnicodeDecodeError as e:
# process error
Instead of employing a for loop, you could call next on the file-iterator yourself and catch the StopIteration manually.
with open('file.txt', 'r') as f:
while True:
try:
line = next(f)
# code
except StopIteration:
break
except UnicodeDecodeError:
# code
Basing on #SergeBallesta's answer. Here's the simplest thing that should work.
Instead of open(), use codecs.open(..., errors='your choice'). It can handle Unicode errors for you.
The list of error handler names includes
'replace': "Replace with a suitable replacement marker; Python will use the official U+FFFD REPLACEMENT CHARACTER for the built-in codecs on decoding, and ‘?’ on encoding"
which should handle the error and add a marker "there was something invalid here" to the text.
import codecs
# ...
# instead of open('filename.txt'), do:
with codecs.open('filename.txt', 'rb', 'utf-8', errors='replace') as f:
for line in f:
# ....
Place your try-except catch inside the for loop, like so:
with open('file.txt', 'r') as f:
for line in f:
try:
print(line)
except:
print("uh oh")
# continue
User may give a bunch of urls as command line args. All URLs given in the past are serialized with pickle. The script checks all given URLs, if they are unique then they are serialized and appended to a file. At least that's what should be happening. Nothing is being appended. However when I open the file in write mode,the new, unique URL is written. So what gives? Code is:
def get_new_urls():
if(len(urls.URLs) != 0): # check if empty
with open(urlFile, 'rb') as f:
try:
cereal = pickle.load(f)
print(cereal)
toDump = []
for arg in urls.URLs:
if (arg in cereal):
print("Duplicate URL {0} given, ignoring it.".format(arg))
else:
toDump.append(arg)
except Exception as e:
print("Holy bleep something went wrong: {0}".format(e))
return(toDump)
urlsToDump = get_new_urls()
print(urlsToDump)
# TODO: append new URLs
if(urlsToDump):
with open(urlFile, 'ab') as f:
pickle.dump(urlsToDump, f)
# TODO check HTML of each page against the serialized copy
with open(urlFile, 'rb') as f:
try:
cereal = pickle.load(f)
print(cereal)
except EOFError: # your URL file is empty, bruh
pass
Pickle writes out the data you give it in a special format, e.g. it will write some header/metadata/etc, to the file you give it.
It is not intended to work this way; concatenating two pickle files doesn't really make sense. To achieve a concatenation of your data, you'd need to first read whatever is in the file into your urlsToDump, then update your urlsToDump with any new data, and then finally dump it out again (overwriting the whole file, not appending).
After
with open(urlFile, 'rb') as f:
you need a while loop, to repeatedly unpickle (repeatedly read) from the file until hitting EOF.
I have a json file on my computer that I need to pass to the following function:
def read_json(file):
try:
logging.debug('Reading from input')
return read_json_from_string(json.load(file))
finally:
logging.debug('Done reading')
How can I move a file from my computer to a method in python? I have tried the following:
file = os.path.abspath('theFile.json')
Then attempting to run the method as
read_json(file)
But I get the following error:
TypeError: expected file
I also tried:
file = open('theFile.json', 'r')
But I always get an error related to the 'file' not being a file.
=====UPDATED=====
Now includes an example of calling the function
Try something like:
import logging
import json
def read_json(file):
try:
print('Reading from input')
with open(file, 'r') as f:
return json.load(f)
finally:
print('Done reading')
return_dict = read_json("my_file.json")
print return_dict
json.load takes a file-like object, and you're passing it a str containing the path. Try this instead:
path = os.path.abspath('theFile.json')
with open(path) as f:
read_json(f)
Note that json.load returns a dictionary, not a string. Also, the finally: is executed even if an exception is raised in the try:, so you'll always log "Done reading", even if an error occurred and the read was aborted.
I have a directory that has bunch of sub directories, each subdir has many csv files, but I am only interest in certain csv file. So I wrote following python method, but I am unable to capture the file name, if I do *.csv it will find all the file but I don't want to all the files to be read in:
def gatherStats(template_file, csv_file):
for lang in getLanguageCodes(csv_file):
lang_dir = os.path.join(template_file, lang)
try:
for file in os.listdir(lang_dir):
if fnmatch.fnmatch(file, '*-*-template-users-data.csv'):
t_file = open(file, 'rb').read()
reader = csv.reader()
for row in reader:
print row
else:
print "didn't find the file"
except Exception, e:
logging.exception(e)
What am I doing wrong here? Is it a regular expression issue? Can we use regular expression with fnmath?
There are several problems with your code. Fix them first, then we might get to the bottom of what your issue really is.
First of all, don't use built-in names as variables, such as file. Rather replace it with filename.
Then os.path.join(lang_dir, filename) before opening the file. Meaning:
t_file = open(os.path.join(lang_dir, filename), 'rb').read()
How do you expect reader = csv.reader() to read your file if you don't reference your open file object in this line?
Your try/except block is a bit too wide for my taste. Take your time and narrow down the errors that actually can happen. Then decide which of them you want to ignore and which should crash your program. Take a close look at the exceptions actually thrown in this block. You'll probably find your issue there.
With the help provided by another user, I manage to fix the problem. I am putting this answer here for future reference for community.
def gatherStats(template_file, csv_file):
for lang in getLanguageCodes(csv_file):
lang_dir = os.path.join(template_file, lang)
try:
for filename in os.listdir(lang_dir):
path = os.path.join(lang_dir, filename)
if re.search(r'-.+-template-users-data.csv$',filename):
with open(path, 'rb') as template_user_data_file:
reader = csv.reader(template_user_data_file)
try:
for row in reader:
print row
except csv.ERROR as e:
logging.error(e)
else:
print "didn't find the file"
except Exception, e:
logging.exception(e)