I am currently using the json module to load some JSON content and I need to exclude NaN values. The standard way to do this seems to be to implement a parse_constant method and then pass this method to json.loads.
Example:
def parse_constant(c: str) -> float:
if c == "NaN":
raise ValueError("NaN is not valid JSON")
return float(c)
json_content = json.loads(some_json_string, parse_constant=parse_constant)
Apparently, the decode implementation does not catch the ValueError and transform it into a JSONDecodeError, which forces me to check for both exception types. I miss the context (line number, JSON document) to build the decode error myself. I know I could just catch ValueError, but is there a way to configure the json module to raise what I would expect to be the logical exception type?
Related
I have a function that processes an AWS Lambda context, looking for a query string parameter.
The return value for the function is always a Tuple that contains the error code and the returned value. In case of success, it returns (None, the_value_of_the_query_string). In case of failure, it returns (Exception, None).
The code is written to behave similarly to what is very commonly seen in the Go world.
Here is the line triggering the warning:
file_name = file_path.split("/")[-1]
And below is the code that takes you through everything.
class QSException(Exception):
pass
def get_query_string(
event: Dict, query_string: str
) -> Union[Tuple[QSException, None], Tuple[None, str]]:
error = QSException()
#* [...snip...]
if not query_string in event["queryStringParameters"]:
return (error, None)
return (None, event["queryStringParameters"][query_string])
def get_file(event: Dict, context: Dict) -> Dict:
err, file_path = get_query_string(event, "file")
if err is not None:
message = {"message": "No file specified."}
return {"statusCode": 403, "body": json.dumps(message)}
# from here on I'm on the happy path
file_name = file_path.split("/")[-1]
#* [...snip...]
return {
#* [...bogus dict...]
}
If you follow the code, I treat the exceptions first and return 403 on the unhappy path. That is, once I've processed all exceptions I know for a fact that the error code was None and my result was a str. So I would expect that doing a .split("/") would work (which it does) and not trigger a typing warning.
Instead, I'm getting Item "None" of "Optional[str]" has no attribute "split" [union-attr].
So the question is how should typing look for this code so that I don't get this typing warning?
It is annotated correctly.
However when you unpack the tuple err, file_path = get_... the connection between those two variables is lost.
A static code analyzer (mypy, pyright, ...) will now assume that err is an Optional[QSException] and file_path is an Optional[str]. And when you check for the type of the first variable, it doesn't have any effect on the second variable type.
If you really want to keep that idiom, returning a tuple (exception, value), then just help the static code analyzers with asserts.
It's manual work (and therefore error prone), but I guess the tools are not clever enough to figure out the correct type in such a case.
err, file_path = get_query_string(event, "file")
if err:
return ...
assert isinstance(file_path, str)
# now static code analyzers know the correct type
However Python is not the same language as Go, and has completely different idioms.
Returning such a tuple is an antipattern in Python. Python, unlike Go, has real exceptions. So use them.
def get_query_string(event: Dict, query_string: str) -> str:
if not query_string in event["queryStringParameters"]:
raise QSException()
return event["queryStringParameters"][query_string]
def get_file(event: Dict, context: Dict) -> Dict:
try:
file_path = get_query_string(event, "file")
except QSException:
message = {"message": "No file specified."}
return {"statusCode": 403, "body": json.dumps(message)}
file_name = file_path.split("/")[-1]
Or alternatively just return an Optional[str] in case you don't wanna raise an exception, or a Union[QSE, str].
I need to parse "order" from below JSON , if only value of "success" = 'true' , else raise an exception.
Tried below, but not sure how to include the 'true' check in try:
{
"success":true,
"order":"123345"
}
below is the code , I am trying , which is not giving any result from print as well.
import json
from pprint import pprint
data = json.load(open('data.json'))
#pprint(data)
try:
check_key = data['success']
except KeyError:
#continue
print(check_key)
#print(data['order'])
You should evaluate data['success'] in a condition, whether it is false, then you raise your exception.
import json
data = json.load(open('data.json'))
if data['success'] is not True:
raise Exception("Success is false")
order = data['order']
print(order)
I need to parse "order" from below JSON , if only value of "success" = 'true' , else raise an exception.
There's no function that will automatically raise an exception if a value is False; you need to write that yourself.
But it's dead easy:
check_key = data.get('success')
if not check_key:
raise MyKindOfError(f'response success was {check_key}')
do_stuff(data['order'])
(You don't actually need to use get there; you could let data['success'] raise a KeyError if it's not present, and then separately check the value for falsiness and raise your own error. But I assume you probably want to handle a missing success the same way as false, and the error you want to raise probably isn't KeyError, in which case this is simpler.)
As a side note, you've already parsed the JSON at this point. What you have is a dict.
The fact that it originally came from parsing JSON doesn't make any difference; it's a plain old Python dict, with all the same methods, etc., as any other dict. So, it really isn't helpful to think of "how do I … with JSON …"; that just misleads you into forgetting about how easy dicts are to use.
When I try to serialize an exception using json.dump, I get errors like
TypeError: IOError('socket error', error(61, 'Connection refused')) is not JSON serializable
and
TypeError: error(61, 'Connection refused') is not JSON serializable
The __dict__ field of exceptions is {} (this is why How to make a class JSON serializable does not help me: the answers there assume that __dict__ contains all the necessary information, they also assume that I have control over the class to be serialized).
Is there something more intelligent that saving str(exn)?
I would prefer a human-readable text representation (not pickle).
PS. Here is what I came up with:
def exception_as_dict(ex):
return dict(type=ex.__class__.__name__,
errno=ex.errno, message=ex.message,
strerror=exception_as_dict(ex.strerror)
if isinstance(ex.strerror,Exception) else ex.strerror)
json.dumps(exception_as_dict(err),indent=2)
{
"errno": "socket error",
"type": "IOError",
"strerror": {
"errno": 61,
"type": "error",
"strerror": "Connection refused"
}
}
You can use exc_info with traceback as below:
import traceback
import sys
try:
raise KeyError('aaa!!!')
except Exception as e:
exc_info = sys.exc_info()
print(''.join(traceback.format_exception(*exc_info)))
Exceptions can not be pickled (by default), you have two options:
Use Python's built in format_exc() and serialize the formatted string.
Use tblib
With the latter, you can pass wrapped exceptions and also reraise them later.
import tblib.pickling_support
tblib.pickling_support.install()
import pickle, sys
def inner_0():
raise Exception('fail')
def inner_1():
inner_0()
def inner_2():
inner_1()
try:
inner_2()
except:
s1 = pickle.dumps(sys.exc_info())
I created a class named Options. It works fine but not not with Python 2.
And I want it to work on both Python 2 and 3.
The problem is identified: FileNotFoundError doesn t exist in Python 2.
But if I use IOError it doesn t work in Python 3
Changed in version 3.3: EnvironmentError, IOError, WindowsError, VMSError, socket.error, select.error and mmap.error have been merged into OSError.
What should I do ???(Please do not discuss my choice of portability, I have reasons.)
Here s the code:
#!/usr/bin/python
#-*-coding:utf-8*
#option_controller.py
#Walle Cyril
#25/01/2014
import json
import os
class Options():
"""Options is a class designed to read, add and change informations in a JSON file with a dictionnary in it.
The entire object works even if the file is missing since it re-creates it.
If present it must respect the JSON format: e.g. keys must be strings and so on.
If something corrupted the file, just destroy the file or call read_file method to remake it."""
def __init__(self,directory_name="Cache",file_name="options.json",imported_default_values=None):
#json file
self.option_file_path=os.path.join(directory_name,file_name)
self.directory_name=directory_name
self.file_name=file_name
#self.parameters_json_file={'sort_keys':True, 'indent':4, 'separators':(',',':')}
#the default data
if imported_default_values is None:
DEFAULT_INDENT = 2
self.default_values={\
"translate_html_level": 1,\
"indent_size":DEFAULT_INDENT,\
"document_title":"Titre"}
else:
self.default_values=imported_default_values
def read_file(self,read_this_key_only=False):
"""returns the value for the given key or a dictionary if the key is not given.
returns None if it s impossible"""
try:
text_in_file=open(self.option_file_path,'r').read()
except FileNotFoundError:#not 2.X compatible
text_in_file=""#if the file is not there we re-make one with default values
if text_in_file=="":#same if the file is empty
self.__insert_all_default_values()
text_in_file=open(self.option_file_path,'r').read()
try:
option_dict=json.loads(text_in_file)
except ValueError:
#if the json file is broken we re-make one with default values
self.__insert_all_default_values()
text_in_file=open(self.option_file_path,'r').read()
option_dict=json.loads(text_in_file)
if read_this_key_only:
if read_this_key_only in option_dict:
return option_dict[read_this_key_only]#
else:
#if the value is not there it should be written for the next time
if read_this_key_only in self.default_values:
self.add_option_to_file(read_this_key_only,self.default_values[read_this_key_only])
return self.default_values[read_this_key_only]
else:
#impossible because there is not default value so the value isn t meant to be here
return None
else:
return option_dict
def add_option_to_file(self,key,value):#or update
"""Adds or updates an option(key and value) to the json file if the option exists in the default_values of the object."""
option_dict=self.read_file()
if key in self.default_values:
option_dict[key]=value
open(self.option_file_path,'w').write(\
json.dumps(option_dict,sort_keys=True, indent=4, separators=(',',':')))
def __insert_all_default_values(self):
"""Recreate json file with default values.
called if the document is empty or non-existing or corrupted."""
try:
open(self.option_file_path,'w').write(\
json.dumps(self.default_values,sort_keys=True, indent=4, separators=(',',':')))
except FileNotFoundError:
os.mkdir(self.directory_name)#Create the directory
if os.path.isdir(self.directory_name):#succes
self.__insert_all_default_values()
else:
print("Impossible to write in %s and file %s not found" % (os.getcwd(),self.option_file_path))
#demo
if __name__ == '__main__':
option_file_object=Options()
print(option_file_object.__doc__)
print(option_file_object.read_file())
option_file_object.add_option_to_file("","test")#this should have no effect
option_file_object.add_option_to_file("translate_html_level","0")#this should have an effect
print("value of translate_html_level:",option_file_object.read_file("translate_html_level"))
print(option_file_object.read_file())
If FileNotFoundError isn't there, define it:
try:
FileNotFoundError
except NameError:
FileNotFoundError = IOError
Now you can catch FileNotFoundError in Python 2 since it's really IOError.
Be careful though, IOError has other meanings. In particular, any message should probably say "file could not be read" rather than "file not found."
You can use the base class exception EnvironmentError and use the 'errno' attribute to figure out which exception was raised:
from __future__ import print_function
import os
import errno
try:
open('no file of this name') # generate 'file not found error'
except EnvironmentError as e: # OSError or IOError...
print(os.strerror(e.errno))
Or just use IOError in the same way:
try:
open('/Users/test/Documents/test') # will be a permission error
except IOError as e:
print(os.strerror(e.errno))
That works on Python 2 or Python 3.
Be careful not to compare against number values directly, because they can be different on different platforms. Instead, use the named constants in Python's standard library errno module which will use the correct values for the run-time platform.
The Python 2 / 3 compatible way to except a FileNotFoundError is this:
import errno
try:
with open('some_file_that_does_not_exist', 'r'):
pass
except EnvironmentError as e:
if e.errno != errno.ENOENT:
raise
Other answers are close, but don't re-raise if the error number doesn't match.
Using IOError is fine for most cases, but for some reason os.listdir() and friends raise OSError instead on Python 2. Since IOError inherits from OSError it's fine to just always catch OSError and check the error number.
Edit: The previous sentence is only true on Python 3. To be cross compatible, instead catch EnvironmentError and check the error number.
For what it's worth, although the IOError is hardly mentioned in Python 3's official document and does not even showed up in its official Exception hierarchy, it is still there, and it is the parent class of FileNotFoundError in Python 3. See python3 -c "print(isinstance(FileNotFoundError(), IOError))" giving you a True. Therefore, you can technically write your code in this way, which works for both Python 2 and Python 3.
try:
content = open("somefile.txt").read()
except IOError: # Works in both Python 2 & 3
print("Oops, we can not read this file")
It might be "good enough" in many cases. Although in general, it is not recommended to rely on an undocumented behavior. So, I'm not really suggesting this approach. I personally use Kindall's answer.
I have a weird error when try to redirect a exception to STDERR.
I have a script that is use to load several "plugins", working as main entry program. The plugins do stuff like connect to databases, parsing text data, connect to web services, etc...
Is like this:
try:
Run plugins here...
#All was ok!
print "Ok!"
sys.exit(0)
except Exception,e:
sys.stderr.writelines([unicode(e),u'\n',u'\n'])
traceback.print_exc(file=sys.stderr)
sys.exit(-1)
This is executed in the command-line, and sometimes I get the error:
TypeError: writelines() argument must be a sequence of strings
I have no clue how in this earth a Exception is not returning as a string here.
My solution to this was to encode the text in UTF-8
file.writelines("some unicode text here".encode('utf-8'))
I finally figure this out.
This happend when:
try:
raise Exception(u'Error with unicode chars')
except:
sys.stderr.write(u'%s\n\n' % e)
I hack with this (from activestate community):
def safe_unicode(obj, * args):
""" return the unicode representation of obj """
try:
return unicode(obj, * args)
except UnicodeDecodeError:
# obj is byte string
ascii_text = str(obj).encode('string_escape')
return unicode(ascii_text)
def safe_str(obj):
""" return the byte string representation of obj """
try:
return str(obj)
except UnicodeEncodeError:
# obj is unicode
return unicode(obj).encode('unicode_escape')
#code
except Exception,e:
sys.stderr.write(u'%s\n\n' % safe_unicode(safe_str(e)))
It's highly likely that e is at fault. I have no idea why. Possibly something is not being coerced properly.
Does this help?
sys.stderr.write(u'%s\n\n' % e)
To get a better clue about what is happening, wrap the offending statement in try:except: ... in the except clause, do
print repr(e), repr(unicode(e))
Why do you need unicode strings instead of str strings?