We Keep Coding

Multiple Try/Except for Validate Config-File

Thats my first question on Stackoverflow and im a totally Python beginner.
I want to write, to get firm with python, a small Backup-Programm, the main part is done, but now i want to make it a bit "portable" and use a Config file, which i want to Validate.
My class "getBackupOptions" should be give Back a validate dict which should be enriched with "GlobalOptions" and "BackupOption" so that i finally get an fully "BackupOption" dict when i call "getBackupOptions.BackupOptions".
My Question now is, (in this Example is it easy, because its only the Function which check if the Path should be Recursive searched or not) how to simplify my Code?
For each (possible) Error i must write a new "TryExcept" Block - Can i Simplify it?
Maybe is there another way to Validate Config Files/Arrays?
class getBackupOptions:
def __init__(self,BackupOption,GlobalOptions):
self.BackupOption = BackupOption
self.GlobalOptions = GlobalOptions
self.getRecusive()
def getRecusive(self):
try:
if self.BackupOption['recursive'] != None:
pass
else:
raise KeyError
except KeyError:
try:
if self.GlobalOptions['recursive'] != None:
self.BackupOption['recursive'] = self.GlobalOptions['recursive']
else:
raise KeyError
except KeyError:
print('Recusive in: ' + str(self.BackupOption) + ' and Global is not set!')
exit()
Actually i only catch an KeyError, but what if the the Key is there but there is something else than "True" or "False"?
Thanks a lot for you help!

You may try this
class getBackupOptions:
def __init__(self,BackupOption,GlobalOptions):
self.BackupOption = BackupOption
self.GlobalOptions = GlobalOptions
self.getRecusive()
def getRecusive(self):
if self.BackupOption.get('recursive') == 'True' and self.GlobalOptions.get('recursive') == 'True':
self.BackupOption['recursive'] = self.GlobalOptions['recursive']
else:
print('Recusive in: ' + str(self.BackupOption) + ' and Global is not set!')
exit()
Here get method is used, therefore KeyError will not be faced.
If any text other than True comes in the field it will be considered as False.

What is the "pythonic" way to create a list with one element or just keep it empty?

Let's say a have Person instance person. person may have an attribute id.
I want to write a function that gets me either an empty list [] when the attribute is None or missing or the list [12, ] when the attribute id is 12.
def get_id_list(person):
try:
return [getattr(person, 'id'), ]
except AttributeError:
return []
This works perfectly but is there a "pythonic" way of doing this without the try-except block and maybe in one line?

I would go for
def get_id_list(person):
_id = getattr(person, 'id', None)
return [] if _id is None else [_id]
However, it is good practice to make sure that attributes are always defined so you don't have to use getattr with a default or use hasattr to check for existence.

You may use hasattr() to check for attribute as:
def get_id_list(person):
if hasattr(person, 'id'):
return [person.id] # better then getattr(person, 'id')
else:
return []
which could be written in one line as:
def get_id_list(person):
return [person.id] if hasattr(person, 'id') else []

There are two basic ways of going about this:
EAFP (It's Easier to Ask Forgiveness than Permission)- which is what you have:
def get_id_list(person):
try:
return [person.id]
except AttributeError:
return []
LBYL (Look Before You Leap)
def get_id_list(person):
if hasattr(person, 'id'):
return [person.id]
else:
return []
Generally, EAFP is "pythonic", but really, it depends on the specifics of your use-case. If you know that person will usually have the id attribute, then you should use EAFP. That is to say, if it is an exceptional case for person not to have id, use exceptions! If, on the other hand, it is common for person to lack id, then LBYL will be more efficient.
If you simply want a one-liner, you could use a conditional expression:
return [person.id] if hasattr(person,'id') else []

Your approach is pythonic, just do some few tweaking.
First off, don't return inside the try block. You can preserve the id value in a variable and return it properly within else clause. The reason is that you are doing multiple operation like getting attribute and converting to list (and maybe more(in other cases)) while you are just catching the AttributeError. And when you use else you can handle other exceptions easily, also it's more elegant from coding style's perspective.
Secondly you don't need to use getattr in this case, you can simply use direct attribute access.
Also instead of using the try-except you might want to check if the object has the attribute then return its value but due to It's easier to ask forgiveness than it is to get permission. principal the try-exepct is better.
def get_id_list(person):
try:
id = person.id
except AttributeError:
return []
else:
return [id]

getattr() allows you to specify a default if the attribute isn't there so you could use that if you want to avoid a try/catch block.
If the named attribute does not exist, default is returned if provided, otherwise AttributeError is raised.
id = getattr(person, 'id', None)
if id is not None:
return [id]
return []

You can provide a default value (a third argument) to getattr() function to be returned if an attribute does not exist:
def get_id_list(person):
id_ = getattr(person, 'id', None)
return [id_] if id_ is not None else []
(assuming here that None is not a valid value for the id attribute)

according to the Python documentation:
EAFP:
Easier to ask for forgiveness than permission. This common Python
coding style assumes the existence of valid keys or attributes and
catches exceptions if the assumption proves false. This clean and fast
style is characterized by the presence of many try and except
statements. The technique contrasts with the LBYL style common to many
other languages such as C.
That means that your code follows the "official" pythonic way, checking if the attribute exists would be less pythonic.
However, performance needs could eventually override pythonic consideration according to the frequence where person object will not have an id attribute, because raising an exception takes more time than evaluating a simple condition.
Consider following code:
import os
from timeit import timeit
def get_id_list_try(person):
try:
return [person.id]
except AttributeError:
return []
def get_id_list_if(person):
if hasattr(person, 'id'):
return [person.id]
else:
return []
class Person(object):
def __init__(self, id):
self.id = id
person_with_id = Person(1664)
person_no_id = object()
print("try with id: {}".format(
timeit("get_id_list_try(person_with_id)", number=1000000,
setup="from __main__ import get_id_list_try, person_with_id")))
print("try no id: {}".format(
timeit("get_id_list_try(person_no_id)", number=1000000,
setup="from __main__ import get_id_list_try, person_no_id")))
print("if with id: {}".format(
timeit("get_id_list_if(person_with_id)", number=1000000,
setup="from __main__ import get_id_list_if, person_with_id")))
print("if no id: {}".format(
timeit("get_id_list_if(person_no_id)", number=1000000,
setup="from __main__ import get_id_list_if, person_no_id")))
It tests the performance of the try/catch and the if/else methods with and without an id. It prints this:
try with id: 0.25232274121
try no id: 2.32747888986
if with id: 0.364873724104
if no id: 0.728008592266
As you can see, the try/catch method is a bit faster when an id exists; but when the id does not exists the if/else method is 3 times faster than try/catch method.

How to suppress displaying the parent exception (the cause) for subsequent exceptions

I'm aware of raise ... from None and have read How can I more easily suppress previous exceptions when I raise my own exception in response?.
However, how can I achieve that same effect (of suppressing the "During handling of the above exception, another exception occurred" message) without having control over the code that is executed from the except clause? I thought that sys.exc_clear() could be used for this, but that function doesn't exist in Python 3.
Why am I asking this? I have some simple caching code that looks like (simplified):
try:
value = cache_dict[key]
except KeyError:
value = some_api.get_the_value_via_web_service_call(key)
cache_dict[key] = value
When there's an exception in the API call, the output will be something like this:
Traceback (most recent call last):
File ..., line ..., in ...
KeyError: '...'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ..., line ..., in ...
some_api.TheInterestingException: ...
This is misleading, as the original KeyError is not really an error at all. I could of course avoid the situation by changing the try/except (EAFP) into a test for the key's presence (LBYL) but that's not very Pythonic and less thread-friendly (not that the above is thread-safe as is, but that's beside the point).
It's unreasonable to expect all code in some_api to change their raise X to raise X from None (and it wouldn't even make sense in all cases). Is there a clean solution to avoid the unwanted exception chain in the error message?
(By the way, bonus question: the cache thing I used in the example is basically equivalent to cache_dict.setdefault(key, some_api.get_the_value_via_web_service_call(key)), if only the second argument to setdefault could be a callable that would only be called when the value needs to be set. Isn't there a better / canonical way to do it?)

You have a few options here.
First, a cleaner version of what orlp suggested:
try:
value = cache_dict[key]
except KeyError:
try:
value = some_api.get_the_value(key)
except Exception as e:
raise e from None
cache_dict[key] = value
For the second option, I'm assuming there's a return value hiding in there somewhere that you're not showing:
try:
return cache_dict[key]
except KeyError:
pass
value = cache_dict[key] = some_api.get_the_value(key)
return value
Third option, LBYL:
if key not in cache_dict:
cache_dict[key] = some_api.get_the_value(key)
return cache_dict[key]
For the bonus question, define your own dict subclass that defines __missing__:
class MyCacheDict(dict):
def __missing__(self, key):
value = self[key] = some_api.get_the_value(key)
return value
Hope this helps!

You can try suppressing the context yourself:
try:
value = cache_dict[key]
except KeyError:
try:
value = some_api.get_the_value_via_web_service_call(key)
except Exception as e:
e.__context__ = None
raise
cache_dict[key] = value

Here is a version of #Zachary's second option whose use is a little simpler. First, a helper subclass of dict which returns a sentinal value on a "miss" rather than throwing an exception:
class DictCache(dict):
def __missing__(self, key):
return self
then in use:
cache = DictCache()
...
value = cache[K]
if value is cache:
value = cache[K] = some_expensive_call(K)
Notice the use of "is" rather than "==" to ensure there is no collision with a valid entry.
If the thing being assigned to is a simple variable (i.e. "value" rather than an attribute of another variable "x.value"), you can even write just 2 lines:
if (value := cache[K]) is cache:
value = cache[K] = some_expensive_call(K)

Concise replacement for long list of Try-Except statement in Python?

I have this long list of try except statement:
try:
uri = entry_obj['media$group']['media$content'][0]['url']
except (KeyError, IndexError):
uri = None
try:
position = entry_obj['yt$position']['$t']
except KeyError:
position = None
try:
description = entry_obj['content']['$t']
except KeyError:
description = None
try:
seconds = entry_obj['media$group']['yt$duration']['seconds']
except KeyError:
seconds = None
try:
thumbnails = entry_obj['media$group']['media$thumbnail']
except KeyError:
thumbnails = None
Is there a more concise way to write this?

If you tire of figuring out what to use for default values in get() calls, just write a helper function:
def resolve(root, *keys):
for key in keys:
try:
root = root[key]
except (KeyError, IndexError):
return None
return root
Then you just write, e.g.:
uri = resolve(entry_obj, 'media$group', 'media$content', 0, 'url')
To simplify the calls a little, you might beef up the helper function to take a single string for the keys and split on spaces; that way you don't have to type so many quotes, and we can also add a default value argument:
def resolve(root, keys, default=None):
for key in keys.split():
try:
root = root[key]
except (TypeError, KeyError):
try:
root = root[int(key)]
except (IndexError, ValueError, KeyError):
return default
uri = resolve(entry_obj, 'media$group media$content 0 url', '')

I thought of another good way to do this, not sure how it compares to kindall's method. We first define a method property:
def res(self, property):
try:
return property()
except (KeyError, IndexError):
return None
Then replace the try-except statements with:
url = res(lambda: entry_obj['media$group']['media$content'][0]['url'])
position = res(lambda: entry_obj['yt$position']['$t'])
description = res(lambda: entry_obj['content']['$t'])
duration = res(lambda: entry_obj['media$group']['yt$duration']['seconds'])
thumbnails = res(lambda: entry_obj['media$group']['media$thumbnail'])

Use the get method of dictionaries instead:
position = entry_object.get('yt$position').get('$t')
get will handle the case of a key not existing for you, and give you a (changable) fallback value instead in that case. You'll still need to handle the first IndexError manually, but all the ones that are just except KeyError: will disappear.

Greedy execution of statements?

I have something like this using BeautifulSoup:
for line in lines:
code = l.find('span', {'class':'boldHeader'}).text
coded = l.find('div', {'class':'Description'}).text
definition = l.find('ul', {'class':'definitions'}).text
print code, coded, def
However, not all elements exist at all times. I can enclose this in a try except so that it does not break the program execution like this:
for line in lines:
try:
code = l.find('span', {'class':'boldHeader'}).text
coded = l.find('div', {'class':'Description'}).text
definition = l.find('ul', {'class':'definitions'}).text
print code, coded, def
except:
pass
But how I execute the statements in a greedy fashion? For instance, if there are only two elements available code and coded, I just want to get those and continue with the execution. As of now, even if code and coded exist, if def does not exist, the print command is never executed.
One way of doing this is to put a try...except for every statement like this:
for line in lines:
try:
code = l.find('span', {'class':'boldHeader'}).text
except:
pass
try:
coded = l.find('div', {'class':'Description'}).text
except:
pass
try:
definition = l.find('ul', {'class':'definitions'}).text
except:
pass
print code, coded, def
But this is an ugly approach and I want something cleaner. Any suggestions?

How about capture the "ugly" code in a function, and just call the function as needed:
def get_txt(l,tag,classname):
try:
txt=l.find(tag, {'class':classname}).text
except AttributeError:
txt=None
return txt
for line in lines:
code = get_txt(l,'span','boldHeader')
coded = get_txt(l,'div','Description')
defn = get_txt(l,'ul','definitions')
print code, coded, defn
PS. I changed def to defn because def is a Python keyword. Using it as a variable name raises a SyntaxError.
PPS. It's not a good practice to use bare exceptions:
try:
....
except:
...
because it almost always captures more that you intend. Much better to be explicit about what you want to catch:
try:
...
except AttributeError as err:
...

First of all, you can test for None instead of catching an exception. l.find should return None if it doesn't find your item. Exceptions should be reserved for errors and really extraordinary situations.
Second thing you can do is to create an array of all HTML elements you want to check and then have a nested for loop. Since it's been a while since I've used python, I will outline the code and then (hopefully) edit the answer when I test it.
Something like:
elementsToCheck = [
[ 'span', {'class':'boldHeader'} ],
[ 'div', {'class':'Description'} ],
[ 'ul', {'class':'definitions'} ]]
concatenated = ''
for line in lines:
for something in elementsToCheck
element = l.find(something[0], something[1])
if element is not None
concatenated += element.text
print concatenated
Obviously the code above won't work, but you should get the idea. :)