We Keep Coding

Overpass a custom exception but print all the other exceptions

I am running the following try-except code:
try:
paths = file_system_client.get_paths("{0}/{1}/0/{2}/{3}/{4}".format(container_initial_folder, container_second_folder, chronological_date[0], chronological_date[1], chronological_date[2]), recursive=True)
list_of_paths=["abfss://{0}#{1}.dfs.core.windows.net/".format(storage_container_name, storage_account_name)+path.name for path in paths if ".avro" in path.name]
except Exception as e:
if e=="AccountIsDisabled":
pass
else:
print(e)
I want neither to print the following error if my try-except fells upon it nor to stop my program execution if I fell upon this error:
"(AccountIsDisabled) The specified account is disabled.
RequestId:3159a59e-d01f-0091-5f71-2ff884000000
Time:2020-05-21T13:09:03.3540242Z"
I just want to overpass it and print any other error/exception (eg. TypeError, ValueError, etc) that will occur.
Is this feasible in Python 3?
Please note that the .get_paths() method belongs to the azure.storage.filedatalake module which enables direct connection of Python with Azure Data Lake for path extraction.
I am giving the note to pinpoint that the Exception I am trying to bypass is not a built-in Exception.
[Update] In sort after following the proposed attached answers I modified my code to this:
import sys
from concurrent.futures import ThreadPoolExecutor
from azure.storage.filedatalake._models import StorageErrorException
from azure.storage.filedatalake import DataLakeServiceClient, DataLakeFileClient
storage_container_name="name1" #confidential
storage_account_name="name2" #confidential
storage_account_key="password" #confidential
container_initial_folder="name3" #confidential
container_second_folder="name4" #confidential
def datalake_connector(storage_account_name, storage_account_key):
global service_client
datalake_client = DataLakeServiceClient(account_url="{0}://{1}.dfs.core.windows.net".format("https", storage_account_name), credential=storage_account_key)
print("Client successfuly created!")
return datalake_client
def create_list_paths(chronological_date,
container_initial_folder="name3",
container_second_folder="name4",
storage_container_name="name1",
storage_account_name="name2"
):
list_of_paths=list()
print("1. success")
paths = file_system_client.get_paths("{0}/{1}/0/{2}/{3}/{4}".format(container_initial_folder, container_second_folder, chronological_date[0], chronological_date[1], chronological_date[2]), recursive=True)
print("2. success")
list_of_paths=["abfss://{0}#{1}.dfs.core.windows.net/".format(storage_container_name, storage_account_name)+path.name for path in paths if ".avro" in path.name]
print("3. success")
list_of_paths=functools.reduce(operator.iconcat, result, [])
return list_of_paths
service_client = datalake_connector(storage_account_name, storage_account_key)
file_system_client = service_client.get_file_system_client(file_system=storage_container_name)
try:
list_of_paths=[]
executor=ThreadPoolExecutor(max_workers=8)
print("Start path extraction!")
list_of_paths=[executor.submit(create_list_paths, i, container_initial_folder, storage_container_name, storage_account_name).result() for i in date_list]
except:
print("no success")
print(sys.exc_info())
Unfortunately the StorageErrorException cannot be handled for a reason, I am still getting the following stdout:

Listing [Python 3.Docs]: Compound statements - The try statement.
There are several ways of achieving this. Here's one:
try:
# ...
except StorageErrorException:
pass
except:
print(sys.exc_info()[1])
Note that except: is tricky because you might silently handle exceptions that you shouldn't. Another way would be to catch any exception the code could raise explicitly.
try:
# ...
except StorageErrorException:
pass
except (SomeException, SomeOtherException, SomeOtherOtherException) as e:
print(e)
Quickly browsing [MS.Docs]: filedatalake package and the sourcecode, revealed that StorageErrorException (which extends [MS.Docs]: HttpResponseError class) is the one that you need to handle.
Might want to check [SO]: About catching ANY exception.
Related to the failure of catching the exception, apparently there are 2 having the same name:
azure.storage.blob._generated.models._models_py3.StorageErrorException (currently imported)
azure.storage.filedatalake._generated.models._models_py3.StorageErrorException
I don't know the rationale (I didn't work with the package), but given the fact the package raises an exception defined in another package when it also defines one with the same name, seems lame. Anyway importing the right exception solves the problem.
As a side note, when dealing with this kind of situation, don't only import the base name, but work with the fully qualified one:
import azure.storage.filedatalake._generated.models.StorageErrorException

you want to compare the type of the exception, change your condition to:
if type(e)==AccountIsDisabled:
example:
class AccountIsDisabled(Exception):
pass
print("try #1")
try:
raise AccountIsDisabled
except Exception as e:
if type(e)==AccountIsDisabled:
pass
else:
print(e)
print("try #2")
try:
raise Exception('hi', 'there')
except Exception as e:
if type(e)==AccountIsDisabled:
pass
else:
print(e)
Output:
try #1
try #2
('hi', 'there')

Python try except fail to catch RemoteDataError

I am using the following code to read data from yahoo finance from a list symbols downloaded from nasdaq.
pnls = {i:dreader.DataReader(i,'yahoo','1985-01-01','2017-03-30') for i in symbols}
for df_name in pnls:
try:
pnls.get(df_name).to_csv("/Users/Jiong/Documents/data/{}_data.csv".format(df_name), index=True, header=True)
except:
print("error {}".format(df_name))
else:
print("done {}".format(df_name))
Guess some symbols may not be valid and Yahoo Finance throws RemoteDataError Exception.
The code above is supposed to continue on, but it stopped still at the errors.
Isn't except to catch all exceptions? or this is run time error ?
Anyway to get the code ignores it and continue? Thanks. See errors below running
118 if params is not None and len(params) > 0:
119 url = url + "?" + urlencode(params)
--> 120 raise RemoteDataError('Unable to read URL: {0}'.format(url))
121
122 def _read_lines(self, out):
RemoteDataError: Unable to read URL: http://ichart.finance.yahoo.com/table.csv?c=1985&f=2017&s=MITT%5EA&g=d&ignore=.csv&d=2&e=30&a=0&b=1

You need to handle the raise exception or it will halt at the place of raised. So, if a raise exception is not caught and handled, it will be interrupted.
What you need is something like this :
except RemoteDataError as exp :
print('Unable to read URL: {0}'.format(url))
You can refer to this documentation for more details on errors.

Grabbed this import from another page; it seemed to correct the issue for me. I was getting a remote error while trying to pull stock data from Yahoo. You'll need to do the following import prior to adding the RemoteDataError exception:
from pandas_datareader._utils import RemoteDataError
df_ = pd.DataFrame()
for i in assets:
print(i)
try:
vec = web.DataReader(i, 'yahoo', start='12/10/2006', end='2/1/2007')
vec['asset'] = i
vec['returns_close_raw'] = np.log(vec.Close/vec.Close.shift())
vec['returns_open_raw'] = np.log(vec.Open/vec.Open.shift())
vec['returns_open_raw10'] = np.log(vec.Open/vec.Open.shift(10))
vec['returns_close_raw10'] = np.log(vec.Close/vec.Close.shift(10))
df_ = pd.concat([df_, vec])
except RemoteDataError:
print('remote error')
except KeyError:
print('key error')

How to handle multiple exceptions?

I'm a Python learner, trying to handle a few scenarios:
Reading a file.
Formatting Data.
Manipulating/Copying Data.
Writing a file.
So far, I have:
try:
# Do all
except Exception as err1:
print err1
#File Reading error/ File Not Present
except Exception as err2:
print err2
# Data Format is incorrect
except Exception as err3:
print err3
# Copying Issue
except Exception as err4:
print err4
# Permission denied for writing
The idea of implementing in this fashion is to catch the exact error for all different scenarios. I can do it in all separate try/except blocks.
Is this possible? And reasonable?

Your try blocks should be as minimal as possible, so
try:
# do all
except Exception:
pass
is not something you want to do.
The code in your example won't work as you expect it to, because in every except block you are catching the most general exception type Exception. In fact, only the first except block will ever be executed.
What you want to be doing is having multiple try/except blocks, each one responsible for as few things as possible and catching the most specific exception.
For example:
try:
# opening the file
except FileNotFoundException:
print('File does not exist')
exit()
try:
# writing to file
except PermissionError:
print('You do not have permission to write to this file')
exit()
However, sometimes it is appropriate to catch different types of exceptions, in the same except block or in several blocks.
try:
ssh.connect()
except (ConnectionRefused, TimeoutError):
pass
or
try:
ssh.connect()
except ConnectionRefused:
pass
except TimeoutError:
pass

As DeepSpace stated,
Your try blocks should be as minimal as possible.
If you want to achieve
try:
# do all
except Exception:
pass
Then you might as well do something like
def open_file(file):
retval = False
try:
# opening the file succesful?
retval = True
except FileNotFoundException:
print('File does not exist')
except PermissionError:
print('You have no permission.')
return retval
def crunch_file(file):
retval = False
try:
# conversion or whatever logical operation with your file?
retval = True
except ValueError:
print('Probably wrong data type?')
return retval
if __name__ == "__main__":
if open_file(file1):
open(file1)
if open_file(file2) and crunch_file(file2):
print('opened and crunched')

Yes this is possible.
Just say as example:
try:
...
except RuntimeError:
print err1
except NameError:
print err2
...
Just define the exact Error you want to intercept.

Safest way to open a file in python 3.4

I was expecting the following would work but PyDev is returning an error:
try fh = open(myFile):
logging.info("success")
except Exception as e:
logging.critical("failed because:")
logging.critical(e)
gives
Encountered "fh" at line 237, column 5. Was expecting: ":"...
I've looked around and cannot find a safe way to open a filehandle for reading in Python 3.4 and report errors properly. Can someone point me in the correct direction please?

You misplaced the :; it comes directly after try; it is better to put that on its own, separate line:
try:
fh = open(myFile)
logging.info("success")
except Exception as e:
logging.critical("failed because:")
logging.critical(e)
You placed the : after the open() call instead.
Instead of passing in e as a separate argument, you can tell logging to pick up the exception automatically:
try:
fh = open(myFile)
logging.info("success")
except Exception:
logging.critical("failed because:", exc_info=True)
and a full traceback will be included in the log. This is what the logging.exception() function does; it'll call logging.error() with exc_info set to true, producing a message at log level ERROR plus a traceback.

closing files properly opened with urllib2.urlopen()

I have following code in a python script
try:
# send the query request
sf = urllib2.urlopen(search_query)
search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())
sf.close()
except Exception, err:
print("Couldn't get programme information.")
print(str(err))
return
I'm concerned because if I encounter an error on sf.read(), then sf.clsoe() is not called.
I tried putting sf.close() in a finally block, but if there's an exception on urlopen() then there's no file to close and I encounter an exception in the finally block!
So then I tried
try:
with urllib2.urlopen(search_query) as sf:
search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())
except Exception, err:
print("Couldn't get programme information.")
print(str(err))
return
but this raised a invalid syntax error on the with... line.
How can I best handle this, I feel stupid!
As commenters have pointed out, I am using Pys60 which is python 2.5.4

I would use contextlib.closing (in combination with from __future__ import with_statement for old Python versions):
from contextlib import closing
with closing(urllib2.urlopen('http://blah')) as sf:
search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())
Or, if you want to avoid the with statement:
try:
sf = None
sf = urllib2.urlopen('http://blah')
search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())
finally:
if sf:
sf.close()
Not quite as elegant though.

finally:
if sf: sf.close()

Why not just try closing sf, and passing if it doesn't exist?
import urllib2
try:
search_query = 'http://blah'
sf = urllib2.urlopen(search_query)
search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())
except urllib2.URLError, err:
print(err.reason)
finally:
try:
sf.close()
except NameError:
pass

Given that you are trying to use 'with', you should be on Python 2.5, and then this applies too: http://docs.python.org/tutorial/errors.html#defining-clean-up-actions

If urlopen() has an exception, catch it and call the exception's close() function, like this:
try:
req = urllib2.urlopen(url)
req.close()
print 'request {0} ok'.format(url)
except urllib2.HTTPError, e:
e.close()
print 'request {0} failed, http code: {1}'.format(url, e.code)
except urllib2.URLError, e:
print 'request {0} error, error reason: {1}'.format(url, e.reason)
the exception is also a full response object, you can see this issue message: http://bugs.jython.org/issue1544

Looks like the problem runs deeper than I thought - this forum thread indicates urllib2 doesn't implement with until after python 2.6, and possibly not until 3.1

You could create your own generic URL opener:
from contextlib import contextmanager
#contextmanager
def urlopener(inURL):
"""Open a URL and yield the fileHandle then close the connection when leaving the 'with' clause."""
fileHandle = urllib2.urlopen(inURL)
try: yield fileHandle
finally: fileHandle.close()
Then you could then use your syntax from your original question:
with urlopener(theURL) as sf:
search_soup = BeautifulSoup.BeautifulSoup(sf.read())
This solution gives you a clean separation of concerns. You get a clean generic urlopener syntax that handles the complexities of properly closing the resource regardless of errors that occur underneath your with clause.

Why not just use multiple try/except blocks?
try:
# send the query request
sf = urllib2.urlopen(search_query)
except urllib2.URLError as url_error:
sys.stderr.write("Error requesting url: %s\n" % (search_query,))
raise
try:
search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())
except Exception, err: # Maybe catch more specific Exceptions here
sys.stderr.write("Couldn't get programme information from url: %s\n" % (search_query,))
raise # or return as in your original code
finally:
sf.close()