We Keep Coding

Item "None" of "Optional[str]" has no attribute "split" [union-attr]

I have a function that processes an AWS Lambda context, looking for a query string parameter.
The return value for the function is always a Tuple that contains the error code and the returned value. In case of success, it returns (None, the_value_of_the_query_string). In case of failure, it returns (Exception, None).
The code is written to behave similarly to what is very commonly seen in the Go world.
Here is the line triggering the warning:
file_name = file_path.split("/")[-1]
And below is the code that takes you through everything.
class QSException(Exception):
pass
def get_query_string(
event: Dict, query_string: str
) -> Union[Tuple[QSException, None], Tuple[None, str]]:
error = QSException()
#* [...snip...]
if not query_string in event["queryStringParameters"]:
return (error, None)
return (None, event["queryStringParameters"][query_string])
def get_file(event: Dict, context: Dict) -> Dict:
err, file_path = get_query_string(event, "file")
if err is not None:
message = {"message": "No file specified."}
return {"statusCode": 403, "body": json.dumps(message)}
# from here on I'm on the happy path
file_name = file_path.split("/")[-1]
#* [...snip...]
return {
#* [...bogus dict...]
}
If you follow the code, I treat the exceptions first and return 403 on the unhappy path. That is, once I've processed all exceptions I know for a fact that the error code was None and my result was a str. So I would expect that doing a .split("/") would work (which it does) and not trigger a typing warning.
Instead, I'm getting Item "None" of "Optional[str]" has no attribute "split" [union-attr].
So the question is how should typing look for this code so that I don't get this typing warning?

It is annotated correctly.
However when you unpack the tuple err, file_path = get_... the connection between those two variables is lost.
A static code analyzer (mypy, pyright, ...) will now assume that err is an Optional[QSException] and file_path is an Optional[str]. And when you check for the type of the first variable, it doesn't have any effect on the second variable type.
If you really want to keep that idiom, returning a tuple (exception, value), then just help the static code analyzers with asserts.
It's manual work (and therefore error prone), but I guess the tools are not clever enough to figure out the correct type in such a case.
err, file_path = get_query_string(event, "file")
if err:
return ...
assert isinstance(file_path, str)
# now static code analyzers know the correct type
However Python is not the same language as Go, and has completely different idioms.
Returning such a tuple is an antipattern in Python. Python, unlike Go, has real exceptions. So use them.
def get_query_string(event: Dict, query_string: str) -> str:
if not query_string in event["queryStringParameters"]:
raise QSException()
return event["queryStringParameters"][query_string]
def get_file(event: Dict, context: Dict) -> Dict:
try:
file_path = get_query_string(event, "file")
except QSException:
message = {"message": "No file specified."}
return {"statusCode": 403, "body": json.dumps(message)}
file_name = file_path.split("/")[-1]
Or alternatively just return an Optional[str] in case you don't wanna raise an exception, or a Union[QSE, str].

Handling different return types for Exceptions

I am working on a small texting application using Twilio API. Recently, I had some issues with the API so I added an Exception in my get_current_credits() function.
I am quite new to Python and programming in general and I would like to know what would be the cleanest way to do that.
If the Exception is throw, I only return a String. If not, I am returning a tuple. What would be the cleanest way to see what was the return from my inject_credits() function, I am thinking about type(res) but does seems a quick and dirty solution?
def get_current_credits():
try:
balance_data = twilio_client.api.v2010.balance.fetch()
balance = float(balance_data.balance)
currency = balance_data.currency
return balance, currency
except Exception:
return "503 Service Unavailable"
def inject_credit():
res = get_current_credits()
# if Exception:
# return the Exception message as a string
# else, do the following:
(credit, currency) = res
if currency != "EUR":
credit = currency_converter.convert(credit, currency, 'EUR')
return dict(credit=(round(credit,2)))

You could move the Exception outside, into the body of inject_credit. Thus, you don't have to do any if statements inside inject_credit, you can just catch the Exception there itself.
Checking the type isn't a bad idea, but it is not very clear what is being done if someone is only reading inject_credit.

You can try this:
if type(get_current_credits()) == str:
# Do something if it is a string
else:
# Do something if it is a tuple
However, the best way would be to add the try statement outside of the function so that you can catch the exception there instead.
try:
get_current_credits()
except Exception as e:
print(e)
# Do whatever

Putting try and catch in the right place

I have a very simple caching service that caches files on S3. There are times when the file I am trying to cache locally does not exist on AWS S3. As such in one of my files that uses the caching service I prefer to return None if the file i am trying to cache is no found.
However I realize that i will be using the caching service in many other places and as a result I have been told by my peers that cache.cache_file() should still raise an error in this case but a simpler one like FileNotFoundError that doesn't require the caller to do if e.response["Error"]["Code"] == "404"
My Caching Code
import logging
import os
from pathlib import Path
from stat import S_IREAD, S_IRGRP, S_IROTH
from mylib import s3
from mylib.aws.clients import s3_client, s3_resource
logger = logging.getLogger(__name__)
class Cache:
def _is_file_size_equal(self, s3_path_of_file: str, local_path: Path, file_name: str) -> bool:
bucket, key = s3.deconstruct_s3_url(f"{s3_path_of_file}/{file_name}")
s3_file_size = s3_resource().Object(bucket, key).content_length
local_file_size = (local_path / file_name).stat().st_size
return s3_file_size == local_file_size
def cache_file(self, s3_path_of_file: str, local_path: Path, file_name: str) -> None:
bucket, key = s3.deconstruct_s3_url(f"{s3_path_of_file}/{file_name}")
if not (local_path / file_name).exists() or not self._is_file_size_equal(
s3_path_of_file, local_path, file_name
):
os.makedirs(local_path, exist_ok=True)
s3_client().download_file(bucket, key, f"{local_path}/{file_name}")
os.chmod(local_path / file_name, S_IREAD | S_IRGRP | S_IROTH)
else:
logger.info("Cached File is Valid!")
My Code that calls the Caching Code
def get_required_stream(environment: str, proxy_key: int) -> Optional[BinaryIO]:
s3_overview_file_path = f"s3://{TRACK_BUCKET}/{environment}"
overview_file = f"{some_key}.mv"
local_path = _cache_directory(environment)
try:
cache.cache_file(s3_overview_file_path, local_path, overview_file)
overview_file_cache = local_path / f"{proxy_key}.mv"
return overview_file_cache.open("rb")
except botocore.exceptions.ClientError as e:
if e.response["Error"]["Code"] == "404":
return None
else:
raise
Issue
Being new to Python I am a little unsure how this would work. I assume it means that my code that calls the caching service especially the except part would look something like this.
except FileNotFoundError:
return None
And in the caching service where i have s3_client().download_file(bucket, key, f"{local_path}/{file_name}") I would wrap it with a try and catch ?
While this question probably comes across as trivial and it is I thought I would ask it here anyway since it would be good learning opportunity for me and also understand how to write clean code. I would love suggestions on how I can achieve the desired and if my assumption is wrong?

def get_required_stream(environment: str, proxy_key: int) -> Optional[BinaryIO]:
s3_overview_file_path = f"s3://{TRACK_BUCKET}/{environment}"
overview_file = f"{some_key}.mv"
local_path = _cache_directory(environment)
try:
cache.cache_file(s3_overview_file_path, local_path, overview_file)
overview_file_cache = local_path / f"{proxy_key}.mv"
return overview_file_cache.open("rb")
except botocore.exceptions.ClientError as e:
if e.response["Error"]["Code"] == "404":
exc = FileNotFoundError()
exc.filename = overview_file_cache
raise exc
raise
# then you can use your function like this
try:
filedesc = get_required_stream(...)
except FileNotFoundError e:
print(f'{e.filename} not found')

If you do not want calling code to catch botocore.exceptions.ClientError exception, you could wrap your cache_file method in a try except block and throw a specific exception. I would also go a step further and create a simple custom exception object that wraps botocore.exceptions.ClientError and exposes error_code and error_message from boto exception. That way, caller doesn't have to separately catch FileNotFoundError when file not found and then again botocore.exceptions.ClientError for a different type of error (say permission or some network error). They can just catch the custom exception and further inspect for more details.
try:
//do something
except botocore.exceptions.ClientError as ex:
raise YourCustomS3Exception(ex) //YourCustomS3Exception needs to handle ex

One clean way would be an additional file_exists() method in the Cache class, so every user of the cache can check before they attempt the actual caching/download. Just like using pythons filesystem/path functions.
An exception can still occur if the file is deleted/becomes unreachable between the file_exists() call and the download, but I think in this rare case, the botocore exception is just fine.

Opening different file on IOError

I've written a python script that wants to write logs to a file at /var/log/myapp.log. However, on some platforms this doesn't exist, or we might not have permission to do that. In that case, I'd like to try writing somewhere else.
def get_logfile_handler():
log_file_handler = None
log_paths = ['/var/log/myapp.log', './myapp.log']
try:
log_file_handler = logging.FileHandler(log_paths[0])
except IOError:
log_file_handler = logging.FileHandler(log_paths[1])
return log_file_handler
The above code may work, but it seems far from elegant - in particular, trying a different file as part of the exception handling seems wrong. It could just throw another exception!
Ideally, it would take an arbitrary list of paths rather than just two. Is there an elegant way to write this?

you can simply use a loop such as:
def get_logfile_handler():
log_file_handler = None
log_paths = ['/var/log/myapp.log', './myapp.log']
for log_path in log_paths:
try:
return logging.FileHandler(log_path)
except IOError:
pass
raise Exception("Cannot open log file!")
HTH

There is, as #PM-2's comment suggests, no need to reference each possible path individually. You could try something like this:
def getlogfile_handler():
log_file_handler = None
log_paths = ('/var/log/myapp.log', './myapp.log') # and more
for log_path in log_paths:
try:
log_file_handler = logging.FileHandler(log_path)
break
except IOError:
continue
else:
raise ValueError("No log path available")
return log_file_handler
The else clause handles the case where the loop terminates without finding a suitable log_path value. If the loop breaks early then (and only then) the return statement is executed.
It's perfectly OK to use exceptions for control flow purposes like this - the cases are exceptional but they aren't errors - the only real error occurs when no path can be found, in which case the code raises its own exception that the caller may catch if it so chooses.

def get_logfile_handler( log_files ):
log_file_handler = None
for path in log_files:
try:
log_file_handler = logging.FileHandler(path)
break
except IOError:
pass
return log_file_handler
This may solve your problem. You can define locally/globally instead of passing as a parameter.

As a possible solution if you want to avoid handling exceptions you can check the Operating System you are on by using the platform module and then choose the required path. Then you can use the os and stat modules in order to check permissions.
import platform
import os
import stat
...
#the logic goes here
...

How to raise Suds.WebFault from python code?

I am trying to raise a Suds.WebFault from python code. The __init__ method\constructor takes three arguments __init__(self, fault, document). The fault has fault.faultcode and fault.detail members\attributes\properties. I could not find out what class fault belongs to no matte what I tried. How do I raise Suds.WebFault type exception from python code?
Thanks in advance.

Not sure what exactly you are asking but you can throw a web fault using:
import suds
try:
client.service.Method(parameter)
except suds.WebFault, e:
print e

WebFault is defined in suds.__init__.py as:
class WebFault(Exception):
def __init__(self, fault, document):
if hasattr(fault, 'faultstring'):
Exception.__init__(self, u"Server raised fault: '%s'" %
fault.faultstring)
self.fault = fault
self.document = document
Therefore to raise a WebFault with a meaningful message you will need to pass an object as the first param with the message. Document can simply be None unless required.
import suds
class Fault(object):
faultstring = 'my error message'
raise suds.WebFault(Fault(), document=None)

WebFault is only meant to be actually raised when a <Fault> element is returned by the web server. So it's probably a bad idea to raise that yourself.
If you still would like to, I would start looking at the code where the framework raises it: https://github.com/unomena/suds/blob/4e90361e13fb3d74915eafc1f3a481400c798e0e/suds/bindings/binding.py#L182 - and work backwards from there.