We Keep Coding

Prevent pytest from caching modules that dynamically modify and reimport themselves

I am trying to use pytest to test some code that uses towerlib, an external Python package. Part of the functionality is that towerlib will dynamically add different attributes to its credential.py module, and then the class imports its own module and looks for that attribute:
This is in towerlib.entities.credential.py:
class Credential: # pylint: disable=too-few-public-methods
"""Credential factory to handle the different credential types returned."""
def __new__(cls, tower_instance, data):
try:
credential_type_name = tower_instance.get_credential_type_by_id(data.get('credential_type')).name
credential_type_name = ''.join(credential_type_name.split())
credential_type = f'{credential_type_name}Credential'
credential_type_obj = getattr(importlib.import_module('towerlib.entities.credential'), credential_type)
credential = credential_type_obj(tower_instance, data)
except Exception: # pylint: disable=broad-except
LOGGER.exception(
'Could not dynamically load credential with type : "%s", trying a generic one.', credential_type
)
credential = GenericCredential(tower_instance, data)
return credential
I assume this pytest does some sort of caching or other behavior that doesn't get updated when the module dynamically adds attributes to itself and reimports itself. Does anyone know if that's how pytest works, and if that's configurable with a flag or decorator on my test?
The function that I'm trying to test works correctly when run outside of pytest, but when running it in a pytest function, I'm getting the "Could not dynamically load credential with type" exception.
Running in the REPL:
>>> e2e.prepare_and_run_e2e(["my_device"], None, None, False, True, True, False, True)
^this returns a full results set with no errors.
My pytest function:
#pytest.mark.parametrize("device", get_all_e2e_devices())
def test_e2e(device):
results = e2e.prepare_and_run_e2e([device], None, None, False, True, True, False, True)
...
Ends up with:
ERROR credentials:credential.py:189 Could not dynamically load credential with type : "NetworkCredential", trying a generic one.
Traceback (most recent call last):
File "{my local path}\.venv\lib\site-packages\towerlib\entities\credential.py", line 186, in __new__
credential_type_obj = getattr(importlib.import_module('towerlib.entities.credential'), credential_type)
AttributeError: module 'towerlib.entities.credential' has no attribute 'NetworkCredential'
I tried running pytest with -p no:cacheprovider, but that gives me the same result.
I don't want to patch/mock around the Credential because this is a special test in my application, and I want the test to actually get the towerlib Credential and talk to an external system. I also don't have the ability to change the somewhat confusing approach that towerlib is taking, since it's an external library, and I need to share this test across a team that will just be using the standard towerlib.

serverless starting error: ModuleNotFoundError: No module named 'handler' [duplicate]

I have a deployment package in the following structure:
my-project.zip
--- my-project.py
------ lambda_handler()
Then I define the handler path in configuration file
my-project.lambda_handler
Get the error:
'handler' missing on module
Can not understand that

There are some issues occurring this error.
Issue#1:
The very first issue you’re gonna run into is if you name the file incorrectly, you get this error:
Unable to import module 'lambda_function': No module named lambda_function
If you name the function incorrectly you get this error:
Handler 'handler' missing on module 'lambda_function_file': 'module'
object has no attribute 'handler'
On the dashboard, make sure the handler field is entered as function_filename.actual_function_name and make sure they match up in your deployment package.
If only the messages were a bit more instructive that would have been a simpler step.
Resource Link:
No lambda_function?
Issue#2:
adrian_praja has solved the issue in aws forum. He answered the following
I belive your index.js should contain
exports.createThumbnailHandler = function(event, context) {}
Issue#3:
Solution: Correctly specify the method call
This happens when the specification of the method called by node.js is incorrect in Lambda's setting.
Please review the specification of the method to call.
In the case of the above error message, I attempted to call the handler method of index.js, but the corresponding method could not be found.
The processing to call is set with "Handler" on the configuration tab.
Below is an example of setting to call the handler method of index.js.
Resource Link:
http://qiita.com/kazuqqfp/items/ac8d93918d0030b31aad
AWS Lambda Function is returning Handler 'handler' missing on module 'index'

I had this issue and had to make sure I had a function called handler in my file, e.g.:
# this just takes whatever is sent to the api gateway and sends it back
def handler(event, context):
try:
return response(event, 200)
except Exception as e:
return response('Error' + e.message, 400)
def response(message, status_code):
return message

CherryPy: what is the difference between `error_page.default` vs. `error_page.404` config settings?

Let's say I want to display my own 404 & 500 pages, I've found 2 possibilities so far:
1: Using cherrypy.config.update
def error_page_404(status, message, traceback, version):
return ('Error 404 Page not found')
def error_page_500(status, message, traceback, version):
return ('Error:')
cherrypy.config.update({'error_page.404': error_page_404, 'error_page.500': error_page_500})
Using _cp_config:
from cherrypy import _cperror
def handle_error():
cherrypy.response.status = 500
cherrypy.log("handle_error() called. Alarm!", "WEBAPP")
cherrypy.response.body = ['Sorry, an error occured. The admin has been notified']
error = _cperror.format_exc()
def error_page(status, message, traceback, version):
cherrypy.log("error_page() called. Probably not very important.", "WEBAPP")
return "Sorry, an error occured."
class Root:
_cp_config = {
'error_page.default': error_page,
'request.error_response': handle_error
}
but is there a difference or a suggestion which is preferable to use?

request.error_response allows you to set a handler for processing of some unexpected errors, like your own exceptions raised from HTTP handlers.
The callable that you'll set for this option will receive no arguments at all and you'll have to inspect sys.exc_info() for the details, to find out what happened.
You'll also have to set cherrypy.response.status and cherrypy.response.body by yourself, explicitly in your error handler.
If you want to modify the error response for HTTP error codes (when instances of cherrypy.HTTPError are raised, like raise cherrypy.NotFound), you can use error_page.default (catch-all) or error_page.404 (error-specific) for handling those errors.
error_page options support both file path and callable values. In case of using a file path, the HTML template file can use the following substitution patterns: %(status)s, %(message)s, %(traceback)s, and %(version)s.
If you opt-in to using a function, it'll receive those as arguments (callback(status, message, traceback, version)). The return value of this callable is then used HTTP response payload.
As you can see, these approaches have different implications and different levels of flexibility and usability. Choose whatever works for you. Internally, the default request.error_response uses error_page settings to figure out what to return. So if you redefine request.error_response, it'll not use error_page.* settings unless you explicitly make it do so.
See the docstring with some explanation here.

Passing a list as a url value to urlopen

Motivation
Motivated by this problem - the OP was using urlopen() and accidentally passed a sys.argv list instead of a string as a url. This error message was thrown:
AttributeError: 'list' object has no attribute 'timeout'
Because of the way urlopen was written, the error message itself and the traceback is not very informative and may be difficult to understand especially for a Python newcomer:
Traceback (most recent call last):
File "test.py", line 15, in <module>
get_category_links(sys.argv)
File "test.py", line 10, in get_category_links
response = urlopen(url)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 420, in open
req.timeout = timeout
AttributeError: 'list' object has no attribute 'timeout'
Problem
Here is the shortened code I'm working with:
try:
from urllib.request import urlopen
except ImportError:
from urllib2 import urlopen
import sys
def get_category_links(url):
response = urlopen(url)
# do smth with response
print(response)
get_category_links(sys.argv)
I'm trying to think whether this kind of an error can be caught statically with either smart IDEs like PyCharm, static code analysis tools like flake8 or pylint, or with language features like type annotations.
But, I'm failing to detect the problem:
it is probably too specific for flake8 and pylint to catch - they don't warn about the problem
PyCharm does not warn about sys.argv being passed into urlopen, even though, if you "jump to source" of sys.argv it is defined as:
argv = [] # real value of type <class 'list'> skipped
if I annotate the function parameter as a string and pass sys.argv, no warnings as well:
def get_category_links(url: str) -> None:
response = urlopen(url)
# do smth with response
get_category_links(sys.argv)
Question
Is it possible to catch this problem statically (without actually executing the code)?

Instead of keeping it editor specific, you can use mypy to analyze your code. This way it will run on all dev environments instead of just for those who use PyCharm.
from urllib.request import urlopen
import sys
def get_category_links(url: str) -> None:
response = urlopen(url)
# do smth with response
get_category_links(sys.argv)
response = urlopen(sys.argv)
The issues pointed out by mypy for the above code:
error: Argument 1 to "get_category_links" has incompatible type List[str]; expected "str"
error: Argument 1 to "urlopen" has incompatible type List[str]; expected "Union[str, Request]"
Mypy here can guess the type of sys.argv because of its definition in its stub file. Right now some standard library modules are still missing from typeshed though, so you will have to either contribute them or ignore the errors related till they get added :-).
When to run mypy?
To catch such errors you can run mypy on the files with annotations with your tests in your CI tool. Running it on all files in project may take some time, for a small project it is your choice.
Add a pre-commit hook that runs mypy on staged files and points out issues right away(could be a little annoying to the dev if it takes a while).

Firstly, you need to check whether the url type is string or not and if string then check for ValueError exception(Valid url)
import sys
from urllib2 import urlopen
def get_category_links(url):
if type(url) != type(""): #Check if url is string or not
print "Please give string url"
return
try:
response = urlopen(url)
# do smth with response
print(response)
except ValueError: #If url is string but invalid
print "Bad URL"
get_category_links(sys.argv)

Searching for equivalent of FileNotFoundError in Python 2

I created a class named Options. It works fine but not not with Python 2.
And I want it to work on both Python 2 and 3.
The problem is identified: FileNotFoundError doesn t exist in Python 2.
But if I use IOError it doesn t work in Python 3
Changed in version 3.3: EnvironmentError, IOError, WindowsError, VMSError, socket.error, select.error and mmap.error have been merged into OSError.
What should I do ???(Please do not discuss my choice of portability, I have reasons.)
Here s the code:
#!/usr/bin/python
#-*-coding:utf-8*
#option_controller.py
#Walle Cyril
#25/01/2014
import json
import os
class Options():
"""Options is a class designed to read, add and change informations in a JSON file with a dictionnary in it.
The entire object works even if the file is missing since it re-creates it.
If present it must respect the JSON format: e.g. keys must be strings and so on.
If something corrupted the file, just destroy the file or call read_file method to remake it."""
def __init__(self,directory_name="Cache",file_name="options.json",imported_default_values=None):
#json file
self.option_file_path=os.path.join(directory_name,file_name)
self.directory_name=directory_name
self.file_name=file_name
#self.parameters_json_file={'sort_keys':True, 'indent':4, 'separators':(',',':')}
#the default data
if imported_default_values is None:
DEFAULT_INDENT = 2
self.default_values={\
"translate_html_level": 1,\
"indent_size":DEFAULT_INDENT,\
"document_title":"Titre"}
else:
self.default_values=imported_default_values
def read_file(self,read_this_key_only=False):
"""returns the value for the given key or a dictionary if the key is not given.
returns None if it s impossible"""
try:
text_in_file=open(self.option_file_path,'r').read()
except FileNotFoundError:#not 2.X compatible
text_in_file=""#if the file is not there we re-make one with default values
if text_in_file=="":#same if the file is empty
self.__insert_all_default_values()
text_in_file=open(self.option_file_path,'r').read()
try:
option_dict=json.loads(text_in_file)
except ValueError:
#if the json file is broken we re-make one with default values
self.__insert_all_default_values()
text_in_file=open(self.option_file_path,'r').read()
option_dict=json.loads(text_in_file)
if read_this_key_only:
if read_this_key_only in option_dict:
return option_dict[read_this_key_only]#
else:
#if the value is not there it should be written for the next time
if read_this_key_only in self.default_values:
self.add_option_to_file(read_this_key_only,self.default_values[read_this_key_only])
return self.default_values[read_this_key_only]
else:
#impossible because there is not default value so the value isn t meant to be here
return None
else:
return option_dict
def add_option_to_file(self,key,value):#or update
"""Adds or updates an option(key and value) to the json file if the option exists in the default_values of the object."""
option_dict=self.read_file()
if key in self.default_values:
option_dict[key]=value
open(self.option_file_path,'w').write(\
json.dumps(option_dict,sort_keys=True, indent=4, separators=(',',':')))
def __insert_all_default_values(self):
"""Recreate json file with default values.
called if the document is empty or non-existing or corrupted."""
try:
open(self.option_file_path,'w').write(\
json.dumps(self.default_values,sort_keys=True, indent=4, separators=(',',':')))
except FileNotFoundError:
os.mkdir(self.directory_name)#Create the directory
if os.path.isdir(self.directory_name):#succes
self.__insert_all_default_values()
else:
print("Impossible to write in %s and file %s not found" % (os.getcwd(),self.option_file_path))
#demo
if __name__ == '__main__':
option_file_object=Options()
print(option_file_object.__doc__)
print(option_file_object.read_file())
option_file_object.add_option_to_file("","test")#this should have no effect
option_file_object.add_option_to_file("translate_html_level","0")#this should have an effect
print("value of translate_html_level:",option_file_object.read_file("translate_html_level"))
print(option_file_object.read_file())

If FileNotFoundError isn't there, define it:
try:
FileNotFoundError
except NameError:
FileNotFoundError = IOError
Now you can catch FileNotFoundError in Python 2 since it's really IOError.
Be careful though, IOError has other meanings. In particular, any message should probably say "file could not be read" rather than "file not found."

You can use the base class exception EnvironmentError and use the 'errno' attribute to figure out which exception was raised:
from __future__ import print_function
import os
import errno
try:
open('no file of this name') # generate 'file not found error'
except EnvironmentError as e: # OSError or IOError...
print(os.strerror(e.errno))
Or just use IOError in the same way:
try:
open('/Users/test/Documents/test') # will be a permission error
except IOError as e:
print(os.strerror(e.errno))
That works on Python 2 or Python 3.
Be careful not to compare against number values directly, because they can be different on different platforms. Instead, use the named constants in Python's standard library errno module which will use the correct values for the run-time platform.

The Python 2 / 3 compatible way to except a FileNotFoundError is this:
import errno
try:
with open('some_file_that_does_not_exist', 'r'):
pass
except EnvironmentError as e:
if e.errno != errno.ENOENT:
raise
Other answers are close, but don't re-raise if the error number doesn't match.
Using IOError is fine for most cases, but for some reason os.listdir() and friends raise OSError instead on Python 2. Since IOError inherits from OSError it's fine to just always catch OSError and check the error number.
Edit: The previous sentence is only true on Python 3. To be cross compatible, instead catch EnvironmentError and check the error number.

For what it's worth, although the IOError is hardly mentioned in Python 3's official document and does not even showed up in its official Exception hierarchy, it is still there, and it is the parent class of FileNotFoundError in Python 3. See python3 -c "print(isinstance(FileNotFoundError(), IOError))" giving you a True. Therefore, you can technically write your code in this way, which works for both Python 2 and Python 3.
try:
content = open("somefile.txt").read()
except IOError: # Works in both Python 2 & 3
print("Oops, we can not read this file")
It might be "good enough" in many cases. Although in general, it is not recommended to rely on an undocumented behavior. So, I'm not really suggesting this approach. I personally use Kindall's answer.