We Keep Coding

Why is os.remove None after executing pytest?

I create a LazyLoader class which downloads files from S3 to the local file system, but only if they were not downloaded before. This class deletes the stuff it downloaded once it gets destroyed:
def __del__(self):
"""Remove downloaded files when this object is deleted."""
for downloaded_file in self.loaded_data.values():
os.remove(downloaded_file)
The tests pass, but after pytest tells me that the tests passed I get:
Exception ignored in: <bound method LazyLoader.__del__ of LazyLoader({})>
Traceback (most recent call last):
File "my_lazy_loader.py", line 47, in __del__
TypeError: 'NoneType' object is not callable
Line 47 is os.remove(downloaded_file). So os.remove is None evaluates to True. Why? How can I fix that?
If I move the code in __del__ to a method clean(), I don't have that problem.

https://docs.python.org/3/reference/datamodel.html#object.del
'del()' can be executed during interpreter shutdown. As a consequence, the global variables it needs to access (including other modules) may already have been deleted or set to None. Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the del() method is called.

If it's just unit tests, use tearDown method.
If the problem occurs when running your application and you want to do a cleanup at the end of the program, consider using atexit.register.
You might also use tempfile module for temporary files automatically removed when closed.
If your use-case allows that, turn LazyLoader into a context manager.
If none of the above applies, simply call clean() explicitly. It will follow explicit is better than implicit rule.

Using a bottle API on python causes an error when passing arguments into functions

I'm writing an API in python using the bottle library but I'm having an issue.
When I run the following code:
from bottle import route, run
apiArray = ["key0","key1","key2"]
#route('/<userApi>/')
def hello(userApi,apiArray):
for item in apiArray:
if item == userApi:
return {True}
return {False}
run(host='localhost', port=8080, debug=True)
When I run that code though I get the following error:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/bottle.py", line 862, in _handle
return route.call(**args)
File "/usr/local/lib/python2.7/dist-packages/bottle.py", line 1732, in wrapper
rv = callback(*a, **ka)
TypeError: hello() takes exactly 2 arguments (1 given)
Does anybody know what I'm doing wrong and why I am getting this error?

Why are you passing in apiArray if it's a variable already declared?? The function takes two variables but that route will only ever receive one. Remove this and it should work. It's in context so it's not needed but if you needed to pass it in as an argument, you should add it to the route. Otherwise it would never be set regardless.
EDIT: quick context tutorial
There are countless resources online to understand variable context in Python so I'm going to give a condensed explanation using examples.
Let's start with the explanation of context in Python. Context (or scope) is defined as the "space" where a variable or function is available for use. If a variable is defined in a given context, then any other variable or function may call on it afterwards. For example:
foo = 3
print foo
This would print 3. However, this would fail
print foo
foo = 3
The reason being that although foo is in the right context, it has not yet been defined.
Now about your question which was something like "if the variables are global then what's the point of having a function", the second part of Python context is that nested functions inherit the previous context. So if I did:
foo = 3
def print_foo():
print foo
print_foo()
Then this would print 3 since foo is in the same context as the function. Where would you use this?
This extends to more nesting of contexts. You could have functions within functions and still at each level the variables would be available to that level and below.
I hope this helps you understand context a little better however I oversimplified and excluded many important details so please take the time to read up on more Python context tutorials.

How to add a custom type to dill's pickleable types

I'm trying to serialize some code I did not write and cannot modify that needs to be pickled/dilled. The script contains a mongodb collection object---it isn't actually used later, but dilling it is throwing an error. When I try dilling it, I receive the error:
Collection object is not callable. If you meant to call __getnewargs__ method on a 'Database' object it is failing because no such method exists.
I see code here that is enumerating the accepted types:
https://github.com/uqfoundation/dill/blob/master/dill/_objects.py (lines 132-190) and my suspicion is this is where I might change something to allow a new type.
However, it's not clear to me what the intended interface is for adding a custom type. (Or maybe for pickling everything except that, is that possible or easier?)

No, the dill._objects module is just a list of types dill can and cannot pickle. Adding to that will just make dill think it can do more, while remaining the same functionally.
If you want to add a pickler, use dill.register (usally as a decorator). It takes a function which does the breaking down. E.g. given an unpicklable class:
class A:
def __init__(self, a):
self.a = a
def __reduce__(self):
raise GoAwayError()
Trying to pickle an instance of A will give you:
Traceback (most recent call last):
File "d.py", line 9, in <module>
dill.dumps(A(1))
File "/home/matthew/GitHub/dill/dill/dill.py", line 192, in dumps
dump(obj, file, protocol, byref, fmode)#, strictio)
File "/home/matthew/GitHub/dill/dill/dill.py", line 182, in dump
pik.dump(obj)
File "/usr/lib/python3.4/pickle.py", line 410, in dump
self.save(obj)
File "/usr/lib/python3.4/pickle.py", line 497, in save
rv = reduce(self.proto)
File "d.py", line 7, in __reduce__
raise GoAwayError()
NameError: name 'GoAwayError' is not defined
You can define a pickler like:
def recreate_A(a):
return A(a)
#dill.register(A)
def save_A(pickler, obj):
pickler.save_reduce(recreate_A, (obj.a,), obj=obj)
recreate_A is the function used for reconstruction, and (obj.a,) is a tuple of args which will be passed to your reconstructer function when loading.
This is probably the most flexible way of doing it, as you can use any function for recreate_A, including A.__init__ if you need to, but as you are trying to pickle a more complex type, you may need to do pre/post-processing. The functionality for skipping objects is still in the works, so you'll have to wait if you want to do it that way. If you want to achieve the same effect, you could just define recreate_A to return None, and take no args.

Python 2.7 mock/patch: understanding assert_called_XYZ()

I'm relatively new to Python and unit testing in Python. From the Java world I know the concept of mocking but it seem to be much different from what I can see in Python.
I found this guide, which I found very helpful: http://www.voidspace.org.uk/python/mock/index.html
But as I wrote my (a bit more complex) tests with mocked out dependencies I noticed a strage behavior.
I decided to create a reduced, simple example which also does not work as I expect it.
Take a look at this, the result and my expectation I have added as comments:
import unittest
from mock import patch, Mock, MagicMock
class BasicTest(unittest.TestCase):
#patch("StringIO.StringIO")
def testSomethingNotWorkingAsExpected(self, StringIOMock):
StringIOMock.assert_called_once() # asserts, but why?
#patch("StringIO.StringIO")
def testSomethingSomehowWorking(self, StringIOMock):
# self.instantiateStringIO() # intentionally commented out
assert StringIOMock.called # does not assert (leading to failure of this test); as expected. If the above line is not commented, this asserts as expected.
def instantiateStringIO(self):
import StringIO
StringIO.StringIO()
Why is assert_called_once() asserting the instantiation of StringIO even it has not been instantiated yet?
And why does assert ClassMock.called bring the expected results?
Using assert not ... to assert a method has not been called I found here: Assert a function/method was not called using Mock. I inverted this pattern in my case by omitting the not.
Somewhere I found the pattern ClassMock.return_value to reference an instance. But I understand this as a way to manupulate the instance of a Mock before it will be called, not as a way to access the instance that might an underliing code have internally created. Or am I wrong?
My environment:
Python 2.7.3
mock 0.8.8
Fedora 19
Probably my understanding of the mock/patch thing is wrong. Could please someone aditionally explain what a class mock does and how it works?
Edit1: Added output
... and added paraphrase in parens to comment in testSomethingSomehowWorking
This is the output:
.F
======================================================================
FAIL: testSomethingSomehowWorking (test_test.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mock.py", line 1224, in patched
return func(*args, **keywargs)
File "test_test.py", line 15, in testSomethingSomehowWorking
assert StringIOMock.called # does not assert; as expected
AssertionError
----------------------------------------------------------------------
Ran 2 tests in 0.001s
FAILED (failures=1)

The method assert_called_once does not exist and it does not perform an assertion. It's no different from writing StringIOMock.assert_foo_bar_does_not_exist() or any other method. The mock library doesn't check whether the method called on the mock actually exists.
If you use assert_called_once_with then it fails as expected.
You can use the spec parameter to raise an error when you call a non-existent method:
#patch("StringIO.StringIO", spec=StringIO.StringIO)
def testSomethingNotWorkingAsExpected(self, StringIOMock):
StringIOMock.assert_called_once() # will fail as the method doesn't exist

Problem grabbing parameters using execfile within a class

I have a problem similar to the first problem in this question, which as far as I can see went unanswered.
I have a file "config.py" which contains a lot of parameters to be used by a class (this config.py file will change), however I can't get these to propagate into the class via execfile.
In an example piece of code:
class Class():
def __init__(self):
execfile("config.py")
print x
# config.py
x = "foo"
>>> t = Class()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in __init__
NameError: global name 'x' is not defined
Any help welcome, or any better methods of retrieving parameters from a file to be used in a class.
Many Thanks.

I don't get what you're trying to do (but i don't like it, and this is just me) but to fix your problem do (test in python2.6):
class Class():
def __init__(self):
execfile('config.py', locals()) # Not recommanded, maybe you want globals().
print x
But from the doc:
Note
The default locals act as described
for function locals() below:
modifications to the default locals
dictionary should not be attempted.
Pass an explicit locals dictionary if
you need to see effects of the code on
locals after function execfile()
returns. execfile() cannot be used
reliably to modify a function’s
locals.
and about :
Any help welcome, or any better
methods of retrieving parameters from
a file to be used in a class.
You can use import.

Even though it might be convenient to keep configuration settings in a Python file I would recommend against it. I think it opens up a whole set of problems that you don't really want to have do deal with. Anything could be placed in your configuration file, including malicious code.
I would use either the json module or the ConfigParser module to hold my configuration.
If you have trouble choosing between those two I would recommend the json module. Json is a simple yet flexible format for structured data.