We Keep Coding

Pytest: How to pass method which returns list as an argument in parametrized test

I have a question regarding parametrizing the test method with another method that returns the list of test data that I want to use in my test:
When I execute code in this way:
class Test:
#pytest.mark.parametrize("country_code", get_country_code_list())
def test_display_country_code(self, country_code):
print(country_code)
#classmethod
def get_country_code_list(cls) -> list:
return [1, 2, 3]
I get error: Unresolved referency: get_country_code_list. It doesn't make a difference if get_country_code_list method is a static method, class method or self.
But if I put the method get_country_code_list() above the test method, I don't get this error.
Does the order of test methods make a difference in Python?

Yes, the order in which you do things is important.
Functions are just like variables in that manner.
Working Example Variables
x = 3
y = 5
z = x + y
Works perfectly fine, because everything is done according to order.
Broken Example Variables
x = 3
z = x + y
y = 5
Doesn't work, of course, because y is neither declared nor defined when y is needed for the initialization of z.
Works just the same with functions
def bar():
return foobar()
def foo():
return bar()
foo()
def foobar()
return 5
Function foo can call function bar perfectly fine, but bar can't call foobar, because foobar is not defined yet at the execution point of foo().

This isn't a test-specific issue only.
You need to understand that #pytest.mark.parametrize is a decorator, a syntactic sugar of a class / method. When you pass an argument in a class / method, it expects the argument to be defined. Hence why this works:
def display(arg):
return print(arg)
word = "Hello World!"
display(word)
while this does not:
def display(arg):
return print(arg)
display(word)
word = "Hello World!"
Here's an example of a class-based decorator:
class MyDecorator:
def __init__(self, decor_arg):
self.decor_arg = decor_arg
def __call__(self, fn):
def wrapper(fn_arg):
return fn_arg
return self.decor_arg
def message(arg):
return f"The parameter passed is {arg}"
#MyDecorator(message)
def display(arg):
return arg
print(display("Hello World!"))
The print result:
The parameter passed is Hello World!
Given the explanation above, I'm sure you can see why the method message needs to be placed before display. If your editor has IntelliSense, changing the order of the two methods will display an error outline on #MyDecorator(concat) with the message "undefined name 'message'" or something similar.

How to get builtin_function_or_method objects copied in ram without wrapping? [duplicate]

I would like to make a deepcopy of a function in Python. The copy module is not helpful, according to the documentation, which says:
This module does not copy types like module, method, stack trace, stack frame, file,
socket, window, array, or any similar types. It does “copy” functions and classes (shallow
and deeply), by returning the original object unchanged; this is compatible with the way
these are treated by the pickle module.
My goal is to have two functions with the same implementation but with different docstrings.
def A():
"""A"""
pass
B = make_a_deepcopy_of(A)
B.__doc__ = """B"""
So how can this be done?

The FunctionType constructor is used to make a deep copy of a function.
import types
def copy_func(f, name=None):
return types.FunctionType(f.func_code, f.func_globals, name or f.func_name,
f.func_defaults, f.func_closure)
def A():
"""A"""
pass
B = copy_func(A, "B")
B.__doc__ = """B"""

My goal is to have two functions with the same implementation but with different docstrings.
Most users will do this, say the original function is in old_module.py:
def implementation(arg1, arg2):
"""this is a killer function"""
and in new_module.py
from old_module import implementation as _implementation
def implementation(arg1, arg2):
"""a different docstring"""
return _implementation(arg1, arg2)
This is the most straightforward way to reuse functionality. It is easy to read and understand the intent.
Nevertheless, perhaps you have a good reason for your main question:
How can I make a deepcopy of a function in Python?
To keep this compatible with Python 2 and 3, I recommend using the function's special __dunder__ attributes. For example:
import types
def copy_func(f, name=None):
'''
return a function with same code, globals, defaults, closure, and
name (or provide a new name)
'''
fn = types.FunctionType(f.__code__, f.__globals__, name or f.__name__,
f.__defaults__, f.__closure__)
# in case f was given attrs (note this dict is a shallow copy):
fn.__dict__.update(f.__dict__)
return fn
And here's an example usage:
def main():
from logging import getLogger as _getLogger # pyflakes:ignore, must copy
getLogger = copy_func(_getLogger)
getLogger.__doc__ += '\n This function is from the Std Lib logging module.\n '
assert getLogger.__doc__ is not _getLogger.__doc__
assert getLogger.__doc__ != _getLogger.__doc__
A commenter says:
This can’t work for built‑in functions
Well I wouldn't do this for a built-in function. I have very little reason to do this for functions written in pure Python, and my suspicion is that if you are doing this, you're probably doing something very wrong (though I could be wrong here).
If you want a function that does what a builtin function does, and reuses the implementation, like a copy would, then you should wrap the function with another function, e.g.:
_sum = sum
def sum(iterable, start=0):
"""sum function that works like the regular sum function, but noisy"""
print('calling the sum function')
return _sum(iterable, start)

from functools import partial
def a():
"""Returns 1"""
return 1
b = partial(a)
b.__doc__ = """Returns 1, OR DOES IT!"""
print help(a)
print help(b)
Wrap it as a partial?

def A():
"""A"""
pass
def B():
"""B"""
return A()

The others answers do not allow for serialization with pickle. Here a code that I am using to clone a function and allow for serialization for python3:
import pickle
import dill
import types
def foo():
print ('a')
oldCode=foo.__code__
name='IAmFooCopied'
newCode= types.CodeType(
oldCode.co_argcount, # integer
oldCode.co_kwonlyargcount, # integer
oldCode.co_nlocals, # integer
oldCode.co_stacksize, # integer
oldCode.co_flags, # integer
oldCode.co_code, # bytes
oldCode.co_consts, # tuple
oldCode.co_names, # tuple
oldCode.co_varnames, # tuple
oldCode.co_filename, # string
name, # string
oldCode.co_firstlineno, # integer
oldCode.co_lnotab, # bytes
oldCode.co_freevars, # tuple
oldCode.co_cellvars # tuple
)
IAmFooCopied=types.FunctionType(newCode, foo.__globals__, name,foo.__defaults__ , foo.__closure__)
IAmFooCopied.__qualname__= name
print ( 'printing foo and the copy', IAmFooCopied, foo )
print ( 'dill output: ', dill.dumps(IAmFooCopied ))
print ( 'pickle Output: ', pickle.dumps (IAmFooCopied) )
Output:
printing foo and the copy <function IAmFooCopied at 0x7f8a6a8159d8> <function foo at 0x7f8a6b5f5268>
dill output: b'\x80\x03cdill._dill\n_create_function\nq\x00(cdill._dill\n_load_type\nq\x01X\x08\x00\x00\x00CodeTypeq\x02\x85q\x03Rq\x04(K\x00K\x00K\x00K\x02KCC\x0ct\x00d\x01\x83\x01\x01\x00d\x00S\x00q\x05NX\x01\x00\x00\x00aq\x06\x86q\x07X\x05\x00\x00\x00printq\x08\x85q\t)X\x10\x00\x00\x00testCloneFunc.pyq\nX\x0c\x00\x00\x00IAmFooCopiedq\x0bK\x05C\x02\x00\x01q\x0c))tq\rRq\x0ec__builtin__\n__main__\nh\x0bNN}q\x0ftq\x10Rq\x11.'
pickle Output: b'\x80\x03c__main__\nIAmFooCopied\nq\x00.'
You may encounter problem with the qualname attribute if you try this snippet with class methods (I think pickle should fail to find your function). I never tried it, however it should be easily fixable. Just check the doc about qualname

It's quite easy to do using lambda and rest parameters:
def my_copy(f):
# Create a lambda that mimics f
g = lambda *args: f(*args)
# Add any properties of f
t = list(filter(lambda prop: not ("__" in prop),dir(f)))
i = 0
while i < len(t):
setattr(g,t[i],getattr(f,t[i]))
i += 1
return g
# Test
def sqr(x): return x*x
sqr.foo = 500
sqr_copy = my_copy(sqr)
print(sqr_copy(5)) # -> 25
print(sqr_copy(6)) # -> 36
print(sqr_copy.foo) # -> 500
print(sqr_copy == sqr) # -> False
Try it online!

put it in a function:
def makefunc( docstring ):
def f():
pass
f.__doc__ = docstring
return f
f = makefunc('I am f')
g = makefunc("I am f too")

Adjusted for python3
import types
def copy_func(f, name=None):
return types.FunctionType(f.__code__, f.__globals__, name or f.__name__,
f.__defaults__, f.__closure__)
def func1(x):
return 2*x
func2=copy_func(func1)
print(func2(7))

I've implemented a general-purpose function copy in haggis, a library which I wrote and maintain (available with pip but probably not conda). haggis.objects.copy_func makes a copy on which you can not only reassign the __doc__ attribute, but also modify the module and __globals__ attributes effectively.
from haggis.objects import copy_func
def a(*args, **kwargs):
"""A docstring"""
a2 = copy_func(a)
a2.__doc__ = """Another docstring"""
>>> a == a2
False
>>> a.__code__ == a2.__code__
True
>>> a.__doc__
'A docstring'
>>> a2.__doc__
'Another docstring'

Python - If a function is a first class object, can a function have a method?

I have a class which maintains a list of functions. These functions are just objects sitting in a queue and every so often the class pops one off and executes it. However, there are times when I would like to print out this list, and I'm imagining code as follows:
for function in self.control_queue:
print function.summarize()
if function.ready():
function()
In other words, I would like to call methods called summarize() and ready(), that I want to define somewhere, on these function objects. Also, I would like to be able to toss anonymous functions on this queue - i.e., generate everything dynamically.

you can make it a class and define __call__
class MyClass():
def summarize(self):
#summarize stuff
pass
def ready(self):
#ready stuff
pass
def _call__(self):
#put the code here, for when you call myClass()
pass
How you run it:
function = MyClass()
print function.summarize()
if function.ready():
function()

You have a couple possible approaches.
You could add the definitions to functions.
def foo():
pass
# later..
foo.summarize = lambda: "To pair with bar"
foo.ready = lambda: True
You could create class objects to wrap the function operation.
class Func():
def summarize(self):
return "Function!"
def ready(self):
return self.ready
def __call__(self):
# Act as a function
Or you can have a function which checks the function label for these capabilities.
def summarize_func(func):
return func.__name__ # Or branch here on specific names/attributes
def ready_func(func):
return True # Or branch on names/attributes
Finally to accommodate anonymous functions you can check for prescience of these attributes and return optimistically if the attributes are absent. Then you can combine above approaches with something that will work on any function.
def summarize_func(func):
if hasattr(func, summarize):
return func.summarize()
else:
# Note this will just be '<lambda>' for anonymous funcs
return func.__name__
def ready_func(func):
if hasattr(func, ready):
return func.ready()
else:
return True

One option is to implement function as a class instance:
class Function(object):
def summarize(self): pass # some relevant code here
def __call__(self): pass # and there
and use it later with
function = Function()
With __call__ magic method implemented, this function becomes a callable object.
For sure, you can assign attributes to functions, but it is rather obscure and conterintuitive:
>>> def summ(a): return sum(a)
...
>>> def function(a): return a
...
>>> function.sum=summ
>>> function.sum([1,2,3])
6

python lazy variables? or, delayed expensive computation

I have a set of arrays that are very large and expensive to compute, and not all will necessarily be needed by my code on any given run. I would like to make their declaration optional, but ideally without having to rewrite my whole code.
Example of how it is now:
x = function_that_generates_huge_array_slowly(0)
y = function_that_generates_huge_array_slowly(1)
Example of what I'd like to do:
x = lambda: function_that_generates_huge_array_slowly(0)
y = lambda: function_that_generates_huge_array_slowly(1)
z = x * 5 # this doesn't work because lambda is a function
# is there something that would make this line behave like
# z = x() * 5?
g = x * 6
While using lambda as above achieves one of the desired effects - computation of the array is delayed until it is needed - if you use the variable "x" more than once, it has to be computed each time. I'd like to compute it only once.
EDIT:
After some additional searching, it looks like it is possible to do what I want (approximately) with "lazy" attributes in a class (e.g. http://code.activestate.com/recipes/131495-lazy-attributes/). I don't suppose there's any way to do something similar without making a separate class?
EDIT2: I'm trying to implement some of the solutions, but I'm running in to an issue because I don't understand the difference between:
class sample(object):
def __init__(self):
class one(object):
def __get__(self, obj, type=None):
print "computing ..."
obj.one = 1
return 1
self.one = one()
and
class sample(object):
class one(object):
def __get__(self, obj, type=None):
print "computing ... "
obj.one = 1
return 1
one = one()
I think some variation on these is what I'm looking for, since the expensive variables are intended to be part of a class.

The first half of your problem (reusing the value) is easily solved:
class LazyWrapper(object):
def __init__(self, func):
self.func = func
self.value = None
def __call__(self):
if self.value is None:
self.value = self.func()
return self.value
lazy_wrapper = LazyWrapper(lambda: function_that_generates_huge_array_slowly(0))
But you still have to use it as lazy_wrapper() not lazy_wrapper.
If you're going to be accessing some of the variables many times, it may be faster to use:
class LazyWrapper(object):
def __init__(self, func):
self.func = func
def __call__(self):
try:
return self.value
except AttributeError:
self.value = self.func()
return self.value
Which will make the first call slower and subsequent uses faster.
Edit: I see you found a similar solution that requires you to use attributes on a class. Either way requires you rewrite every lazy variable access, so just pick whichever you like.
Edit 2: You can also do:
class YourClass(object)
def __init__(self, func):
self.func = func
#property
def x(self):
try:
return self.value
except AttributeError:
self.value = self.func()
return self.value
If you want to access x as an instance attribute. No additional class is needed. If you don't want to change the class signature (by making it require func), you can hard code the function call into the property.

Writing a class is more robust, but optimizing for simplicity (which I think you are asking for), I came up with the following solution:
cache = {}
def expensive_calc(factor):
print 'calculating...'
return [1, 2, 3] * factor
def lookup(name):
return ( cache[name] if name in cache
else cache.setdefault(name, expensive_calc(2)) )
print 'run one'
print lookup('x') * 2
print 'run two'
print lookup('x') * 2

Python 3.2 and greater implement an LRU algorithm in the functools module to handle simple cases of caching/memoization:
import functools
#functools.lru_cache(maxsize=128) #cache at most 128 items
def f(x):
print("I'm being called with %r" % x)
return x + 1
z = f(9) + f(9)**2

You can't make a simple name, like x, to really evaluate lazily. A name is just an entry in a hash table (e.g. in that which locals() or globals() return). Unless you patch access methods of these system tables, you cannot attach execution of your code to simple name resolution.
But you can wrap functions in caching wrappers in different ways.
This is an OO way:
class CachedSlowCalculation(object):
cache = {} # our results
def __init__(self, func):
self.func = func
def __call__(self, param):
already_known = self.cache.get(param, None)
if already_known:
return already_known
value = self.func(param)
self.cache[param] = value
return value
calc = CachedSlowCalculation(function_that_generates_huge_array_slowly)
z = calc(1) + calc(1)**2 # only calculates things once
This is a classless way:
def cached(func):
func.__cache = {} # we can attach attrs to objects, functions are objects
def wrapped(param):
cache = func.__cache
already_known = cache.get(param, None)
if already_known:
return already_known
value = func(param)
cache[param] = value
return value
return wrapped
#cached
def f(x):
print "I'm being called with %r" % x
return x + 1
z = f(9) + f(9)**2 # see f called only once
In real world you'll add some logic to keep the cache to a reasonable size, possibly using a LRU algorithm.

To me, it seems that the proper solution for your problem is subclassing a dict and using it.
class LazyDict(dict):
def __init__(self, lazy_variables):
self.lazy_vars = lazy_variables
def __getitem__(self, key):
if key not in self and key in self.lazy_vars:
self[key] = self.lazy_vars[key]()
return super().__getitem__(key)
def generate_a():
print("generate var a lazily..")
return "<a_large_array>"
# You can add as many variables as you want here
lazy_vars = {'a': generate_a}
lazy = LazyDict(lazy_vars)
# retrieve the variable you need from `lazy`
a = lazy['a']
print("Got a:", a)
And you can actually evaluate a variable lazily if you use exec to run your code. The solution is just using a custom globals.
your_code = "print('inside exec');print(a)"
exec(your_code, lazy)
If you did your_code = open(your_file).read(), you could actually run your code and achieve what you want. But I think the more practical approach would be the former one.

Lambda function for classes in python?

There must be an easy way to do this, but somehow I can wrap my head around it. The best way I can describe what I want is a lambda function for a class. I have a library that expects as an argument an uninstantiated version of a class to work with. It then instantiates the class itself to work on. The problem is that I'd like to be able to dynamically create versions of the class, to pass to the library, but I can't figure out how to do it since the library expects an uninstantiated version. The code below describes the problem:
class Double:
def run(self,x):
return x*2
class Triple:
def run(self,x):
return x*3
class Multiply:
def __init__(self,mult):
self.mult = mult
def run(self,x):
return x*self.mult
class Library:
def __init__(self,c):
self.c = c()
def Op(self,val):
return self.c.run(val)
op1 = Double
op2 = Triple
#op3 = Multiply(5)
lib1 = Library(op1)
lib2 = Library(op2)
#lib3 = Library(op3)
print lib1.Op(2)
print lib2.Op(2)
#print lib3.Op(2)
I can't use the generic Multiply class, because I must instantiate it first which breaks the library "AttributeError: Multiply instance has no call method". Without changing the Library class, is there a way I can do this?

Does the library really specify that it wants an "uninitialized version" (i.e. a class reference)?
It looks to me as if the library actually wants an object factory. In that case, it's acceptable to type:
lib3 = Library(lambda: Multiply(5))
To understand how the lambda works, consider the following:
Multiply5 = lambda: Multiply(5)
assert Multiply5().run(3) == Multiply(5).run(3)

There's no need for lambda at all. lambda is just syntatic sugar to define a function and use it at the same time. Just like any lambda call can be replaced with an explicit def, we can solve your problem by creating a real class that meets your needs and returning it.
class Double:
def run(self,x):
return x*2
class Triple:
def run(self,x):
return x*3
def createMultiplier(n):
class Multiply:
def run(self,x):
return x*n
return Multiply
class Library:
def __init__(self,c):
self.c = c()
def Op(self,val):
return self.c.run(val)
op1 = Double
op2 = Triple
op3 = createMultiplier(5)
lib1 = Library(op1)
lib2 = Library(op2)
lib3 = Library(op3)
print lib1.Op(2)
print lib2.Op(2)
print lib3.Op(2)

This is sort of cheating, but you could give your Multiply class a __call__ method that returns itself:
class Multiply:
def __init__(self,mult):
self.mult = mult
def __call__(self):
return self
def run(self,x):
return x*self.mult
That way when the library calls c() it actually calls c.__call__() which returns the object you want.

def mult(x):
def f():
return Multiply(x)
return f
op3 = mult(5)
lib3 = Library(op3)
print lib3.Op(2)

If I understand your problem space correctly, you have a general interface that takes 1 argument which is called using the Library class. Unfortunately, rather than calling a function, Library assumes that the function is wrapped in a class with a run method.
You can certainly create these classes programatically. Classes may be returned by methods, and thanks to the concept of closures you should be able to wrap any function in a Class that meets your needs. Something like:
def make_op(f):
class MyOp(object):
def run(self, x):
return f(x)
return MyOp
op1 = make_op(lambda x: return x*2)
op2 = make_op(lambda x: return x*3)
def multiply_op(y):
return make_op(lambda x: return x*y)
op3 = multiply_op(3)
lib1 = Library(op1)
lib2 = Library(op2)
lib3 = Library(op3)
print( lib1.Op(2) )
print( lib2.Op(2) )
print( lib3.Op(2) )
That being said, changing Library to take a function and then providing functions is probably the stronger way to do this.

Since type is the default class of a python class object, and calling a class creates a new instance of that class, calling type with the correct arguments will result in a new class.
my_class = type("my_class", (object,), {"an_attribute": 1})
my_class now refers to a new class named "my_class", which is a subclass of object, with an attribute called "an_attribute", whose value is 1. Since methods are also just class attributes pointing to a function object, you can add them to the dictionary of attributes as well:
{"an_attribute": 1, "a_method": lambda self: print("Hello")}
This is how it works. I do not recommend doing it this way, unless you absolutely need to. In 99% of all cases, you don't. Refer to #Parker Coates' answer for the clean way to achieve your goal.