We Keep Coding

python call all functions with name patterns or other signature

I'm writing a feature generation class that can be extendable. For example in the following example any method that starts with generate is a feature generation method:
class FeatureGenerator:
def __init__(self):
self.generate_a()
self.generate_b()
def method_a(self): pass
def generate_a(self): do stuffs
def generate_b(self): do stuffs
I want to execute all methods with generate prefix within init. However, I don't want to add it manually every time I write a new method. One solution could be writing a decorator that will add it to a list and then execute all elements in the list within init. But I am not sure that's a good idea. Is there any pythonic way to do that?

Using the dir of the instance:
class FeatureGenerator:
def __init__(self):
for name in dir(self):
attr = getattr(self, name)
if callable(attr) and name.startswith("generate"):
attr()
def method_a(self): pass
def generate_a(self): print("gen a")
def generate_b(self): print("gen b")

Finding parameters of `__init__()` or parameters needed to construct an object in python

I have scenario where I am passing a file name and checking if it has argument start as constructor if it has then I have to create instance of that class.
Consider the example where I have a file named test.py which have three class namely A,B,C now only class A has start parameter others have other different parameter or extra parameter.
#test.py
class A:
def __init__(self, start=""):
pass
class B:
def __init__(self, randomKeyword, start=""):
pass
class C:
def __init__(self):
pass
Now I want to write a script which takes test.py as an argument and create instance of A. Till now my progress is
detail = importlib.util.spec_from_file_location('test.py', '/path/to/test.py')
module = importlib.util.module_from_spec(detail)
spec.loader.exec_module(mod)
Bacially I need to write a program which finds init argument of all class in file and create an instance of file with start as init argument.

As mentioned by #deceze it's not a good idea to instantiate a class on the basis of it's init parameter as we're not sure what is there. But it's possible to do it. So I am posting this answer just so that you know how it can be done.
#test.py
class A:
def __init__(self, start=""):
pass
class B:
def __init__(self, randomKeyword, start=""):
pass
class C:
def __init__(self):
pass
One of the possibility is
#init.py
import importlib.util
from inspect import getmembers, isclass, signature
detail = importlib.util.spec_from_file_location('test.py', '/path/to/test.py')
module = importlib.util.module_from_spec(detail)
spec.loader.exec_module(module)
for name, data in getmembers(mod, isclass):
cls = getattr(mod, name)
parameter = signature(cls.__init__).parameters.keys()
# parameter start
if len(parameter) == 2 and 'start' in parameter:
object = cls(start="Whatever you want")
Ofcourse it's not the best approach so more answer are welcome and if you are in this scenario consider #deceze comment and define a builder.

how to pickle class dynamically created from decorator in python 3.6

Disclaimer:
Before reading this post know that I am trying to do something that is unconventional in python. Since "Don't do x" is not an answer to "how do I do x?" let's assume there is a very good reason to do this, even though in most cases it would not be good practice.
The Question
Given I have a class that is dynamically created by applying a decorator to a function, how would I go about pickling an instance of said class?
For example, to set this up it might look like this:
import inspect
from functools import wraps
class BaseClass:
pass
def _make_method(func):
""" decorator for adding self as first argument to function """
#wraps(func)
def decorator(self, *args, **kwargs):
return func(*args, **kwargs)
# set signature to include self
sig = inspect.signature(decorator)
par = inspect.Parameter('self', 1)
new_params = tuple([par] + list(sig.parameters.values()))
new_sig = sig.replace(parameters=new_params,
return_annotation=sig.return_annotation)
decorator.__signature__ = new_sig
return decorator
def snake2camel(snake_str):
""" convert a snake_string to a CamelString """
return "".join(x.title() for x in snake_str.split('_'))
def make_class(func):
""" dynamically create a class setting the call method to function """
name = snake2camel(func.__name__) # get the name of the new class
method = _make_method(func)
cls = type(name, (BaseClass,), {'__call__': method})
return cls()
#make_class
def something(arg):
return arg
Now something is an instance of the dynamically created class Something.
type(something) # -> __main__.Something
isinstance(something, BaseClass) # -> True
which works fine, but when I try to pickle it (or use the multiprocessing module which uses pickle under the hood):
import pickle
pickle.dumps(something) # -> raises
it throws this error:
# PicklingError: Can't pickle <class '__main__.Something'>: attribute lookup Something on __main__ failed
So I thought I could redefine BaseClass to use a reduce method like so:
class BaseClass:
def __reduce__(self):
return make_class, (self.__call__.__func__,)
but then it throws the dreaded "not the same object" error:
# PicklingError: Can't pickle <function something at 0x7fe124cb2d08>: it's not the same object as __main__.something
How can I make this work without bringing in dependencies? I need to be able to pickle the something object so I can use it with the ProcessPoolExecutor class from the concurrent.futures module in python 3.6, so simply using dill or cloudpickle is probably not an option here.

Alternatives to decorators for saving metadata about classes

I'm writing a GUI library, and I'd like to let the programmer provide meta-information about their program which I can use to fine-tune the GUI. I was planning to use function decorators for this purpose, for example like this:
class App:
#Useraction(description='close the program', hotkey='ctrl+q')
def quit(self):
sys.exit()
The problem is that this information needs to be bound to the respective class. For example, if the program is an image editor, it might have an Image class which provides some more Useractions:
class Image:
#Useraction(description='invert the colors')
def invert_colors(self):
...
However, since the concept of unbound methods has been removed in python 3, there doesn't seem to be a way to find a function's defining class. (I found this old answer, but that doesn't work in a decorator.)
So, since it looks like decorators aren't going to work, what would be the best way to do this? I'd like to avoid having code like
class App:
def quit(self):
sys.exit()
Useraction(App.quit, description='close the program', hotkey='ctrl+q')
if at all possible.
For completeness' sake, the #Useraction decorator would look somewhat like this:
class_metadata= defaultdict(dict)
def Useraction(**meta):
def wrap(f):
cls= get_defining_class(f)
class_metadata[cls][f]= meta
return f
return wrap

You are using decorators to add meta data to methods. That is fine. It can be done e.g. this way:
def user_action(description):
def decorate(func):
func.user_action = {'description': description}
return func
return decorate
Now, you want to collect that data and store it in a global dictionary in form class_metadata[cls][f]= meta. For that, you need to find all decorated methods and their classes.
The simplest way to do that is probably using metaclasses. In metaclass, you can define what happens when a class is created. In this case, go through all methods of the class, find decorated methods and store them in the dictionary:
class UserActionMeta(type):
user_action_meta_data = collections.defaultdict(dict)
def __new__(cls, name, bases, attrs):
rtn = type.__new__(cls, name, bases, attrs)
for attr in attrs.values():
if hasattr(attr, 'user_action'):
UserActionMeta.user_action_meta_data[rtn][attr] = attr.user_action
return rtn
I have put the global dictionary user_action_meta_data in the meta class just because it felt logical. It can be anywhere.
Now, just use that in any class:
class X(metaclass=UserActionMeta):
#user_action('Exit the application')
def exit(self):
pass
Static UserActionMeta.user_action_meta_data now contains the data you want:
defaultdict(<class 'dict'>, {<class '__main__.X'>: {<function exit at 0x00000000029F36C8>: {'description': 'Exit the application'}}})

I've found a way to make decorators work with the inspect module, but it's not a great solution, so I'm still open to better suggestions.
Basically what I'm doing is to traverse the interpreter stack until I find the current class. Since no class object exists at this time, I extract the class's qualname and module instead.
import inspect
def get_current_class():
"""
Returns the name of the current module and the name of the class that is currently being created.
Has to be called in class-level code, for example:
def deco(f):
print(get_current_class())
return f
def deco2(arg):
def wrap(f):
print(get_current_class())
return f
return wrap
class Foo:
print(get_current_class())
#deco
def f(self):
pass
#deco2('foobar')
def f2(self):
pass
"""
frame= inspect.currentframe()
while True:
frame= frame.f_back
if '__module__' in frame.f_locals:
break
dict_= frame.f_locals
cls= (dict_['__module__'], dict_['__qualname__'])
return cls
Then in a sort of post-processing step, I use the module and class names to find the actual class object.
def postprocess():
global class_metadata
def findclass(module, qualname):
scope= sys.modules[module]
for name in qualname.split('.'):
scope= getattr(scope, name)
return scope
class_metadata= {findclass(cls[0], cls[1]):meta for cls,meta in class_metadata.items()}
The problem with this solution is the delayed class lookup. If classes are overwritten or deleted, the post-processing step will find the wrong class or fail altogether. Example:
class C:
#Useraction(hotkey='ctrl+f')
def f(self):
print('f')
class C:
pass
postprocess()

How to give a class a referencable string name?

The scenerio is I'm using an arg parser to get a command line argument auth_application.
auth_application command can have many values, for example:
cheese
eggs
noodles
pizza
These values are related to a programmable class.
I'd like a way to name the class, possible using a decorator.
So I can say
if auth_application is Cheese.__name__:
return Cheese()
Currently I maintain a tuple of auth_application names and have to expose that to my arg parser class as well as import the classes I need.
Anyways to make this better? Is there a decorator for classes to name them?
I'm looking for a python 2.7 solution, but a python 3 solution might be useful to know.

Easy peasy.
class command(object):
map = {}
def __init__(self, commandname):
self.name = commandname
def __call__(self, cls):
command.map[self.name] = cls
return cls
class NullCommand(object):
pass
#command('cheese')
class Cheese(object):
pass
#command('eggs')
class Eggs(object):
pass
def func(auth_application):
return command.map.get(auth_application, command.NullCommand)()

You can just keep a sinlge list of all of your "allowed classes" and iterate over that to find the class being referred to from the command line.
allow_classes = [Cheese,Eggs,Noodles,Pizza]
for cls in allow_classes:
if auth_application.lower() is cls.__name__.lower():
return cls()

Absolutely you can! You need to understand class attributes.
class NamedClass(object):
name = "Default"
class Cheese(NamedClass):
name = "Cheese"
print(Cheese.name)
> Cheese

You can use the standard Inspect Library to get the real class names, without having to augment your classes with any extra data - and this works for any class, in any module - even if you don't have the source code.
For instance - to list all the classes defined in mymodule :
import mymodule
import inspect
for name, obj in inspect.getmembers(mymodule, inspect.isclass):
print name
the obj variable is a real class object - which you can use to declare an instance, access class methods etc.
To get the definition of a class by it's name string - you can write a simple search function :
import mymodule
import inspect
def find_class(name):
"""Find a named class in mymodule"""
for this_name, _cls_ in inspect.getmembers(mymodule, inspect.isclass):
if this_name = name:
return _cls_
return None
....
# Create an instance of the class named in auth_application
find_class(auth_application)(args, kwargs)
NB: Code snippets not tested