I am working on a piece of code that contains a variable named varname. Here is a simplified version of it:
class Task:
def __init__(self, context, taskname, varname='results'):
self.context = context
self.taskname = taskname
self.varname = varname
def execute(self):
"""
Dispatch this task to the correct handler.
"""
logger.info(f"Running task: {self.taskname}")
try:
handler = getattr(self, self.taskname)
except AttributeError:
raise RuntimeError(f'Task "{self.taskname}" is currently not implemented')
# Assign output of task to ``varname``
context[self.varname] = handler()
return context
The Task class is intended to be used by a task runner where callers can specify the name of the task and also a "varname" where the results will be stored. That way, subsequent tasks can refer to the results of earlier tasks. Multiple tasks are run like this:
execution_context = {}
todolist = [
('some_task', 'results'),
('some_other_task', None),
]
for task_name, varname in todolist:
task = Task(execution_context, task_name, varname)
execution_context.update(task.execute())
Here is my question: How can I avoid using a variable named varname in this situation? It seems silly to have a variable that holds the name of another variable (well, actually the name of a dictionary key). Of course, I can rename the variable, but in the past when I've been confronted with situations like this, there was usually a more elegant solution than "evaluating" variable names. Or is there perhaps nothing wrong with this approach?
Related
If you create a new Process in python, it will serialize and copy the entire available scope, as far as I understand it. If you use multiprocessing.Pipe() it also allows sending various things, not just raw bytes.
However, instead of sending, I simply want to update a variable that contains a simple POD object like this:
class MyStats:
def __init__(self):
self.bytes_read = 0
self.bytes_written = 0
So say that in a process, when I update these stats, I want to tell python to serialize it and send it to the parent process' side somehow. I don't want to have to create multiprocessing.Value for each and every one of these things, that sounds super tedious.
Is there a way to tell python to pass and overwrite a specific object property somehow?
A manager is what you need here: it will be slower but all data stored inside will be automatically synced with other processes. Here is a simple example below:
from multiprocessing.managers import BaseManager, public_methods, NamespaceProxy
from multiprocessing import Process
def make_proxy(name, cls, base=None):
"""
Args:
name : A string that should match the variable name the proxy will be assigned to
cls : The class for which you want to create a proxy for
base : If you are subclassing NamespaceProxy (or any other implementation) and want to use that subclass as the
base for this new proxy, then pass the subclass as the base using this argument
"""
exposed = public_methods(cls) + ['__getattribute__', '__setattr__', '__delattr__']
return _MakeProxyType(name, exposed, base)
def _MakeProxyType(name, exposed, base=None):
"""
Attempts to replicate multiprocessing.managers.MakeProxType properly
"""
if base is None:
base = NamespaceProxy
exposed = tuple(exposed)
dic = {}
for meth in exposed:
if hasattr(base, meth):
continue
exec('''def %s(self, *args, **kwds):
return self._callmethod(%r, args, kwds)''' % (meth, meth), dic)
ProxyType = type(name, (base,), dic)
ProxyType._exposed_ = exposed
return ProxyType
class MyStats:
def __init__(self):
self.bytes_read = 0
self.bytes_written = 0
def worker(my_stats):
my_stats.bytes_read = 100
print("Worker process read 100 bytes!")
# Remember to set the name of the variable and the "name" argument to the same value otherwise you will have trouble
# pickling this. If for some reason you cannot do this then you must change the variable's __qualname__ property to
# reflect where the object actually resides so pickle can find it.
MyStatsProxy = make_proxy('MyStatsProxy', MyStats)
if __name__ == "__main__":
# Register our proxy and start the manager process
BaseManager.register("MyStats", MyStats, MyStatsProxy)
manager = BaseManager()
manager.start()
# Create our shared instance and modify it from another process
my_stats = manager.MyStats()
p = Process(target=worker, args=(my_stats,))
p.start()
p.join()
# Check value from main process
print(f"In main process, bytes read are {my_stats.bytes_read}!")
Output
Worker process read 100 bytes!
In main process, bytes read are 100!
Check this question and its answers for more useful information about managers/registering classes and alternate methods to achieve the same result
Note: Keep in mind that managers return pickled values for all objects you access through it. So any modifications you do on mutable objects should be done from within an instance method rather than requesting the mutable object through the proxy and modifying it from outside. For example, doing below will not modify the attribute some_list in the manager at all, only the local copy (to the process) of this attribute will be modified:
my_stats.some_list[0] = "some value"
Instead, you should create an instance method for modifications and call that instead:
my_stats.modify_list(0, "some value")
Alternatively, you can also force the manager to update the mutable object by re-assigning the new value for the object:
local_copy = my_stats.some_list
local_copy[0] = "some value"
my_stats.some_list = local_copy
I am using structlog - http://www.structlog.org/en/stable/ in my Python Project. I have one if the processors in the configuration to be
stdlib.add_logger_name
This adds the key in the event_dict to be logger. But, I want to change the key string to something else like namespace rather than logger. How can I do that?
I have checked the function for
stdlib.add_logger_name(logger, method_name, event_dict)
but that function uses hardcoded string logger as
event_dict["logger"] = logger.name
Currently, structlog.stdlib.add_logger_name() is 6 LoC, of which you most likely only need two:
def add_logger_name(logger, method_name, event_dict):
"""
Add the logger name to the event dict.
"""
record = event_dict.get("_record")
if record is None:
event_dict["logger"] = logger.name
else:
event_dict["logger"] = record.name
return event_dict
Just copy and paste it and adapt it to your needs.
It wouldn't be worth it to add options to the processor and slow it down for everybody since it didn't come up until today, but structlog has been engineered purposefully to make such customizations easy.
Thanks to hynek's answer.
I solved this by adding a local function:
def add_logger_name(logger, method_name, event_dict):
"""
Add the logger name to the event dict with namespace as the key as per logging convention
"""
record = event_dict.get("_record")
if record is None:
event_dict["namespace"] = logger.name
else:
event_dict["namespace"] = record.name
return event_dict
Setting this in the
processors=[add_logger_name,...]
I am trying to build some UI panels for an Eclipse based tool. The API for the tool has a mechanism for event handling based on decorators, so for example, the following ties callbackOpen to the opening of a_panel_object:
#panelOpenHandler(a_panel_object)
def callbackOpen(event):
print "opening HERE!!"
This works fine, but I wanted to wrap all of my event handlers and actual data processing for the panel behind a class. Ideally I would like to do something like:
class Test(object):
def __init__(self):
# initialise some data here
#panelOpenHandler(a_panel_object)
def callbackOpen(self, event):
print "opening HERE!!"
But this doesn't work, I think probably because I am giving it a callback that takes both self and event, when the decorator is only supplying event when it calls the function internally (note: I have no access to source code on panelOpenHandler, and it is not very well documented...also, any error messages are getting swallowed by Eclipse / jython somewhere).
Is there any way that I can use a library decorator that provides one argument to the function being decorated on a function that takes more than one argument? Can I use lambdas in some way to bind the self argument and make it implicit?
I've tried to incorporate some variation of the approaches here and here, but I don't think that it's quite the same problem.
Your decorator apparently registers a function to be called later. As such, it's completely inappropriate for use on a class method, since it will have no idea of which instance of the class to invoke the method on.
The only way you'd be able to do this would be to manually register a bound method from a particular class instance - this cannot be done using the decorator syntax. For example, put this somewhere after the definition of your class:
panelOpenHandler(context.controls.PerformanceTuneDemoPanel)(Test().callbackOpen)
I found a work around for this problem. I'm not sure if there is a more elegant solution, but basically the problem boiled down to having to expose a callback function to global() scope, and then decorate it with the API decorator using f()(g) syntax.
Therefore, I wrote a base class (CallbackRegisterer), which offers the bindHandler() method to any derived classes - this method wraps a function and gives it a unique id per instance of CallbackRegisterer (I am opening a number of UI Panels at the same time):
class CallbackRegisterer(object):
__count = 0
#classmethod
def _instanceCounter(cls):
CallbackRegisterer.__count += 1
return CallbackRegisterer.__count
def __init__(self):
"""
Constructor
#param eq_instance 0=playback 1=record 2=sidetone.
"""
self._id = self._instanceCounter()
print "instantiating #%d instance of %s" % (self._id, self._getClassName())
def bindHandler(self, ui_element, callback, callback_args = [], handler_type = None,
initialize = False, forward_event_args = False, handler_id = None):
proxy = lambda *args: self._handlerProxy(callback, args, callback_args, forward_event_args)
handler_name = callback.__name__ + "_" + str(self._id)
if handler_id is not None:
handler_name += "_" + str(handler_id)
globals()[handler_name] = proxy
# print "handler_name: %s" % handler_name
handler_type(ui_element)(proxy)
if initialize:
proxy()
def _handlerProxy(self, callback, event_args, callback_args, forward_event_args):
try:
if forward_event_args:
new_args = [x for x in event_args]
new_args.extend(callback_args)
callback(*new_args)
else:
callback(*callback_args)
except:
print "exception in callback???"
self.log.exception('In event callback')
raise
def _getClassName(self):
return self.__class__.__name__
I can then derive a class from this and pass in my callback, which will be correctly decorated using the API decorator:
class Panel(CallbackRegisterer):
def __init__(self):
super(Panel, self).__init__()
# can bind from sub classes of Panel as well - different class name in handle_name
self.bindHandler(self.controls.test_button, self._testButtonCB, handler_type = valueChangeHandler)
# can bind multiple versions of same function for repeated ui elements, etc.
for idx in range(0, 10):
self.bindHandler(self.controls["check_box_"+str(idx)], self._testCheckBoxCB,
callback_args = [idx], handler_type = valueChangeHandler, handler_id = idx)
def _testCheckBoxCB(self, *args):
check_box_id = args[0]
print "in _testCheckBoxCB #%d" % check_box_id
def _testButtonCB(self):
"""
Handler for test button
"""
print "in _testButtonCB"
panel = Panel()
Note, that I can also derive further sub-classes from Panel, and any callbacks bound there will get their own unique handler_name, based on class name string.
I want to force object instantiation via class context manager. So make it impossible to instantiate directly.
I implemented this solution, but technically user can still instantiate object.
class HessioFile:
"""
Represents a pyhessio file instance
"""
def __init__(self, filename=None, from_context_manager=False):
if not from_context_manager:
raise HessioError('HessioFile can be only use with context manager')
And context manager:
#contextmanager
def open(filename):
"""
...
"""
hessfile = HessioFile(filename, from_context_manager=True)
Any better solution ?
If you consider that your clients will follow basic python coding principles then you can guarantee that no method from your class will be called if you are not within the context.
Your client is not supposed to call __enter__ explicitly, therefore if __enter__ has been called you know your client used a with statement and is therefore inside context (__exit__ will be called).
You just need to have a boolean variable that helps you remember if you are inside or outside context.
class Obj:
def __init__(self):
self._inside_context = False
def __enter__(self):
self._inside_context = True
print("Entering context.")
return self
def __exit__(self, *exc):
print("Exiting context.")
self._inside_context = False
def some_stuff(self, name):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
print("Doing some stuff with", name)
def some_other_stuff(self, name):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
print("Doing some other stuff with", name)
with Obj() as inst_a:
inst_a.some_stuff("A")
inst_a.some_other_stuff("A")
inst_b = Obj()
with inst_b:
inst_b.some_stuff("B")
inst_b.some_other_stuff("B")
inst_c = Obj()
try:
inst_c.some_stuff("c")
except Exception:
print("Instance C couldn't do stuff.")
try:
inst_c.some_other_stuff("c")
except Exception:
print("Instance C couldn't do some other stuff.")
This will print:
Entering context.
Doing some stuff with A
Doing some other stuff with A
Exiting context.
Entering context.
Doing some stuff with B
Doing some other stuff with B
Exiting context.
Instance C couldn't do stuff.
Instance C couldn't do some other stuff.
Since you'll probably have many methods that you want to "protect" from being called from outside context, then you can write a decorator to avoid repeating the same code to test for your boolean:
def raise_if_outside_context(method):
def decorator(self, *args, **kwargs):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
return method(self, *args, **kwargs)
return decorator
Then change your methods to:
#raise_if_outside_context
def some_other_stuff(self, name):
print("Doing some other stuff with", name)
I suggest the following approach:
class MainClass:
def __init__(self, *args, **kwargs):
self._class = _MainClass(*args, **kwargs)
def __enter__(self):
print('entering...')
return self._class
def __exit__(self, exc_type, exc_val, exc_tb):
# Teardown code
print('running exit code...')
pass
# This class should not be instantiated directly!!
class _MainClass:
def __init__(self, attribute1, attribute2):
self.attribute1 = attribute1
self.attribute2 = attribute2
...
def method(self):
# execute code
if self.attribute1 == "error":
raise Exception
print(self.attribute1)
print(self.attribute2)
with MainClass('attribute1', 'attribute2') as main_class:
main_class.method()
print('---')
with MainClass('error', 'attribute2') as main_class:
main_class.method()
This will outptut:
entering...
attribute1
attribute2
running exit code...
---
entering...
running exit code...
Traceback (most recent call last):
File "scratch_6.py", line 34, in <module>
main_class.method()
File "scratch_6.py", line 25, in method
raise Exception
Exception
None that I am aware of. Generally, if it exists in python, you can find a way to call it. A context manager is, in essence, a resource management scheme... if there is no use-case for your class outside of the manager, perhaps the context management could be integrated into the methods of the class? I would suggest checking out the atexit module from the standard library. It allows you to register cleanup functions much in the same way that a context manager handles cleanup, but you can bundle it into your class, such that each instantiation has a registered cleanup function. Might help.
It is worth noting that no amount of effort will prevent people from doing stupid things with your code. Your best bet is generally to make it as easy as possible for people to do smart things with your code.
You can think of hacky ways to try and enforce this (like inspecting the call stack to forbid direct calls to your object, boolean attribute that is set upon __enter__ that you check before allowing other actions on the instance) but that will eventually become a mess to understand and explain to others.
Irregardless, you should also be certain that people will always find ways to bypass it if wanted. Python doesn't really tie your hands down, if you want to do something silly it lets you do it; responsible adults, right?
If you need an enforcement, you'd be better off supplying it as a documentation notice. That way if users opt to instantiate directly and trigger unwanted behavior, it's their fault for not following guidelines for your code.
If I have a class with attributes...
class Test(object):
def __init__():
self.variable='test'
self.variable2=''
def testmethod():
print self.variable2
t=Test()
#celery.task(name="tasks.application")
def application():
t.testmethod()
t.variable2 = '1234'
job = application.apply_async()
and I want to access the attributes of my class...
In my testing I am not able to access t.variable2 once inside of my celery task... How can I get access to those attributes?
Thanks!
Tasks are executed by a separate worker process, which being in a different process does not have access to the thread where you assigned those values. You need to send the data required by the class you're instantiating inside the task as arguments to the task, and create the instance inside the task as well:
#celery.task(name="tasks.application")
def application(variable, variable2):
t = Test()
t.variable = variable
t.variable2 = variable2
t.testmethod()
job = application.apply_async(['test', '1234'])