We Keep Coding

Do I need to explicitly del this object?

class Tokenizer()
def __init__(self):
self.name = 'MyTokenizer'
self.tokenizer = Language.create_tokenizer(nlp)
def __call__(self, text):
if text:
with CoreClient(timeout=60000) as client:
doc = client.annotate(text, output_format='json')
else:
doc = Document("")
...
The question I am having is with the creation of 'CoreClient', which creates a http request to a server. The current code introduced by "with ... as client", can insure that the client is destroyed when 'client.annotate' is out of scope after it's done. However, the problem is that, the object 'client' has to be created for each request of processing 'text'. In order to avoid this, I had better create the object in the init method:
self.client = CoreClient(timeout=60000)
But then:
1) How to destroy the 'client' after all requests have been completed? OR
2) Is the current way of creating a Coreclient OK for each request? The creation of the object is heavy, which needs a lot of initialization.
EDIT:
def __enter__(self):
self.start()
return self
def start(self):
if self.start_cmd:
if self.be_quiet:
# Issue #26: subprocess.DEVNULL isn't supported in python 2.7.
stderr = open(os.devnull, 'w')
else:
stderr = self.stderr
print(f"Starting server with command: {' '.join(self.start_cmd)}")
self.server = subprocess.Popen(self.start_cmd,
stderr=stderr,
stdout=stderr)
To make it more clear, I added the implementation of the method enter. It seems it simply returns the object 'self'.

You only need to create the instance of CoreClient once. The with statement just ensures that the __enter__ and __exit__ methods of that instance are called before and after the body of the with statement; you don't need to create a new instance each time.
class Tokenizer()
def __init__(self):
self.name = 'MyTokenizer'
self.tokenizer = Language.create_tokenizer(nlp)
self.client = CoreClient(timeout=60000) # Create client here
def __call__(self, text):
if text:
with self.client:
doc = self.client.annotate(text, output_format='json')
else:
doc = Document("")
It appears that __enter__ and __exit__ together spin up and tear down a new server each time the CoreClient instance is used as a context manager.
The client will be collected when the Tokenizer instance gets collected. However, unless you are in an active with statement, the CoreClient instance isn't doing anything.

In this case I wouldn't worry about it because when the reference count goes to zero, Python will take care of it. Also, del does not actually delete and object. It might, but it might not. del will decrement the reference count to an object.
Take this for example:
In [1]: class Test:
...: def __del__(self):
...: print('deleted')
...:
In [2]: t = Test()
In [3]: del t
deleted
In [4]: t = Test()
In [5]: t1 = t
In [6]: del t # Nothing gets printed here because t1 still exists
In [7]: del t1 # reference count goes to 0 and now gets printed
deleted
This is why I think you should just let Python handle the destruction of your objects. Python keeps track of objects reference counts and knows when they are no longer needed. So let it take care of that stuff for you.

Python: Passing a class member function to another class's callback

Can I pass class A into class B so that B can run a callback using A's member function?
I am trying to write a Python leg class for a robot I am building. I am using a Raspberry Pi as the main computer, and Martin O'Hanlon's KY040 rotary encoder library KY040 to detect each 1/4 rotation of the leg. To this end, I watch for the first of several clicks, sleep for a short time, stop the servo, and now a 1/4 rotation has been achieved. In standalone, unthreaded code this works fine, but creating a class has been a challenge.
Details:
A threaded sentinel loop watches a boolean (quarterTurn) to signal that a rotation must be carried out.
def run(self):
print "leg running"
while self._running:
sleep(.0001)
if self.quarterTurn:
print "quarterTurn is: " + str(self.quarterTurn)
self.qTurn(self.quarterCount)
qTurn accesses a pwm controller to activate the motors, and reset quarterTurn to false.
def qTurn(self, quarters):
count = 0
while count < quarters:
sleep(.0001)
self.setMotor(self.maxPulse)
if self.ClickedOnce:
count = count + 1
sleep(.17)
self.parkMotor()
sleep(.04)
self.clickedOnce = False
self.quarterTurn = False
The trick is that O'Hanlon's class is already threaded. On one hand, it is convenient, on the other, it makes my class more complex. The KY040 makes use of a callback function to provide feedback, but using this within my class is the source of my trouble.
I need the callback to modify a a boolean in my leg class, but this function is only called by the KY040 class, which tries to pass itself into the function.
def rotaryChange(self, pin):
self.clickedOnce = True
Since the code is open source (thank you, O'Hanlon), I thought I could modify the constructor of the KY040 to let me pass my leg class into it, so that I could modify the correct data.
O'Hanlon's Original Constructor:
def __init__(self, clockPin, dataPin, switchPin=None, rotaryCallback=None, switchCallback=None,rotaryBouncetime=250, switchBouncetime=300):
# persist values
self.clockPin = clockPin
self.dataPin = dataPin
self.switchPin = switchPin
self.rotaryCallback = rotaryCallback
self.switchCallback = switchCallback
self.rotaryBouncetime = rotaryBouncetime
self.switchBouncetime = switchBouncetime
#setup pins
GPIO.setup(clockPin, GPIO.IN)
GPIO.setup(dataPin, GPIO.IN)
if None != self.switchPin:
GPIO.setup(switchPin, GPIO.IN, pull_up_down=GPIO.PUD_UP)
I added a "host" variable, into which I pass the leg class:
def __init__(self, clockPin, dataPin, switchPin=None, rotaryCallback=None, switchCallback=None, host=None, rotaryBouncetime=250, switchBouncetime=300):
# persist values
self.clockPin = clockPin
self.dataPin = dataPin
self.switchPin = switchPin
self.rotaryCallback = rotaryCallback
self.switchCallback = switchCallback
self.rotaryBouncetime = rotaryBouncetime
self.switchBouncetime = switchBouncetime
# My Change
self.host = host
#setup pins
GPIO.setup(clockPin, GPIO.IN)
GPIO.setup(dataPin, GPIO.IN)
if None != self.switchPin:
GPIO.setup(switchPin, GPIO.IN, pull_up_down=GPIO.PUD_UP)
The modified constructor would be called like so:
self.encoder = KY040(self.clockPin, self.dataPin, rotaryCallback=self.rotaryChange, host=self)
O'Hanlon's callback now passes the host along:
def _clockCallback(self, pin):
# My change
self.rotaryCallback(pin, self.host)
My new callback:
def rotaryChange(pin, host):
host.clickedOnce = True
Unfortunately, after making sure the modified code is installed with the setup script, it doesn't seem to acknowledge my new additions. I run my program and receive the follwing error:
Traceback (most recent call last):
File "ctf.py", line 18, in <module>
LR = leg.leg(lr_chan, lr_max, lr_park, lr_clk, lr_data);
File "/home/[user]/hexacrescentapod/leg.py", line 47, in __init__
self.encoder = KY040(self.clockPin, self.dataPin,
rotaryCallback=self.rotaryChange, host=self)
TypeError: __init__() got an unexpected keyword argument 'host'

This is a little confusing because of your wording. Are you actually trying to pass a class in as you say, or an instance of that class as you seem to be doing? Which class is rotaryChange defined in?
Anyway, it looks like what you're actually trying to do is pass self.rotaryChange as a callback.
This already works, without any changes. self.rotaryChange is a bound method, meaning it knows what that self was when it was created, and will pass it when it's called. This may be easier to see with an example:
>>> class Spam:
... def eggs(self):
... pass
>>> spam = Spam()
>>> spam
<__main__.Spam at 0x119947630>
>>> spam.eggs
<bound method Spam.eggs of <__main__.Spam object at 0x119947630>>
Notice that it's a bound method of the spam object. When you call spam.eggs(), that spam object will be passed as the self argument.
This means you don't need to pass a host in, because it's already available as self. And, since that's the only thing you do with host, you don't need to pass around host in the first place. Which means you can revert all of your changes to the library code.
You do need to define your callback method as a proper method, with self as the first argument. But that's it. Then you can just pass rotaryCallback=self.rotaryChange to the constructor, and everything will work.

At a first look, it looks like your new callback is missing a self field?
The original function was
def rotaryChange(self, pin):
self.clickedOnce = True
But your implementation is:
def rotaryChange(pin, host):
host.clickedOnce = True
If this function sits inside a class it needs to have a self parameter

Is my method right for gargabe collecting circular referenced objects?

When I was playing with my newly created html module, I used weakref module to overcome the circular reference problem. Everything seems to be fine for me! but I am not sure about the way I followed and not sure about the Scope class below. I tried to have a smallest working example (Here is a link for full code). Html class is just for creating html output with python objects. The example below does not do that for simplicity, of course.
# encoding: utf-8
from __future__ import print_function, unicode_literals
import weakref
class Scope(object):
def __init__(self):
self.ref_holder = set()
def add(self, obj):
self.ref_holder.add(obj)
def __enter__(self):
return self
def __exit__(self, *args, **kwargs):
self.ref_holder = None
class Html(object):
def __init__(self, parent=None, tag="", scope=None):
self.scope = scope
if parent is None:
self.parent = None
elif type(parent) != weakref.CallableProxyType:
self.parent = weakref.proxy(parent)
if self.scope:
self.scope.add(parent)
elif parent.scope:
parent.scope.add(self)
else:
self.parent = parent
self.tag = tag
if self.scope:
self.scope.add(self)
self.children = []
def append(self, html):
if isinstance(html, basestring):
html = Html(tag=html)
return self.append(html)
elif isinstance(html, self.__class__):
self.children.append(html)
return html
else:
raise Exception("Unknown type")
def __unicode__(self):
return 'Html "{tag}" children = {children}'.format(tag=self.tag,
children=list(map(str, self.children)))
def __str__(self):
return self.__unicode__()
if __name__ == "__main__":
with Scope() as scope:
test_form = Html(tag="form", scope=scope)
test_form.append(Html(tag="label"))
test_input = Html(tag="input")
test_form.append(test_input)
print(test_form)
Here are my concerns and I will appreciate your guidance:
I call reference holder class as Scope. It just holds the references to objects even if they are not assigned to any variable so Html object is not garbage collected (note: some objects can change parent/child relation ship and therefore there is not left any strong reference to object, in the real code).
I could simply hold the object references in a list and delete it after that but using with statement seems nicer. Is the class name Scope right for this task and the way I hold references is right? Is there a good way to hold the objects' strong references created on the fly different than my method?
I believe setting the Scope.ref_holder variable to None after exiting with statement, frees all the strong references and then gc collects them. I tested this by disabling gc and calling gc.collect then no object exists as unreacable, am I right to assume this method assures there is no leakage?
EDIT
I added the link for full source code.
Code is compliant with Python 2.7

I think the point here is that you are using __exit__ matched with with in Python.
Basically it should not happen like this. In __enter__ you are returning the self, the __exit__ should remove all the references. Are you sure that you don't have any exception in the middle? Due to exception __exit__ may return false and your garbage collection may not be complete.
However to answer your question, I can tell you that the way you are making it None in the __exit__ method is forceful garbage cleaning & I think it completely depends on the OS. If self.ref_holder is somehow liked with addressed of those objects, they may be cleaned properly.
Try to do del self.ref_holder rather. Assure more powerful cleanup.

Perhaps a bit OT: why explicit class for scope?
How about something simpler instead?
#contextlib.contextmanager
def scope():
rv = set()
try:
yield rv
finally:
pass
# if you must be explicit or
# if you want side-effect in leaked scopes
rv.clear()

Share plugin resources with implemented permission rules

I have multiple scripts that are exporting a same interface and they're executed using execfile() in insulated scope.
The thing is, I want them to share some resources so that each new script doesn't have to load them again from the start, thus loosing starting speed and using unnecessary amount of RAM.
The scripts are in reality much better encapsulated and guarded from malicious plug-ins than presented in example below, that's where problems for me begins.
The thing is, I want the script that creates a resource to be able to fill it with data, remove data or remove a resource, and of course access it's data.
But other scripts shouldn't be able to change another's scripts resource, just read it. I want to be sure that newly installed plug-ins cannot interfere with already loaded and running ones via abuse of shared resources.
Example:
class SharedResources:
# Here should be a shared resource manager that I tried to write
# but got stuck. That's why I ask this long and convoluted question!
# Some beginning:
def __init__ (self, owner):
self.owner = owner
def __call__ (self):
# Here we should return some object that will do
# required stuff. Read more for details.
pass
class plugin (dict):
def __init__ (self, filename):
dict.__init__(self)
# Here some checks and filling with secure versions of __builtins__ etc.
# ...
self["__name__"] = "__main__"
self["__file__"] = filename
# Add a shared resources manager to this plugin
self["SharedResources"] = SharedResources(filename)
# And then:
execfile(filename, self, self)
# Expose the plug-in interface to outside world:
def __getattr__ (self, a):
return self[a]
def __setattr__ (self, a, v):
self[a] = v
def __delattr__ (self, a):
del self[a]
# Note: I didn't use self.__dict__ because this makes encapsulation easier.
# In future I won't use object itself at all but separate dict to do it. For now let it be
----------------------------------------
# An example of two scripts that would use shared resource and be run with plugins["name"] = plugin("<filename>"):
# Presented code is same in both scripts, what comes after will be different.
def loadSomeResource ():
# Do it here...
return loadedresource
# Then Load this resource if it's not already loaded in shared resources, if it isn't then add loaded resource to shared resources:
shr = SharedResources() # This would be an instance allowing access to shared resources
if not shr.has_key("Default Resources"):
shr.create("Default Resources")
if not shr["Default Resources"].has_key("SomeResource"):
shr["Default Resources"].add("SomeResource", loadSomeResource())
resource = shr["Default Resources"]["SomeResource"]
# And then we use normally resource variable that can be any object.
# Here I Used category "Default Resources" to add and/or retrieve a resource named "SomeResource".
# I want more categories so that plugins that deal with audio aren't mixed with plug-ins that deal with video for instance. But this is not strictly needed.
# Here comes code specific for each plug-in that will use shared resource named "SomeResource" from category "Default Resources".
...
# And end of plugin script!
----------------------------------------
# And then, in main program we load plug-ins:
import os
plugins = {} # Here we store all loaded plugins
for x in os.listdir("plugins"):
plugins[x] = plugin(x)
Let say that our two scripts are stored in plugins directory and are both using some WAVE files loaded into memory.
Plugin that loads first will load the WAVE and put it into RAM.
The other plugin will be able to access already loaded WAVE but not to replace or delete it, thus messing with other plugin.
Now, I want each resource to have an owner, some id or filename of the plugin script, and that this resource is writable only by it's owner.
No tweaking or workarounds should enable the other plugin to access the first one.
I almost did it and then got stuck, and my head is spining with concepts that when implemented do the thing, but only partially.
This eats me, so I cannot concentrate any more. Any suggestion is more than welcome!
Adding:
This is what I use now without any safety included:
# Dict that will hold a category of resources (should implement some security):
class ResourceCategory (dict):
def __getattr__ (self, i): return self[i]
def __setattr__ (self, i, v): self[i] = v
def __delattr__ (self, i): del self[i]
SharedResources = {} # Resource pool
class ResourceManager:
def __init__ (self, owner):
self.owner = owner
def add (self, category, name, value):
if not SharedResources.has_key(category):
SharedResources[category] = ResourceCategory()
SharedResources[category][name] = value
def get (self, category, name):
return SharedResources[category][name]
def rem (self, category, name=None):
if name==None: del SharedResources[category]
else: del SharedResources[category][name]
def __call__ (self, category):
if not SharedResources.has_key(category):
SharedResources[category] = ResourceCategory()
return SharedResources[category]
__getattr__ = __getitem__ = __call__
# When securing, this must not be left as this, it is unsecure, can provide a way back to SharedResources pool:
has_category = has_key = SharedResources.has_key
Now a plugin capsule:
class plugin(dict):
def __init__ (self, path, owner):
dict.__init__()
self["__name__"] = "__main__"
# etc. etc.
# And when adding resource manager to the plugin, register it with this plugin as an owner
self["SharedResources"] = ResourceManager(owner)
# ...
execfile(path, self, self)
# ...
Example of a plugin script:
#-----------------------------------
# Get a category we want. (Using __call__() ) Note: If a category doesn't exist, it is created automatically.
AudioResource = SharedResources("Audio")
# Use an MP3 resource (let say a bytestring):
if not AudioResource.has_key("Beep"):
f = open("./sounds/beep.mp3", "rb")
Audio.Beep = f.read()
f.close()
# Take a reference out for fast access and nicer look:
beep = Audio.Beep # BTW, immutables doesn't propagate as references by themselves, doesn't they? A copy will be returned, so the RAM space usage will increase instead. Immutables shall be wrapped in a composed data type.
This works perfectly but, as I said, messing resources is too much easy here.
I would like an instance of ResourceManager() to be in charge to whom return what version of stored data.

So, my general approach would be this.
Have a central shared resource pool. Access through this pool would be read-only for everybody. Wrap all data in the shared pool so that no one "playing by the rules" can edit anything in it.
Each agent (plugin) maintains knowledge of what it "owns" at the time it loads it. It keeps a read/write reference for itself, and registers a reference to the resource to the centralized read-only pool.
When an plugin is loaded, it gets a reference to the central, read-only pool that it can register new resources with.
So, only addressing the issue of python native data structures (and not instances of custom classes), a fairly locked down system of read-only implementations is as follows. Note that the tricks that are used to lock them down are the same tricks that someone could use to get around the locks, so the sandboxing is very weak if someone with a little python knowledge is actively trying to break it.
import collections as _col
import sys
if sys.version_info >= (3, 0):
immutable_scalar_types = (bytes, complex, float, int, str)
else:
immutable_scalar_types = (basestring, complex, float, int, long)
# calling this will circumvent any control an object has on its own attribute lookup
getattribute = object.__getattribute__
# types that will be safe to return without wrapping them in a proxy
immutable_safe = immutable_scalar_types
def add_immutable_safe(cls):
# decorator for adding a new class to the immutable_safe collection
# Note: only ImmutableProxyContainer uses it in this initial
# implementation
global immutable_safe
immutable_safe += (cls,)
return cls
def get_proxied(proxy):
# circumvent normal object attribute lookup
return getattribute(proxy, "_proxied")
def set_proxied(proxy, proxied):
# circumvent normal object attribute setting
object.__setattr__(proxy, "_proxied", proxied)
def immutable_proxy_for(value):
# Proxy for known container types, reject all others
if isinstance(value, _col.Sequence):
return ImmutableProxySequence(value)
elif isinstance(value, _col.Mapping):
return ImmutableProxyMapping(value)
elif isinstance(value, _col.Set):
return ImmutableProxySet(value)
else:
raise NotImplementedError(
"Return type {} from an ImmutableProxyContainer not supported".format(
type(value)))
#add_immutable_safe
class ImmutableProxyContainer(object):
# the only names that are allowed to be looked up on an instance through
# normal attribute lookup
_allowed_getattr_fields = ()
def __init__(self, proxied):
set_proxied(self, proxied)
def __setattr__(self, name, value):
# never allow attribute setting through normal mechanism
raise AttributeError(
"Cannot set attributes on an ImmutableProxyContainer")
def __getattribute__(self, name):
# enforce attribute lookup policy
allowed_fields = getattribute(self, "_allowed_getattr_fields")
if name in allowed_fields:
return getattribute(self, name)
raise AttributeError(
"Cannot get attribute {} on an ImmutableProxyContainer".format(name))
def __repr__(self):
proxied = get_proxied(self)
return "{}({})".format(type(self).__name__, repr(proxied))
def __len__(self):
# works for all currently supported subclasses
return len(get_proxied(self))
def __hash__(self):
# will error out if proxied object is unhashable
proxied = getattribute(self, "_proxied")
return hash(proxied)
def __eq__(self, other):
proxied = get_proxied(self)
if isinstance(other, ImmutableProxyContainer):
other = get_proxied(other)
return proxied == other
class ImmutableProxySequence(ImmutableProxyContainer, _col.Sequence):
_allowed_getattr_fields = ("count", "index")
def __getitem__(self, index):
proxied = get_proxied(self)
value = proxied[index]
if isinstance(value, immutable_safe):
return value
return immutable_proxy_for(value)
class ImmutableProxyMapping(ImmutableProxyContainer, _col.Mapping):
_allowed_getattr_fields = ("get", "keys", "values", "items")
def __getitem__(self, key):
proxied = get_proxied(self)
value = proxied[key]
if isinstance(value, immutable_safe):
return value
return immutable_proxy_for(value)
def __iter__(self):
proxied = get_proxied(self)
for key in proxied:
if not isinstance(key, immutable_scalar_types):
# If mutable keys are used, returning them could be dangerous.
# If owner never puts a mutable key in, then integrity should
# be okay. tuples and frozensets should be okay as keys, but
# are not supported in this implementation for simplicity.
raise NotImplementedError(
"keys of type {} not supported in "
"ImmutableProxyMapping".format(type(key)))
yield key
class ImmutableProxySet(ImmutableProxyContainer, _col.Set):
_allowed_getattr_fields = ("isdisjoint", "_from_iterable")
def __contains__(self, value):
return value in get_proxied(self)
def __iter__(self):
proxied = get_proxied(self)
for value in proxied:
if isinstance(value, immutable_safe):
yield value
yield immutable_proxy_for(value)
#classmethod
def _from_iterable(cls, it):
return set(it)
NOTE: this is only tested on Python 3.4, but I tried to write it to be compatible with both Python 2 and 3.
Make the root of the shared resources a dictionary. Give a ImmutableProxyMapping of that dictionary to the plugins.
private_shared_root = {}
public_shared_root = ImmutableProxyMapping(private_shared_root)
Create an API where the plugins can register new resources to the public_shared_root, probably on a first-come-first-served basis (if it's already there, you can't register it). Pre-populate private_shared_root with any containers you know you're going to need, or any data you want to share with all plugins but you know you want to be read-only.
It might be convenient if the convention for the keys in the shared root mapping were all strings, like file-system paths (/home/dalen/local/python) or dotted paths like python library objects (os.path.expanduser). That way collision detection is immediate and trivial/obvious if plugins try to add the same resource to the pool.

Methods on descriptors

I'm trying to implement a wrapper around a redis database that does some bookkeeping, and I thought about using descriptors. I have an object with a bunch of fields: frames, failures, etc., and I need to be able to get, set, and increment the field as needed. I've tried to implement an Int-Like descriptor:
class IntType(object):
def __get__(self,instance,owner):
# issue a GET database command
return db.get(my_val)
def __set__(self,instance,val):
# issue a SET database command
db.set(instance.name,val)
def increment(self,instance,count):
# issue an INCRBY database command
db.hincrby(instance.name,count)
class Stream:
_prefix = 'stream'
frames = IntType()
failures = IntType()
uuid = StringType()
s = Stream()
s.frames.increment(1) # float' object has no attribute 'increment'
Is seems like I can't access the increment() method in my descriptor. I can't have increment be defined in the object that the __get__ returns. This would require an additional db query if all I want to do is increment! I also don't want increment() on the Stream class, as later on when I want to have additional fields like strings or sets in Stream, then I'd need to type check the heck out of everything.

Does this work?
class Stream:
_prefix = 'stream'
def __init__(self):
self.frames = IntType()
self.failures = IntType()
self.uuid = StringType()

Why not define the magic method iadd as well as get and set. This will allow you to do normal addition with assignment on the class. It will also mean you can treat the increment separately from the get function and thereby minimise the database accesses.
So change:
def increment(self,instance,count):
# issue an INCRBY database command
db.hincrby(instance.name,count)
to:
def __iadd__(self,other):
# your code goes here

Try this:
class IntType(object):
def __get__(self,instance,owner):
class IntValue():
def increment(self,count):
# issue an INCRBY database command
db.hincrby(self.name,count)
def getValue(self):
# issue a GET database command
return db.get(my_val)
return IntValue()
def __set__(self,instance,val):
# issue a SET database command
db.set(instance.name,val)