We Keep Coding

Two Objects Created from the Same Class, isinstance = False

I'm trying to create some unit tests for some code here at work.
The code takes in an object and based on information within that object imports a specific module then creates an instance of it.
The test I am trying to write creates the object and then I check that it is an instance of the class I expect it to import. The issue is the isinstance check is failing.
Here is what my test looks like.
import importlib
from path.to.imported_api import SomeApi
api = importlib.import_module("path.to.imported_api").create_instance() # create_instance() is a function that returns SomeApi().
assert isinstance(api, SomeApi) # This returns false, but I am not sure why.

The reason for the difference is, that whereas both objects refer to the same module, they get different identifiers as you load a new module and bypass sys.modules. See also the explanation here: https://bugs.python.org/issue40427
A hack might be to compare the name:
assert isinstance(api.__class__.__name__, SomeApi.__name__)

There are a few things that could cause that:
So first, it could be that the api is just returning something that looks like SomeApi(). Also it coud is be that SomeApi is overwriting isinstance behaviour.

Module Instantiation in myhdl

I'm currently looking into myHdl to see if it's worth using or not. However, I've come across a hiccup regarding the instantiation of modules. I've got two files, one that's a module and one that's the testbench. Inside the testbench, I've instantiated the module following the example they have on the website:
http://www.myhdl.org/examples/flipflops.html
The instantiation specifically is this line: dff_inst = dff(q, d, clk)
However, I get an error when I try to run the testbench:
Exception TypeError: 'isinstance() arg 2 must be a class, type, or tuple of classes and types' in <generator object _LabelGenerator at 0x7f6070b2ea50> ignored
I assume this has something to do with the fact that I have two separate files, so my guess is that python isn't finding the dff module(since it's in a separate file). I tried adding in an import dff line, but that simply gave me a 'module' object is not callable type error, which makes sense.
Looking in the documentation, they don't have a full .py file, so I'm not sure how they're linking these testbenches with the module. They specifically mention a hierarchy system and being able to instantiate other modules, but I can't seem to get it to work.
From what I understand from documentation, it looks like they're just writing the testbench and the module in the same file. However, to my understanding, it looks like they imply you can import modules, but I can't figure out how that's done. Is there just some simple thing I'm overlooking?

After experimenting a bit, it seems like I just need to use the following command: from dff import dff,
which makes a lot of sense.

What is the mechanism that allows Python monkey patching in this instance?

Can someone explain the logic behind how this works with the Python interpreter? Is this behavior only thread local? Why does the assignment in the first module import persist after the second module import? I just had a long debugging session that came down to this.
external_library.py
def the_best():
print "The best!"
modify_external_library.py
import external_library
def the_best_2():
print "The best 2!"
external_library.the_best = the_best_2
main.py
import modify_external_library
import external_library
external_library.the_best()
Test:
$ python main.py
The best 2!

Nothing thread-local about this. somemodule.anattr = avalue is very global behavior! After this assignment the attribute is changed for good (until maybe changed back later) no matter what.
There's no mysterious mechanics at play! Assignment to any attribute of an object that allows such assignment (as module objects do) just work in the obvious way -- no thread-local anything, nothing strange -- and assignment to attribute persists, as long as the object whose attribute you've assigned persists, of course.
The repeated import external_library doesn't reload the module (reload is a totally separate builtin and import does not call it!) -- it just checks sys.modules, finds an external_library key in that dict, and binds the corresponding value (which was previously modified by that assignment) to name external_library in the appropriate namespace (here, globals of module main).

As Alex indicated you need to reload external_library, simply importing it will do nothing if it's already been imported. You can check that by putting print statements into your external_library and modify_external_library modules.
import modify_external_library
#import external_library
reload(external_library)
external_library.the_best()
output
The best!

Modules are instances of new-style classes. When you modify the attributes of a module (the function in this case), you are modifying the module instance. When you try to import it again (with import external_library), you're just getting the same module object already referenced inside of modify_external_library.py.
Edit: Of course, trying to import the same module again does not really work (as Alex Martelli points out). Once loaded, modules are not re-initialized unless done so explicitly with reload.

Monkey patching works because classes are modifiable in python but the mechanism that allows it to spread like this is that once any module has been imported, and initialised, later imports simply add the existing instance to the local namespace without rerunning the initialisation, this also saves time when a module has a lot of initialisation as well as allowing monkey patches.

Serialize a python function with dependencies

I have tried multiple approaches to pickle a python function with dependencies, following many recommendations on StackOverflow, (such as dill, cloudpickle, etc.) but all seem to run into a fundamental issue that I cannot figure out.
I have a main module that tries to pickle a function from an imported module, sends it over ssh to be unpickled and executed at a remote machine.
So main has:
import dill (for example)
import modulea
serial=dill.dumps( modulea.func )
send (serial)
On the remote machine:
import dill
receive serial
funcremote = dill.loads( serial )
funcremote()
If the functions being pickled and sent are top level functions defined in main itself, everything works. When they are in an imported module, the loads function fails with messages of the type "module modulea not found".
It appears that the module name is pickled along with the function name. I do not see any way to "fix up" the pickle to remove the dependency, or alternately, to create a dummy module in the receiver to become the recipient of the unpickling.
Any pointers will be much appreciated.
--prasanna

I'm the dill author. I do this exact thing over ssh, but with success. Currently, dill and any of the other serializers pickle modules by reference… so to successfully pass a function defined in a file, you have to ensure that the relevant module is also installed on the other machine. I do not believe there is any object serializer that serializes modules directly (i.e. not by reference).
Having said that, dill does have some options to serialize object dependencies. For example, for class instances, the default in dill is to not serialize class instances by reference… so the class definition can also be serialized and send with the instance. In dill, you can also (use a very new feature to) serialize file handles by serializing the file, instead of the doing so by reference. But again, if you have the case of a function defined in a module, you are out-of-luck, as modules are serialized by reference pretty darn universally.
You might be able to use dill to do so, however, just not with pickling the object, but with extracting the source and sending the source code. In pathos.pp and pyina, dill us used to extract the source and the dependencies of any object (including functions), and pass them to another computer/process/etc. However, since this is not an easy thing to do, dill can also use the failover of trying to extract a relevant import and send that instead of the source code.
You can understand, hopefully, this is a messy messy thing to do (as noted in one of the dependencies of the function I am extracting below). However, what you are asking is successfully done in the pathos package to pass code and dependencies to different machines across ssh-tunneled ports.
>>> import dill
>>>
>>> print dill.source.importable(dill.source.importable)
from dill.source import importable
>>> print dill.source.importable(dill.source.importable, source=True)
def _closuredsource(func, alias=''):
"""get source code for closured objects; return a dict of 'name'
and 'code blocks'"""
#FIXME: this entire function is a messy messy HACK
# - pollutes global namespace
# - fails if name of freevars are reused
# - can unnecessarily duplicate function code
from dill.detect import freevars
free_vars = freevars(func)
func_vars = {}
# split into 'funcs' and 'non-funcs'
for name,obj in list(free_vars.items()):
if not isfunction(obj):
# get source for 'non-funcs'
free_vars[name] = getsource(obj, force=True, alias=name)
continue
# get source for 'funcs'
#…snip… …snip… …snip… …snip… …snip…
# get source code of objects referred to by obj in global scope
from dill.detect import globalvars
obj = globalvars(obj) #XXX: don't worry about alias?
obj = list(getsource(_obj,name,force=True) for (name,_obj) in obj.items())
obj = '\n'.join(obj) if obj else ''
# combine all referred-to source (global then enclosing)
if not obj: return src
if not src: return obj
return obj + src
except:
if tried_import: raise
tried_source = True
source = not source
# should never get here
return
I imagine something could also be built around the dill.detect.parents method, which provides a list of pointers to all parent object for any given object… and one could reconstruct all of any function's dependencies as objects… but this is not implemented.
BTW: to establish a ssh tunnel, just do this:
>>> t = pathos.Tunnel.Tunnel()
>>> t.connect('login.university.edu')
39322
>>> t
Tunnel('-q -N -L39322:login.university.edu:45075 login.university.edu')
Then you can work across the local port with ZMQ, or ssh, or whatever. If you want to do so with ssh, pathos also has that built in.

How can I instantiate a variable?

I have the following:
objects
__init__.py
define.py
define.py:
class Place:
def __init__(self,name,inhabitants):
self.name=name
self.inhabitants=inhabitants
myFunction.toStoreThings.on.db(name,inhabitants,'places')
def someUsefulFunction(self):
pass
If I run import objects, moon=objects.Place('Moon',[]), close the interpreter and open it again. I obviously loose the moon instance, but I have (u'Moon',u'[]') stored in the database. I already made __init__.py retrieve that information from the database and unstring it, but I'd also like it to instantiate 'Moon' as Moon=Place('Moon',[]) so I can use Moon.someUsefulFunction() or objects.Moon.someUsefulFunction() even after I close the interpreter. How can I achieve this?
I was able to do it like this:
__init__.py:
# myFunction() creates a dictionary `objdic` of the stuff in the database
# >>>objects.objdic
# {'places' : [['Moon',[]]]}
instancesdic={}
instancesdic['places']={}
instancesdic['places'][objdic['places'][0][0]]=Place(*objdic['places'][0])
Which gives
>>> objects.instancesdic
{'places': {'Moon': <objects.Place instance at 0x1b29248>}}
This way I can use
objects.instancesdic['places']['Moon'].someUsefulFunction()
Which is ok, but I really wanted objects.Moon.someUsefulFunction(). Any attempt to call that whole thing Moon results either in:
TypeError: 'str' object does not support item assignment
Or in just the key in the dictionary being changed to an instance, instead of the Moon instance being created.

You could use the setattr function to set module attributes on the objects module, or you could update globals within that module. So within your __init__.py you could do:
objDict = {obj[0]: Place(*obj) for obj in objdict['places']}
globals().update(objDict)
This will then let you do object.Moon, etc.
There is some danger to be aware of, though. If any of your objects have the same name as anything else already created in objects, they will overwrite those things. So if objects has a function called myFunc and then you create an object called myFunc, it could overwrite the function with the object. (Which will overwrite which depends on which order you do things in.)
For this reason, it's probably not a good idea to do this automatically in __init__.py. It can make sense to do this for ease of use in the interactive interpreter, but modifying globals in this way will get ugly if you use it in scripts. It might be a better idea to create a function called initGlobals or something, and then call that function to set up your interactive environment. If you put the code I showed above into such a function, then call it, it will set up the environment. This lets you separate the simple importing of the module from actually creating global objects from the db, because sometimes you might want to do one but not the other.