I am trying to load the function in a remote environment using cPickle. But I got the
error "the 'module' object has no attribute ..." . Where I really stuck is the namespace has
already contain that attributes , even though it fails to load
Please Help
import inspect
import cPickle as pickle
from run import run
def get_source(func):
sourcelines = inspect.getsourcelines(func)[0]
sourcelines[0] = sourcelines[0].lstrip()
return "".join(sourcelines)
def fun(f):
return f()
def fun1():
return 10
funcs = (fun, fun1)
sources = [get_source(func) for func in funcs]
funcs_serialized = pickle.dumps((fun.func_name,sources),0)
args_serialized = pickle.dumps(fun1,0)
#Creating the Environment where fun & fun1 doesnot exist
del globals()['fun']
del globals()['fun1']
r = run()
r.work(funcs_serialized,args_serialized)
Here is run.py
import cPickle as pickle
class run():
def __init__(self):
pass
def work(self,funcs_serialized,args_serialized):
func, fsources = pickle.loads(funcs_serialized)
fobjs = [compile(fsource, '<string>', 'exec') for fsource in fsources]
#After eval fun and fun1 should be there in globals/locals
for fobj in fobjs:
try:
eval(fobj)
globals().update(locals())
except:
pass
print "Fun1 in Globals: ",globals()['fun1']
print "Fun1 in locals: ",locals()['fun1']
arg = pickle.loads(args_serialized)
The error is
Fun1 in Globals: <function fun1 at 0xb7dae6f4>
Fun1 in locals: <function fun1 at 0xb7dae6f4>
Traceback (most recent call last):
File "fun.py", line 32, in <module>
r.work(funcs_serialized,args_serialized)
File "/home/guest/kathi/python/workspace/run.py", line 23, in work
arg = pickle.loads(args_serialized)
AttributeError: 'module' object has no attribute 'fun1'
I found this link helpful:
http://stefaanlippens.net/python-pickling-and-dealing-with-attributeerror-module-object-has-no-attribute-thing.html
It gives two solutions. The better solution is to add to the head of the loading module (or __main__):
from myclassmodule import MyClass
But I think a better solution should exist.
From http://docs.python.org/library/pickle.html#what-can-be-pickled-and-unpickled:
Note that functions (built-in and
user-defined) are pickled by “fully
qualified” name reference, not by
value. This means that only the
function name is pickled, along with
the name of module the function is
defined in. Neither the function’s
code, nor any of its function
attributes are pickled. Thus the
defining module must be importable in
the unpickling environment, and the
module must contain the named object,
otherwise an exception will be raised.
You deleted the reference to fun1 in the module that defines fun1, thus the error.
The module name of the function is saved into the pickle, when you are doing the loads it is looking for fun1 in __main__ or whereever it was originally
try to add
from your_first_module import fun,fun1
into run.py
Related
I'm trying to bypass importing from a module, so in my __init__.py I can inject code like this:
globals().update(
{
"foo": lambda: print("Hello stackoverflow!")
}
)
so if I do import mymodule I will be able to call mymodule.foo. That is a simple concept, useless for the purpose because you can actually just define foo.
So, the idea is to modify the globals module dictionary, so in case it doesn't find the function foo it will go wherever and I can inject the code, for that I tried:
from importer import load #a load function to search for the code
from functools import wraps
def global_get_wrapper(f):
#wraps(f)
def wrapper(*args):
module_name, default = args
res = f(*args)
if res is None:
return load(module_name)
return res
return wrapper
globals().get = global_get_wrapper(globals().get) # trying to substitute get method
But it gives me an error:
AttributeError: 'dict' object attribute 'get' is read-only
The other idea I had is to preload the available function, class, etc names into the module dictionary and lazily load them later.
I run out of ideas to accomplish this and I don't know if this is even possible.
Should I go for writing my own python importer? or is there any other possibility I could not think about?
Thanks in advance.
Instead of hacking globals() it would be better to define __getattr__ for your module as follows:
module_name.py
foo = 'foo'
def bar():
return 'bar'
my_module.py
import sys
import module_name
class MyModule(object):
def foobar(self):
return 'foobar'
def __getattr__(self, item):
return getattr(module_name, item)
sys.modules[__name__] = MyModule()
and then:
>>> import my_module
>>> my_module.foo
'foo'
>>> my_module.bar()
'bar'
>>> my_module.foobar()
'foobar'
PEP 562, which targets Python 3.7, introduces __getattr__ for modules. In the rationale it also describes workarounds for previous Python versions.
It is sometimes convenient to customize or otherwise have control over access to module attributes. A typical example is managing deprecation warnings. Typical workarounds are assigning __class__ of a module object to a custom subclass of types.ModuleType or replacing the sys.modules item with a custom wrapper instance. It would be convenient to simplify this procedure by recognizing __getattr__ defined directly in a module that would act like a normal __getattr__ method, except that it will be defined on module instances.
So your mymodule can look like:
foo = 'bar'
def __getattr__(name):
print('load you custom module and return it')
Here's how it behaves:
>>> import mymodule
>>> mymodule.foo
'bar'
>>> mymodule.baz
load you custom module and return it
I don't quite understand. Would this work for you?
try:
mymodule.foo()
except:
print("whatever you wanted to do")
In both Python2 and Python3, in the stack trace the __name__ of a function is not used, the original name (the one that is specified after def) is used instead.
Consider the example:
import traceback
def a():
return b()
def b():
return c()
def c():
print("\n".join(line.strip() for line in traceback.format_stack()))
a.__name__ = 'A'
b.__name__ = 'B'
c.__name__ = 'C'
a();
The output is:
File "test.py", line 16, in <module>
a();
File "test.py", line 4, in a
return b()
File "test.py", line 7, in b
return c()
File "test.py", line 10, in c
print("\n".join(line.strip() for line in traceback.format_stack()))
Why so? How do I change the name that is used in the stack trace? Where is the __name__ attribute used then?
So, basically every function has three things that can be considered being name of the function:
The original name of the code block
It's stored in the f.__code__.co_name (where f is the function object). If you use def orig_name to create function, orig_name is that name. For lambas it's <lambda>.
This attribute is readonly and can't be changed. So the only way to create function with the custom name in runtime I'm aware of is exec:
exec("""def {name}():
print '{name}'
""".format(name='any')) in globals()
any() # prints 'any'
(There is also more low-level way to do this that was mentioned in a comment to the question.)
The immutability of co_name actually makes sense: with that you can be sure that the name you see in the debugger (or just stack trace) is exactly the same you see in the source code (along with the filename and line number).
The __name__ attribute of the function object
It's also aliased to func_name.
You can modify it (orig_name.__name__ = 'updated name') and you surely do on a daily basis: #functools.wraps copies the __name__ of the decorated function to the new one.
__name__ is used by tools like pydoc, that's why you need #functools.wraps: so you don't see the technical details of every decorator in your documentation. Look at the example:
from functools import wraps
def decorator1(f):
def decorated(*args, **kwargs):
print 'start1'
f(*args, **kwargs)
return decorated
def decorator2(f):
#wraps(f)
def decorated(*args, **kwargs):
print 'start2'
f(*args, **kwargs)
return decorated
#decorator1
def test1():
print 'test1'
#decorator2
def test2():
print 'test2'
Here is the pydoc output:
FUNCTIONS
decorator1(f)
decorator2(f)
test1 = decorated(*args, **kwargs)
test2(*args, **kwargs)
With wraps there is no sign of decorated in the documentation.
Name of the reference
One more thing that can be called function name (though it hardly is) is the name of a variable or an attribute where reference to that function is stored.
If you create function with def name, the name attribute will be added to the current scope. In case of lambda you should assign the result to some variable: name = lambda: None.
Obviously you can create more than one reference to the same function and all that references can have different names.
The only way all that three things are connected to each other is the def foo statement that creates function object with both __name__ and __code__.co_name equal to foo and assign it to the foo attribute of the current scope. But they are not bound in any way and can be different from each other:
import traceback
def make_function():
def orig_name():
"""Docstring here
"""
traceback.print_stack()
return orig_name
globals()['name_in_module'] = make_function()
name_in_module.__name__ = 'updated name'
name_in_module()
Output:
File "my.py", line 13, in <module>
name_in_module()
File "my.py", line 7, in orig_name
traceback.print_stack()
Pydoc:
FUNCTIONS
make_function()
name_in_module = updated name()
Docstring here
I thank other people for comments and answers, they helped me to organize my thoughts and knowledge.
Tried to explore the CPython implementation, definitely not an expert. As pointed out in the comments, when the stack entry of f is printed, the attribute f.__code__.co_name is used. Also, f.__name__ is initially set to f.__code__.co_name, but when you modify the former, the latter is not modified accordingly.
Therefore, I tried to modify that directly, but it is not possible:
>>> f.__code__.co_name = 'g'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: readonly attribute
>>>
Why are there two ways to say a function's name? Well, according to the documentation, __name__ is defined for "class, function, method, descriptor, or generator instance", so in the case of functions it maps to that attribute, for other objects it will map to something else.
I want to refer to an object in the namespace of the file that imports the one that I am writing.
this is an example:
main.py
from imp import * # main is importing the file I'm writing
...more code...
obj=1 # main defines obj
f() # f(), defined in imp, needs to use obj
...more code using obj...
This is the file that defines f():
imp.py
def f():
return obj # I want to refer to main's obj here
error on runtime:
error: global name 'obj' is not defined
How can it be done?
Thanks.
Relying on global variables across modules is not really a good idea. You should pass obj as a parameter to the function f(), like this:
f(obj)
Then just declare the parameter in the function:
def f(obj):
# code to operate on obj
return obj
Are there any ways to obtain a path of a file corresponding to a function object which is passed to a decorator function?
Finally I need a directory of the file.
def mydec(arg):
def dec_inner(func):
def wrapper(*args, **kwargs):
# how to detect a path where func is defined?
return wrapper
return dec_inner
You can find the name of the module a function comes from using the __module__ attribute:
>>> from random import choice
>>> choice.__module__
'random'
You can get the module from its name via the sys.modules dictionary:
>>> sys.modules['random']
<module 'random' from 'C:\Python27\lib\random.pyc'>
And the file path itself from the module attribute __file__. Putting all of that together:
>>> sys.modules[choice.__module__].__file__
'C:\\Python27\\lib\\random.pyc'
I tried to use the property definition from PythonDecoratorLibrary (example 3).
=> https://wiki.python.org/moin/PythonDecoratorLibrary#Property_Definition
import sys
def property(function):
keys = 'fget', 'fset', 'fdel'
[...]
After additionally importing sys I get this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in Angle
File "<stdin>", line 12, in property
TypeError: property() got an unexpected keyword argument 'doc'
Line 12 is: function() because of the sys import :)
My python version is 3.4.1 on Windows.
First of all: That is one ugly way to allow for local functions to define the 3 property functions.
The sample decorator masks the property built-in, but then tries to use it still to produce the property object. Oops.
You can still access the orginal built-in with:
import builtins
def property(function):
keys = 'fget', 'fset', 'fdel'
func_locals = {'doc':function.__doc__}
def probe_func(frame, event, arg):
if event == 'return':
locals = frame.f_locals
func_locals.update(dict((k, locals.get(k)) for k in keys))
sys.settrace(None)
return probe_func
sys.settrace(probe_func)
function()
return builtins.property(**func_locals)
The builtins module lets you access the builtins even when a local name has overridden it.
I've updated the wiki page to reflect this.
The example is relying on the built-in property function, while also naming itself property:
def property(function):
keys = 'fget', 'fset', 'fdel'
func_locals = {'doc':function.__doc__}
def probe_func(frame, event, arg):
if event == 'return':
locals = frame.f_locals
func_locals.update(dict((k, locals.get(k)) for k in keys))
sys.settrace(None)
return probe_func
sys.settrace(probe_func)
function()
return property(**func_locals) # This is supposed to be the built-in property
So it ends up calling itself (which doesn't take a doc keyword argument), instead of the built-in property (which takes a doc keyword argument). So yes, the example is broken. The function it's calling property should be named something else, or it should save a reference to the built-in property and call that internally.
Edit: Using builtins.property is obviously much nicer than saving a reference to property before masking it. So go with that.