I am trying to save a dictionary that contains a lambda function in django.core.cache. The example below fails silently.
from django.core.cache import cache
cache.set("lambda", {"name": "lambda function", "function":lambda x: x+1})
cache.get("lambda")
#None
I am looking for an explanation for this behaviour. Also, I would like to know if there is a workaround without using def.
The example below fails silently.
No, it doesn't. The cache.set() call should give you an error like:
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
Why? Internally, Django is using Python's pickle library to serialize the value you are attempting to store in cache. When you want to pull it out of cache again with your cache.get() call, Django needs to know exactly how to reconstruct the cached value. And due to this desire not to lose information or incorrectly/improperly reconstruct a cached value, there are several restrictions on what kinds of objects can be pickled. You'll note that only these types of functions may be pickled:
functions defined at the top level of a module
built-in functions defined at the top level of a module
And there is this further explanation about how pickling functions works:
Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.
Related
I have a challenge where I'm given a function where I can pass only a single argument which must be a builtin (no modules of any kind), for example chr or IndexError and use its attributes and call its functions to get access to other builtin types.
For example, if I choose the getattr function, I can access the builtins like this:
def main(a):
builtins = a(a, '__self__')
main(getattr)
Most other functions aren't of much help for my challenge. I know that the attributes are deep and a lot of information can be extracted.
This is a good reference: https://book.hacktricks.xyz/misc/basic-python/bypass-python-sandboxes
What can I get access to using an Ellipsis object, in Python written as ... ?
Subclasses can be accessed using ....__class__.__base__.__subclasses__() which returns a list and eventually get access back using a for loop to find which of those classes's __name__ attribute is catch-warnings, and that class's _module attribute has all the builtins (Code). I cannot use that because the index at which it will appear is always different
The python version I target is 3.9.
This question already has answers here:
Is there an easy way to pickle a python function (or otherwise serialize its code)?
(12 answers)
Closed 2 years ago.
I am trying to run the following code:
import pickle
def foo():
print("i am foo")
pickle_foo = pickle.dumps(foo)
def foo():
print("i am the new foo")
fkt = pickle.loads(pickle_foo)
return fkt()
foo()
The expected behavior would be:
the new defined function "foo" is called
in the new function the old function gets unpickeled and then called
output:
i am the new foo
i am foo
What actually happens is:
the new function foo gets called, and then recursively calls itself until a Recursion Error gets thrown:
RecursionError: maximum recursion depth exceeded while calling a Python object
The error does not occur, when the two functions are named differently, but that would be very unpractical for my project.
Could anyone explain, why this behavior occurs and how to avoid it (without changing the function names)?
The pickle module pickles functions based on their fully-qualified name reference. This means that if your function is redefined somewhere in code, and then you unpickle a pickled reference to it, calling it will result in a call to the new definition.
From the Python docs on pickle:
Note that functions (built-in and user-defined) are pickled by “fully
qualified” name reference, not by value. 2 This means that only the
function name is pickled, along with the name of the module the
function is defined in. Neither the function’s code, nor any of its
function attributes are pickled. Thus the defining module must be
importable in the unpickling environment, and the module must contain
the named object, otherwise an exception will be raised.
What you can do however, is use inspect.getsource() to retrieve the source code for your function, and pickle that. This requires that your code be available as source somewhere on the file system, so compiled C code imported, or other outside sources (interpreter input, dynamically loaded modules) will not work.
When you unpickle it, you can use exec to convert it into a function and execute it.
Note: this will redefine foo every time, so calls to foo cannot be guaranteed to have the same effect.
Note 2: exec is unsafe and usually unsuitable for code that will be interacting with external sources. Make sure you protect calls to exec from potential external attacks that attempt to execute arbitrary code.
So the question is basically:
I have a module called Fibo that has, for example, a function called fibonacci() that calculates a Fibonacci sequence. Since I want to use this in my program, I have to do this:
import Fibo
Fibo.fibonacci()
But this last line isn't an object with a method called fibonacci or actually it is?
Everything in Python is an object. If you're importing an object from a module, you will have the object in your current/actual module (i.e. program). You can create different namespaces and make your code more organized by using different files for your Python code. That's the advantage of using other modules and importing from it.
According to the Python documentation, it is better to simply avoid calling fibonacci a method and say that it is an attribute of an object, see:
method
A function which is defined inside a class body. [...]
attribute
A value associated with an object which is referenced by name using dotted expressions. For example, if an object o has an attribute a it would be referenced as o.a.
In the official documentation you can find the answer.
It boils down to this:
"When a module is first imported, Python searches for the module and if found, it creates a module object 1, initializing it."
So basically, yes, Fibo is an object.
I have a pickle dump which I got from a friend and he asked me to read it like :
f = open('file.pickle')
import pickle
l = pickle.loads(f.read())
But I get an ImportError saying no module named sql.models
Can someone help me understand what is happening ?
You are missing the code required to reconstruct the pickled objects.
Pickles store the location where the class can be imported from, together with the instance attributes. The original module is still required to recreate the module. From the documentation:
Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. [4]
Similarly, classes are pickled by named reference, so the same restrictions in the unpickling environment apply. Note that none of the class’s code or data is pickled, so in the following example the class attribute attr is not restored in the unpickling environment:
class Foo:
attr = 'a class attr'
picklestring = pickle.dumps(Foo)
These restrictions are why picklable functions and classes must be defined in the top level of a module.
In other words, the original data used to create the pickle includes at least one instance of a custom class that originates in a module named sql.models.
Do be careful reading arbitrary pickles, even from friends. A pickle is just a stack language that recreates arbitrary Python structures. You can construct a pickle that spawns a secret back-door server on your computer, with enough determination and skill. The pickle documention warns you explicitly:
Warning: The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.
This has been a problem in the past, even for experienced developers.
(You may read this question for some background)
I would like to have a gracefully-degrading way to pickle objects in Python.
When pickling an object, let's call it the main object, sometimes the Pickler raises an exception because it can't pickle a certain sub-object of the main object. For example, an error I've been getting a lot is "can’t pickle module objects." That is because I am referencing a module from the main object.
I know I can write up a little something to replace that module with a facade that would contain the module's attributes, but that would have its own issues(1).
So what I would like is a pickling function that automatically replaces modules (and any other hard-to-pickle objects) with facades that contain their attributes. That may not produce a perfect pickling, but in many cases it would be sufficient.
Is there anything like this? Does anyone have an idea how to approach this?
(1) One issue would be that the module may be referencing other modules from within it.
You can decide and implement how any previously-unpicklable type gets pickled and unpickled: see standard library module copy_reg (renamed to copyreg in Python 3.*).
Essentially, you need to provide a function which, given an instance of the type, reduces it to a tuple -- with the same protocol as the reduce special method (except that the reduce special method takes no arguments, since when provided it's called directly on the object, while the function you provide will take the object as the only argument).
Typically, the tuple you return has 2 items: a callable, and a tuple of arguments to pass to it. The callable must be registered as a "safe constructor" or equivalently have an attribute __safe_for_unpickling__ with a true value. Those items will be pickled, and at unpickling time the callable will be called with the given arguments and must return the unpicked object.
For example, suppose that you want to just pickle modules by name, so that unpickling them just means re-importing them (i.e. suppose for simplicity that you don't care about dynamically modified modules, nested packages, etc, just plain top-level modules). Then:
>>> import sys, pickle, copy_reg
>>> def savemodule(module):
... return __import__, (module.__name__,)
...
>>> copy_reg.pickle(type(sys), savemodule)
>>> s = pickle.dumps(sys)
>>> s
"c__builtin__\n__import__\np0\n(S'sys'\np1\ntp2\nRp3\n."
>>> z = pickle.loads(s)
>>> z
<module 'sys' (built-in)>
I'm using the old-fashioned ASCII form of pickle so that s, the string containing the pickle, is easy to examine: it instructs unpickling to call the built-in import function, with the string sys as its sole argument. And z shows that this does indeed give us back the built-in sys module as the result of the unpickling, as desired.
Now, you'll have to make things a bit more complex than just __import__ (you'll have to deal with saving and restoring dynamic changes, navigate a nested namespace, etc), and thus you'll have to also call copy_reg.constructor (passing as argument your own function that performs this work) before you copy_reg the module-saving function that returns your other function (and, if in a separate run, also before you unpickle those pickles you made using said function). But I hope this simple cases helps to show that there's really nothing much to it that's at all "intrinsically" complicated!-)
How about the following, which is a wrapper you can use to wrap some modules (maybe any module) in something that's pickle-able. You could then subclass the Pickler object to check if the target object is a module, and if so, wrap it. Does this accomplish what you desire?
class PickleableModuleWrapper(object):
def __init__(self, module):
# make a copy of the module's namespace in this instance
self.__dict__ = dict(module.__dict__)
# remove anything that's going to give us trouble during pickling
self.remove_unpickleable_attributes()
def remove_unpickleable_attributes(self):
for name, value in self.__dict__.items():
try:
pickle.dumps(value)
except Exception:
del self.__dict__[name]
import pickle
p = pickle.dumps(PickleableModuleWrapper(pickle))
wrapped_mod = pickle.loads(p)
Hmmm, something like this?
import sys
attribList = dir(someobject)
for attrib in attribList:
if(type(attrib) == type(sys)): #is a module
#put in a facade, either recursively list the module and do the same thing, or just put in something like str('modulename_module')
else:
#proceed with normal pickle
Obviously, this would go into an extension of the pickle class with a reimplemented dump method...