I'm trying to integrate Flask with Dill to dump/load Python sessions on the server side. The code below has two functions, the first one sets the value of x to zero and imports the datetime library. The second one increments x by 1 and gets the timestamp.
The first function dumps the session and the second function loads it.
In the dump, the pickle file is generated correctly, but I cannot reuse x or get the timestamp.
This is the error when I try to execute x = x + 1 in the second function:
UnboundLocalError: local variable 'x' referenced before assignment
Can Dill be used with Flask? Do I need a different approach?
The code:
from flask import Flask
from dill import dump_session, load_session
app = Flask(__name__)
app.config['SECRET_KEY'] = 'super secret'
session_file = '/tmp/session.pkl'
#app.route('/start_counter')
def start_counter():
import datetime
x = 0
dump_session(filename = session_file)
return 'New counter started!'
#app.route('/count')
def count():
load_session(filename = session_file)
x = x + 1
now = datetime.datetime.now()
dump_session(filename = session_file)
return str(x) + '-' + str(now)
How to fix?
To make things simple, you need a data structure to hold your application state. I would use a dict because it's simple, but you can define a class for that too.
The easy (and tedious) way is to call state = dill.load('filename')and dill.dump(object,'filename') every time you need your application state.
This will work if your application is small. If you need to maintain a proper application state, you should use a database.
Ok. But WHAT happened here?
There are no compatibility issues with dill and Flask.
When you call dill.dump_session() it saves the state of __main__.
But, when you increase x in function count(), it is undefined because it wasn't saved by dill.
An easy way to see that is to put a breakpoint() before x = x + 1 or print the content inside a try..except clause:
try:
print(x)
except ee:
print(ee)
pass;
x = x + 1
So, it didn't worked because the variable x wasn't defined in __main__ but in in the scope of the start_counter() function and dill.load_session() restores the stuff in __main__.
And what does that __main__ stuff means?
Let's see that using a Repl:
~/$ python
Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']
Perfect. We have an empty python interpreter. That dir() shows what we have in __main__.
Now we'll load some libraries, and assign a variable, and define a function just because we can:
>>> import pandas, numpy, dill, pickle, json, datetime
>>> foo = "bar"
>>> def functionWithUglyName():
... print("yep")
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'datetime', 'dill', 'foo', 'functionWithUglyName', 'json', 'numpy', 'pandas', 'pickle']```
Well. That __main__ stuff looks more populated.
Now let's save the session and exit the Repl:
>>> dill.dump_session('session_01')
>>> exit()
What happens when we load the session with `dill.load_session()' ?
Let's open another Repl to discover it:
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']
Ok. Just another empty python interpreter...
Let's load the session and see what happens:
>>> import dill
>>> dill.load_session('session_01')
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'datetime', 'dill', 'foo', 'functionWithUglyName', 'json', 'numpy', 'pandas', 'pickle']
It loaded the contents of __main__ as expected.
Wait a second.
It loaded the functionWithUglyName we defined before.
Is it real?
>>> functionWithUglyName()
yep
Turns out that dill is really good at serializing stuff.
Most of the time you'll just need to pickle some data, but dill can do much more... and it is great for debugging and testing.
Related
i want to implement sort of plugin architecture that dynamically loads modules and calls a function from them
for instance plugin code looks like (in file "foo_func.py")
foo_local = []
def foo_add(a: int, b: int) -> int:
c = a + b
foo_local.append(c)
return c
def foo_print():
print(foo_local)
i need to support two plugins with the same code but with different memory state, so i created directory structure like this:
<ROOT_PROJECT>
app.py
bar/
apple/
foo/
foo_func.py
__init__.py
orange/
foo/
foo_func.py
__init__.py
code in "apple" and "orange" folders is the same.
then in app file i try to load modules and invoke functions from them
import importlib
from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print
apple = importlib.import_module('bar.apple.foo')
orange = importlib.import_module('bar.orange.foo')
apple_foo = getattr(apple, 'foo_func')
orange_foo = getattr(orange, 'foo_func')
apple_foo_add_my = getattr(apple_foo, 'foo_add')
apple_foo_print_my = getattr(apple_foo, 'foo_print')
apple_foo_add_my(1, 2)
apple_foo_print_my()
and this works fine, but you see these import lines at the top
from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print
they are not used in code (even pycharm complains about it)
but if i try to comment code and run it - then failure
AttributeError: module 'bar.apple.foo' has no attribute 'foo_func'
why ?
I suppose normal plugins should deal only with "importlib.import_module" and "getattr" and it must be enough ?
what is wrong here ?
Let's switch completely to direct imports for this explanation, because:
import something.whatever as name
is the same as:
name = importlib.import_module("something.whatever")
So let's rewrite your apple code:
apple = importlib.import_module('bar.apple.foo')
apple_foo = getattr(apple, 'foo_func')
will become:
import bar.apple.foo as apple
apple_foo = apple.foo_func
Now, the first line loads bar.apple.foo as a module. In case of packages, this means importing package's __init__.py code. And treating it as a module itself.
And what's the code in the package's init? Usually nothing! That's why the name lookup fails.
However, when you do any import my_package.whatever, the package gets its insides checked and the name becomes visible. You're basically pre-loading the module for interpreter to look at.
Why is pycharm giving you not used suggestion? Because it's not used as a variable anywhere. You're only using a side-effect + pycharm doesn't analyze strings for imports or attributes.
Visual example, with a part of standard library:
>>> import xml
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> xml.etree
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'xml' has no attribute 'etree'
>>>
>>> import xml.etree
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'etree']
And another example, what happens if there are multiple modules in the package:
>>> import dateutil
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_version']
but:
>>> import dateutil.parser
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_common', '_version', 'parser', 'relativedelta', 'tz']
All sub-modules are now visible and usable with their qualified name.
tl;dr: import my_package == only my_package/__init__.py is looked at. import my_package.whatever == python now knows it's a package and registers its insides, all modules of my_package are visible and usable.
I am using ubuntu18.04. I have installed pygame module to play songs. But the time.Clock() function is not working. It is showing this following error after running the program: AttributeError: 'function' object has no attribute 'Clock'.
Here is the code:
def playSong(filename):
mixer.init()
mixer.music.load('/home/mjiabir/Music/rangamati songs/Roar.mp3')
mixer.music.play()
while mixer.music.get_busy:
time.Clock.tick(10)
mixer.music.stop()
Looks like Linux doesn't support this module. What should I do now?
I think you want to use pygame.time.Clock().tick(10).
One way to look up modules of a package or sub-package is to use dir(). For example:
# in a python interactive shell
import time
dir(time)
# output => ['CLOCK_BOOTTIME', 'CLOCK_MONOTONIC', 'CLOCK_MONOTONIC_RAW', 'CLOCK_PROCESS_CPUTIME_ID', 'CLOCK_REALTIME', 'CLOCK_THREAD_CPUTIME_ID', '_STRUCT_TM_ITEMS', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'altzone', 'asctime', 'clock', 'clock_getres', 'clock_gettime', 'clock_gettime_ns', 'clock_settime', 'clock_settime_ns', 'ctime', 'daylight', 'get_clock_info', 'gmtime', 'localtime', 'mktime', 'monotonic', 'monotonic_ns', 'perf_counter', 'perf_counter_ns', 'process_time', 'process_time_ns', 'pthread_getcpuclockid', 'sleep', 'strftime', 'strptime', 'struct_time', 'thread_time', 'thread_time_ns', 'time', 'time_ns', 'timezone', 'tzname', 'tzset']
import pygame
dir(pygame.time)
# output => ['Clock', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'delay', 'get_ticks', 'set_timer', 'wait']
Although you can use pygame.time.Clock().tick(), I would suggest constructing a Clock object, through clock = pygame.time.Clock() and clock.tick(10). Generally, it is good practice to utilize the object-oriented nature of Python, as opposed to calling construct methods directly.
I am reading about how import works in python.
When I do:
import A.B.C
A, A.B, A.B.C are put in sys.modules. Expected.
A's __init__, A.B's __init__ get executed. Expected.
But here is a surprise: When I print globals(), only A is put into the namespace, while 'A.B.C' is not. I expect 'A.B.C' to be in global namespace.
And this means, I can access A.x defined in A's __init__.
Why is import implemented this way?
Only objects/names are put in globals namespace. A.B.C is not a valid name.
In your above case, the object is the module object for A , and its name is A .
In this particular case, if you do -
dir(A)
You would see B inside it, and that means its an attribute of the module object A . If you do -
hasattr(A,'B')
It would return True.
And in the same way if you do - dir(A.B) , you would be able to see C in it , and C is an attribute of A.B .
A Very simple example to show this -
My directory structur -
shared/
__init__.py
pkg/
__init__.py
b.py
Then in code I do -
>>> import shared.pkg.b
>>> dir(shared)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'pkg']
>>> hasattr(shared,'pkg')
True
>>>
>>> dir(shared.pkg)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'b']
>>> hasattr(shared.pkg,'b')
True
B and C are reachable through A.
eg.
import A.B.C
print(A.B.C)
If you want B and C to appear directly in your current namespace then do
from A import B
from A.B import C
print(B, C)
i have installed a python package plivo using the sudo pip install plivo.
and interpreter i test it with some code like:
>>> import plivo
>>> p = plivo.RestAPI('xxx', 'yyy')
everything working fine in python interpreter.
exactly same code is not working in a python script test_plivio.py
giving error : AttributeError: 'module' object has no attribute 'RestAPI'
then i checked with dir()
in interpreter
>>> dir(plivo)
['Account', 'Application', 'Call', 'Carrier', 'Conference', 'ConferenceMember', 'EndPoint', 'Message', 'Number', 'PLIVO_VERSION', 'PlivoError', 'PlivoResponse', 'Pricing', 'Recording', 'RestAPI', 'SubAccount', 'XML', '__builtins__', '__doc__', '__file__', '__name__', '__package__', 'base64', 'hmac', 'json', 'requests', 'sha1', 'validate_signature']
RestAPI is there.
while in test_plivo.py dir(plivo) is like:
['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'main']
clearly dir(plivo) in script is missing RestAPI with other functions.
why is that behavior and how to resolve that ?
You are importing a different module; on your path you have a different plivo.py (or plivo.pyc cached bytecode) file.
Print out the __file__ attribute to see what is imported instead:
print plivo.__file__
and rename that or move it somewhere else.
It states in the Python documentation that pickle is not secure and shouldn't parse untrusted user input. If you research this; almost all examples demonstrate this with a system() call via os.system.
Whats not clear to me, is how os.system is interpreted correctly without the os module being imported.
>>> import pickle
>>> pickle.loads("cos\nsystem\n(S'ls /'\ntR.") # This clearly works.
bin boot cgroup dev etc home lib lib64 lost+found media mnt opt proc root run sbin selinux srv sys tmp usr var
0
>>> dir() # no os module
['__builtins__', '__doc__', '__name__', '__package__', 'pickle']
>>> os.system('ls /')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'os' is not defined
>>>
Can someone explain?
The name of the module (os) is part of the opcode, and pickle automatically imports the module:
# pickle.py
def find_class(self, module, name):
# Subclasses may override this
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)
return klass
Note the __import__(module) line.
The function is called when the GLOBAL 'os system' pickle bytecode instruction is executed.
This mechanism is necessary in order to be able to unpickle instances of classes whose modules haven't been explicitly imported into the caller's namespace.
For altogether too much information on writing malicious Pickles that go much further than the standard os.system() example, see this presentation and its accompanying paper.
If you use pickletools.dis to disassemble the pickle you can see how this is working:
import pickletools
print pickletools.dis("cos\nsystem\n(S'ls ~'\ntR.")
Output:
0: c GLOBAL 'os system'
11: ( MARK
12: S STRING 'ls ~'
20: t TUPLE (MARK at 11)
21: R REDUCE
22: . STOP
Pickle uses a simple stack-based virtual machine that records the instructions used to reconstruct the object. In other words the pickled instructions in your example are:
Push self.find_class(module_name, class_name) i.e. push os.system
Push the string 'ls ~'
Build tuple from topmost stack items
Apply callable to argtuple, both on stack. i.e. os.system(*('ls ~',))
Source
Importing a module only adds it to the local namespace, which is not necessarily the one you're in. Except when it doesn't:
>>> dir()
['__builtins__', '__doc__', '__name__', '__package__']
>>> __import__('os')
<module 'os' from '/usr/lib64/python2.7/os.pyc'>
>>> dir()
['__builtins__', '__doc__', '__name__', '__package__']