I'm trying to mimic the matlab load and save functions. I'm following this thread: Shelve Code gives KeyError
It is smart. However, if I write that code in a separate module, and try to import that module and invoke that function, then it can't access the global variables.
Specifically, I write a happy.py and have the functions inside:
def save(filename='tmp', globals_=None):
if globals_ is None:
globals_ = globals()
globals()
import shelve
my_shelf = shelve.open(filename, 'n')
for key, value in globals_.items():
if not key.startswith('__'):
try:
my_shelf[key] = value
except Exception:
print('ERROR shelving: "%s"' % key)
else:
print('shelved: "%s"' % key)
my_shelf.close()
def load(filename='tmp', globals_=None):
import shelve
my_shelf = shelve.open(filename)
for key in my_shelf:
globals()[key] = my_shelf[key]
my_shelf.close()
and when I try
a = 1
b = 2
happy.save()
It would not give save a and b.
Is this because global() would not give the objects outside the module? How can I do what I want to do then?
The following will work as a separate module:
import shelve
import sys
import types
EXCLUDED_TYPES = (types.ModuleType,) # Everything can't be shelved.
def save(filename='tmp', globals_=None):
if globals_ is None:
globals_ = sys._getframe(1).f_globals # Caller's globals.
with shelve.open(filename, 'n') as my_shelf:
for key, value in globals_.items():
if not (key.startswith('__') or isinstance(value, EXCLUDED_TYPES)):
try:
my_shelf[key] = value
except Exception as e:
print('ERROR shelving: "%s"' % key, 'Exception:', e)
else:
print('shelved: "%s"' % key)
def load(filename='tmp', globals_=None):
if globals_ is None:
globals_ = sys._getframe(1).f_globals # Caller's globals.
with shelve.open(filename) as my_shelf:
for key in my_shelf:
globals_[key]=my_shelf[key]
print('unshelved: "%s"' % key)
Generally speaking, I don't think it's a good idea for a function to create global variables like this. Also note that load() might silently change existing values in the caller's namespace.
You can't easily save all global namespaces, since there's one associated with every module loaded, in addition to __main__'s. If you really want to do that, it might be possible to do so by iterating through the contents of sys.modules.
You can use inspect to look at the stack. This silly (poorly named function) that I've defined seems to do an OK job of picking up the global variables from the calling namespace although I haven't tested it extensively. I am also unsure about whether it will work with different python implementations. (I mention this because the inspect.currentframe function is definitely implementation dependent). It seems to work OK with Cpython for what it's worth.
import inspect
def save(globals=None):
if globals is None:
frames = inspect.stack()
caller_frame = frames[-1][0]
globals = dict((k,v) for (k,v) in caller_frame.f_globals.items() if not k.startswith('__'))
return globals
if __name__ == "__main__":
a = 1
b = 2
print save()
I don't have a problem with this code when it is pasted into an console:
>>> def save(filename='tmp',globals_=None):
... import shelve
... globals_ = globals_ or globals()
... my_shelf= shelve.open(filename, 'n')
... for key, value in globals_.items():
... if not key.startswith('__'):
... try:
... my_shelf[key] = value
... except Exception:
... print('ERROR shelving: "%s"' % key)
... else:
... print('shelved: "%s"' % key)
... my_shelf.close()
...
>>> def load(filename='tmp',globals_=None):
... import shelve
... my_shelf = shelve.open(filename)
... for key in my_shelf:
... globals()[key]=my_shelf[key]
... my_shelf.close()
...
>>> a, b = 1, 2
>>> save()
shelved: "load"
shelved: "a"
shelved: "b"
shelved: "save"
And then:
>>> def save(filename='tmp',globals_=None):
... import shelve
... globals_ = globals_ or globals()
... my_shelf= shelve.open(filename, 'n')
... for key, value in globals_.items():
... if not key.startswith('__'):
... try:
... my_shelf[key] = value
... except Exception:
... print('ERROR shelving: "%s"' % key)
... else:
... print('shelved: "%s"' % key)
... my_shelf.close()
...
>>> def load(filename='tmp',globals_=None):
... import shelve
... my_shelf = shelve.open(filename)
... for key in my_shelf:
... globals()[key]=my_shelf[key]
... my_shelf.close()
...
>>> load()
>>> a, b
(1, 2)
But it is a bit odd when you use it as a module:
>>> from happy import *
>>> a, b
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>> load()
>>> a, b
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>> happy.a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'happy' is not defined
>>> from happy import *
>>> a, b
(1, 2)
Is there enough here for you to have a work-around?
Related
suppose I have the following function:
def test():
...
if x['error']:
raise
This would raise an exception regardless if x['error'] is defined or not.
Instead if I try this, it doesn't throw any exception:
def test():
...
try:
if x['error']:
raise
except:
return
How can I test for a specific value and return an exception if it is defined, and to return successfully if it is not defined?
If you want to return error as string:
>>> def test():
try:
if x['error']:raise
except Exception as err:
return err
>>> test()
NameError("name 'x' is not defined",)
If you want to an error to occur:
>>> def test():
try:
if x['error']:raise
except:
raise
>>> test()
Traceback (most recent call last):
File "<pyshell#20>", line 1, in <module>
test()
File "<pyshell#19>", line 3, in test
if x['error']:raise
NameError: name 'x' is not defined
def test():
...
if x.get(‘error’):
raise
You can avoid unintentionally raising an error using the dictionary's built in get function. Get will return None if the value at the specified key does not exist instead of throwing an exception.
try this one
def check_not_exist(d,k):
#if keys exists in dict,raise it
if k in d:
raise
else:
return True
I am trying to write a save/load command like the one in MATLAB (ability to save local variables to disk or load them into current context, or work space in MATLAB's terminology).
I wrote the following code, but it doesn't seem to work, as the variables in the outer scope are not replaced, probability because of a memory copy which takes place somewhere.
Here is the code:
import shelve
import logging
import inspect
logger = logging.getLogger()
def save_locals(filename, keys=None):
my_shelf = shelve.open(filename, 'n') # 'n' for new
caller_locals = inspect.stack()[1][0].f_locals
if keys is None:
keys = caller_locals.keys()
for key in keys:
try:
my_shelf[key] = caller_locals[key]
except TypeError:
#
# __builtins__, my_shelf, and imported modules can not be shelved.
#
print('ERROR shelving: {0}'.format(key))
my_shelf.close()
def load_locals(filename, keys=None):
my_shelf = shelve.open(filename)
caller_locals = inspect.stack()[1][0].f_locals
if keys is None:
keys = list(my_shelf.keys())
for key in keys:
try:
caller_locals[key] = my_shelf[key]
except ValueError:
print('cannot get variable %s'.format(key))
Here is the test which fails:
from unittest import TestCase
from .io import save_locals, load_locals
class TestIo(TestCase):
def test_save_load(self):
sanity = 'sanity'
an_int = 3
a_float = 3.14
a_list = [1, 2, 3]
a_dict = [{'a': 5, 'b': 3}]
save_locals('temp')
an_int = None
a_float = None
a_list = None
a_dict = None
load_locals('temp')
self.assertIn('an_int', locals())
self.assertIn('a_float', locals())
self.assertIn('a_list', locals())
self.assertIn('a_dict', locals())
self.assertEqual(an_int, 3)
self.assertEqual(a_float, 3.14)
self.assertEqual(a_list, [1, 2, 3])
self.assertEqual(a_dict, [{'a': 5, 'b': 3}])
When I break-point inside load_locals I can see it changes the f_locals dictionary but when the function returns they do not change.
No, you can't update local variables on the fly. The reason is because the local symbol table is saved as a C array for optimization and both locals() and frame.f_locals end up returning a copy to that local symbol table. The official response is that modifying locals() has undefined behavior. This thread talks a bit about it.
It ends up being extra weird because calling locals() or frame.f_locals returns the same dictionary each time, which gets re-synced at different times. Here just calling frame.f_locals resets the local
def test_locals():
frame = inspect.stack()[1][0]
caller_locals = frame.f_locals
caller_locals['an_int'] = 5
print(caller_locals)
_ = frame.f_locals
print(caller_locals)
def call_test_locals():
an_int = 3
test_locals()
call_test_locals()
output:
{'an_int': 5}
{'an_int': 3}
The behavior is going to depend on the Python implementation and probably other edge cases, but a few examples where (1) the variable is defined and is not updated; (2) the variable is not defined and is updated; (3) the variable is defined and subsequently deleted and is not updated.
def test_locals():
frame = inspect.stack()[1][0]
caller_locals = frame.f_locals
caller_locals['an_int'] = 5
def call_test_locals1():
an_int = 3
print('calling', locals())
test_locals()
print('done', locals())
def call_test_locals2():
print('calling', locals())
test_locals()
print('done', locals())
def call_test_locals3():
an_int = 3
del an_int
print('calling', locals())
test_locals()
print('done', locals())
print('\n1:')
call_test_locals1()
print('\n2:')
call_test_locals2()
print('\n3:')
call_test_locals3()
output:
1:
calling {'an_int': 3}
done {'an_int': 3}
2:
calling {}
done {'an_int': 5}
3:
calling {}
done {}
If you're running Python 2, you could use exec to execute a string into the local namespace, but it won't work in Python 3 and is in general probably a bad idea.
import shelve
import logging
import inspect
logger = logging.getLogger()
def save_locals(filename, keys=None):
my_shelf = shelve.open(filename, 'n') # 'n' for new
caller_locals = inspect.stack()[1][0].f_locals
if keys is None:
keys = caller_locals.keys()
for key in keys:
try:
my_shelf[key] = caller_locals[key]
except TypeError:
#
# __builtins__, my_shelf, and imported modules can not be shelved.
#
print('ERROR shelving: {0}'.format(key))
my_shelf.close()
def load_locals_string(filename, keys=None):
my_shelf = shelve.open(filename)
if keys is None:
keys = list(my_shelf.keys())
return ';'.join('{}={!r}'.format(key, my_shelf[key]) for key in keys)
and
from unittest import TestCase
from .io import save_locals, load_locals
class TestIo(TestCase):
def test_save_load(self):
sanity = 'sanity'
an_int = 3
a_float = 3.14
a_list = [1, 2, 3]
a_dict = [{'a': 5, 'b': 3}]
save_locals('temp')
an_int = None
a_float = None
a_list = None
a_dict = None
exec load_locals_string('temp')
self.assertIn('an_int', locals())
self.assertIn('a_float', locals())
self.assertIn('a_list', locals())
self.assertIn('a_dict', locals())
self.assertEqual(an_int, 3)
self.assertEqual(a_float, 3.14)
self.assertEqual(a_list, [1, 2, 3])
self.assertEqual(a_dict, [{'a': 5, 'b': 3}])
In Python 2, exec uses PyFrame_LocalsToFast to copy the variables back to the local scope, but can't in Python 3 because exec is a function. Martijn Pieters has a good post about it.
In py.test monkeypatching/mocking documentation this is not mentioned, but it is possible to monkeypatch a local variable introduced in a function body?
my experiment:
def my_method():
my_var = 'foo'
return my_var[:2]
test is:
def test_my_method(monkeypatch):
monkeypatch.setattr(my_module.MyClass.my_method.my_var, lambda: 'bar')
assert my_method() == 'ba'
AttributeError: 'function' object at MyClass.my_method has no
attribute 'my_var'
This is not possible as the variable does not exist ahead of time and py.test cannot hook into the creation of a local variable as far as i know.
With a little care, it would be possible to patch the consts in the function code object using ctypes.
import ctypes
from contextlib import contextmanager
def tuple_setitem(tup, index, item):
obj = ctypes.py_object(tup)
item = ctypes.py_object(item)
ref_count = ctypes.c_long.from_address(id(tup))
original_count = ref_count.value
if original_count != 1:
ref_count.value = 1
ctypes.pythonapi.Py_IncRef(item)
ctypes.pythonapi.PyTuple_SetItem(obj, ctypes.c_ssize_t(index), item)
ref_count.value = original_count
#contextmanager
def patch_tuple_item(tup, index, item):
old = tup[index]
try:
tuple_setitem(tup, index, item)
yield
finally:
tuple_setitem(tup, index, old)
Demo:
>>> def my_method():
... my_var = "foo"
... return my_var[:2]
...
>>> consts = my_method.__code__.co_consts
>>> consts
(None, 'foo', 2)
>>> with patch_tuple_item(consts, index=1, item="bar"):
... print(my_method())
...
ba
>>> print(my_method())
fo
I have a function which load data into a dictionnary.
But, How can I load the dictionnary into Globals() inside a function.
Inside a function is important since we can do it easily outside on a script side.
def load237(filename):
filename = osp.abspath(filename)
old_cwd = os.getcwdu()
os.chdir(osp.dirname(filename))
error_message = None
try:
tar = tarfile.open(filename, "r")
tar.extractall()
pickle_filename = osp.splitext(filename)[0]+'.pickle'
data = cPickle.load(file(pickle_filename))
saved_arrays = {}
if load_array is not None:
try:
saved_arrays = data.pop('__saved_arrays__')
for (name, index), fname in saved_arrays.iteritems():
arr = np.load( osp.join(osp.dirname(filename), fname) )
if index is None:
data[name] = arr
elif isinstance(data[name], dict):
data[name][index] = arr
else:
data[name].insert(index, arr)
except KeyError:
pass
for fname in [pickle_filename]+[fn for fn in saved_arrays.itervalues()]:
os.remove(fname)
except (EOFError, ValueError), error:
error_message = unicode(error)
os.chdir(old_cwd)
return data, error_message
This one does not work (globals is local to the module/function...)
def load_inmemory(fpath):
globals().update(load237(fpath)[0])
You should really be storing those names on an object stored in a global and not as global variables. But you asked how to do it and so here is how:
Using Getting corresponding module from function with a for loop and setattr as modules do not support dictionary operations and it is possible to write the function as:
import sys
def load_inmemory():
module = sys.modules[load_inmemory.__module__]
for k, v in load237(fpath)[0].items():
setattr(module, k, v)
load_inmemory()
print x
I tested the following:
import sys
def func():
module = sys.modules[func.__module__]
for k,v in {'x':4}.items():
setattr(module, k, v)
func()
print x
Prints 4. Tested in Python 2.7.3.
I wanted to use the following code from here:
How can I save all the variables in the current python session?
import shelve
T='Hiya'
val=[1,2,3]
filename='/tmp/shelve.out'
my_shelf = shelve.open(filename,'n') # 'n' for new
for key in dir():
try:
my_shelf[key] = globals()[key]
except TypeError:
#
# __builtins__, my_shelf, and imported modules can not be shelved.
#
print('ERROR shelving: {0}'.format(key))
my_shelf.close()
But it gives the following error:
Traceback (most recent call last):
File "./bingo.py", line 204, in <module>
menu()
File "./bingo.py", line 67, in menu
my_shelf[key] = globals()[key]
KeyError: 'filename'
Can you help me please?
Thanks!
From your traceback, it appears you are trying to run that code from inside a function.
But dir looks up names in the current local scope. So if filename is defined inside the function, it will be in locals() rather than globals().
You probably want something more like this:
import shelve
T = 'Hiya'
val = [1, 2, 3]
def save_variables(globals_=None):
if globals_ is None:
globals_ = globals()
filename = '/tmp/shelve.out'
my_shelf = shelve.open(filename, 'n')
for key, value in globals_.items():
if not key.startswith('__'):
try:
my_shelf[key] = value
except Exception:
print('ERROR shelving: "%s"' % key)
else:
print('shelved: "%s"' % key)
my_shelf.close()
save_variables()
Note that when globals() is called from within the function, it returns the variables from the module where the function is defined, not from where it's called.
So if the save_variables function is imported, and you want the variables from the current module, then do:
save_variables(globals())