I'm storing f-strings with function calls in a separate file (with lots of variables).
I am writing a script that has hundreds of variables which are then loaded into an HTML table. Some of the contents in the HTML table require function calls.
This works:
def add_one(a):
return a + 1
a = 1
s = f"A is {a} and next comes {add_one(a)}"
print(s)
When I store s in a file, I can use **locals() to format it and it works when I store variables in s.txt.
Contents of s.txt:
A is {a}
Contents of script that works:
a = 1
print(open('s.txt').read().format(**locals()))
However, when I try to call functions, it does not work:
Contents of s.txt:
A is {a} and next comes {add_one(a)}
Contents of script that does not work:
def add_one(a):
return a + 1
a = 1
print(open('s.txt').read().format(**locals()))
What can I do to make it work (given my actual case is hundreds of function calls and not this simple 2 variable example)?
In this example it should result in A is 1 and next comes 2.
You might want to consider using a templating language rather than f-strings if you have a complex HTML table with hundreds of variables. e.g. Jinja2.
For simplicity I've stored the a value in a dictionary as this then simplifies passing it to the Jinja2 render and also converting it to JSON for storing it in a file.
Here is your example using Jinja2 templates and storing the data to a json file:
import json
from pathlib import Path
import jinja2
json_file = Path('/tmp/test_store.json')
jinja_env = jinja2.Environment()
# Set variable values
values = {'a': 3}
# Save to json file
json_file.write_text(json.dumps(values))
# Read from json file to dictionary with new variable name
read_values = json.loads(json_file.read_text())
def add_one(a):
return a + 1
# Add custom filter to jinja environment
jinja_env.filters['add_one'] = add_one
# Define template
template = jinja_env.from_string("A is {{a}} and next comes {{a | add_one}}")
# Print rendered template
print(template.render(read_values))
This gave the output of:
A is 3 and next comes 4
The JSON file is the following:
{"a": 3}
As mentioned in the discussion e.g. here, what you want does not really work in any simple way. There is one obvious workaround: storing an f-string (e.g. f"A is {a} and next comes {add_one(a)}") in your text file and then eval'ing it:
with open('s.txt', 'r') as f:
print(eval(f.read())) # A is 1 and next comes 2
Of course, all the usual warnings about shooting yourself in the foot apply here, but your problem definition sounds exactly like this use case. You can try sandboxing your functions and whatnot, but it generally does not work well. I would say it is still a viable use case for homebrew automation, but it has a massive potential for backfiring, and the only reason I am suggesting it is because alternative solutions are likely to be about as dangerous.
Use serialization and deserialization to store data
import json
data = {
"a" : 1,
"b" : 2,
"name" : "Jack",
"bunch_of_numbers" : [1, 2, 3, 5, 6]
}
file_name = "s.txt"
with open(file_name, 'w') as file:
file.write(json.dumps(data)) #serialization
with open(file_name, 'rb') as file:
data = json.load(file) # de-serialization
print(data)
Output:
{'a': 1, 'b': 2, 'name': 'Jack', 'bunch_of_numbers': [1, 2, 3, 5, 6]}
Related
In my program, I have certain settings that can be modified by the user, saved on the disk, and then loaded when application is restarted. Some these settings are stored as dictionaries. While trying to implement this, I noticed that after a dictionary is restored, it's values cannot be used to access values of another dictionary, because it throws a KeyError: 1 exception.
This is a minimal code example that ilustrates the issue:
import json
motorRemap = {
1: 3,
2: 1,
3: 6,
4: 4,
5: 5,
6: 2,
}
motorPins = {
1: 6,
2: 9,
3: 10,
4: 11,
5: 13,
6: 22
}
print(motorPins[motorRemap[1]]); #works correctly
with open('motorRemap.json', 'w') as fp:
json.dump(motorRemap, fp)
with open('motorRemap.json', 'r') as fp:
motorRemap = json.load(fp)
print(motorPins[motorRemap[1]]); #throws KeyError: 1
You can run this code as it is. First print statement works fine, but after the first dictionary is saved and restored, it doesn't work anymore. Apparently, saving/restoring somehow breaks that dictionary.
I have tried saving and restoring with json and pickle libraries, and both produce in the same error. I tried printing values of the first dictionary after it is restored directly ( print(motorRemap[1]), and it prints out correct values without any added spaces or anything. KeyError usually means that the specified key doesn't exist in the dictionary, but in this instance print statement shows that it does exist - unless some underlying data types have changed or something. So I am really puzzled as to why this is happening.
Can anyone help me understand what is causing this issue, and how to solve it?
What happens becomes clear when you look at what json.dump wrote into motorRemap.json:
{"1": 3, "2": 1, "3": 6, "4": 4, "5": 5, "6": 2}
Unlike Python, json can only use strings as keys. Python, on the other hand, allows many different types for dictionary keys, including booleans, floats and even tuples:
my_dict = {False: 1,
3.14: 2,
(1, 2): 3}
print(my_dict[False], my_dict[3.14], my_dict[(1, 2)])
# Outputs '1 2 3'
The json.dump function automatically converts some of these types to string when you try to save the dictionary to a json file. False becomes "false", 3.14 becomes "3.14" and, in your example, 1 becomes "1". (This doesn't work for the more complex types such as a tuple. You will get a TypeError if you try to json.dump the above dictionary where one of the keys is (1, 2).)
Note how the keys change when you dump and load a dictionary with some of the Python-specific keys:
import json
my_dict = {False: 1,
3.14: 2}
print(my_dict[False], my_dict[3.14])
with open('my_dict.json', 'w') as fp:
json.dump(my_dict, fp)
# Writes {"false": 1, "3.14": 2} into the json file
with open('my_dict.json', 'r') as fp:
my_dict = json.load(fp)
print(my_dict["false"], my_dict["3.14"])
# And not my_dict[False] or my_dict[3.14] which raise a KeyError
Thus, the solution to your issue is to access the values using strings rather than integers after you load the dictionary from the json file.
print(motorPins[motorRemap["1"]]) instead of your last line will fix your code.
From a more general perspective, it might be worth considering keeping the keys as strings from the beginning if you know you will be saving the dictionary into a json file. You could also convert the values back to integers after loading as discussed here; however, that can lead to bugs if not all the keys are integers and is not a very good idea in bigger scale.
Checkout pickle if you want to save the dictionary keeping the Python format. It is, however, not human-readable unlike json and it's also Python-specific so it cannot be used to transfer data to other languages, missing virtually all the main benefits of json.
If you want to save and load the dictionary using pickle, this is how you would do it:
# import pickle
...
with open('motorRemap.b', 'wb') as fp:
pickle.dump(motorRemap, fp)
with open('motorRemap.b', 'rb') as fp:
motorRemap = pickle.load(fp)
...
since the keys (integers) from a dict will be written to the json file as strings, we can modify the reading of the json file. using a dict comprehension restores the original dict values:
...
with open('motorRemap.json', 'r') as fp:
motorRemap = {int(item[0]):item[1] for item in json.load(fp).items()}
...
I am writing a function that exports variables as a dictionary to an external file.
The problem comes when calling that function from another script. I think it has something to do with the globals() parameter.
import sys
import os
mydict = {} #'initialising" the an empty dictionary to be used locally in the function below
def writeToValues(name):
fileName = os.path.splitext(os.path.basename(sys.argv[0]))[0]
valuePrint=open("values.py","a")
def namestr(obj,namespace):
return[name for name in namespace if namespace[name] is obj]
b = namestr(name, globals())
c = "".join(str(x) for x in b)
mydict[(c)] = name
valuePrint.write(fileName)
valuePrint.write("=")
valuePrint.write(str(mydict))
valuePrint.write("\n")
valuePrint.close()
return mydict
a = 2
b = 3
writeToValues(a)
writeToValues(b)
I get the following result:
Main Junkfile={'a': 2, 'b': 3}
note the word Main Junkfile is the name of the script I ran as that is what the function first does, to get the name of the file and use that to name the dictionary.
Now help me as I cannot generate the same if I import the function from another script.
Another problem is that running the script twice generates the values in steps.
Main Junkfile={'a': 2}
Main Junkfile={'b': 3, 'a': 2}
I cannot change the file open mode from append to write since I want to store values from other scripts, too.
this is not perfect but might help as an example:
import sys
import os
mydict = {}
def namestr(obj,namespace):
return[name for name in namespace if namespace[name] is obj]
def writeto(name):
fout = 'values.py'
filename = os.path.splitext(os.path.basename(sys.argv[0]))[0]
with open (fout, 'a') as f:
b = namestr(name, globals())
c = "".join(str(x) for x in b)
mydict[(c)] = name
data = filename + '=' + str(mydict) + '\n'
f.write(data)
return mydict
a = 2
b = 3
if __name__ == '__main__':
writeto(a)
writeto(b)
First of all, to get the current executing script name, or rather the module that called your function you'll have to pick it up from the stack trace. Same goes for globals() - it will execute in the same context of writeToValues() function so it won't be picking up globals() from the 'caller'. To remedy that you can use the inspect module:
import inspect
import os
def writeToValues(name):
caller = inspect.getmodule(inspect.stack()[1][0])
caller_globals = caller.__dict__ # use this instead of globals()
fileName = os.path.splitext(os.path.basename(caller.__file__))[0]
# etc.
This will ensure that you get the name of the module that imported your script and is calling writeToValues() within it.
Keep in mind that this is a very bad idea if you intend to write usable Python files - if your script name has spaces (like in your example) it will write a variable name with spaces, which will further result in syntax error if you try to load the resulting file into a Python interpreter.
Second, why in the name of all things fluffy are you trying to do a reverse lookup to find a variable name? You are aware that:
a = 2
b = 2
ab = 5
writeToValues(b)
will write {"ab": 2}, and not {"b": 2} making it both incorrect in intent (saves the wrong var) as well as in state representation (saves a wrong value), right? You should pass a variable name you want to store/update instead to ensure you're picking up the right property.
The update part is more problematic - you need to update your file, not just merely append to it. That means that you need to find the line of your current script, remove it and then write a new dict with the same name in its place. If you don't expect your file to grow to huge proportions (i.e. you're comfortable having it partially in the working memory), you could do that with:
import os
import inspect
def writeToValues(name):
caller = inspect.getmodule(inspect.stack()[1][0])
caller_globals = caller.__dict__ # use this instead of globals()
caller_name = os.path.splitext(os.path.basename(caller.__file__))[0]
# keep 'mydict' list in the caller space so multiple callers can use this
target_dict = caller_globals['mydict'] = caller_globals.get('mydict', {})
if name not in caller_globals: # the updated value no longer exists, remove it
target_dict.pop(name, None)
else:
target_dict[name] = caller_globals[name]
# update the 'values.py':
# optionaly check if you should update - if values didn't change no need for slow I/O
with open("values.py", "a+") as f:
last_pos = 0 # keep the last non-update position
while True:
line = f.readline() # we need to use readline() for tell() accuracy
if not line or line.startswith(caller_name): # break at the matching line or EOF
break
last_pos = f.tell() # new non-update position
append_data = f.readlines() # store in memory the rest of the file content, if any
f.seek(last_pos) # rewind to the last non-update position
f.truncate() # truncate the rest of the file
f.write("".join((caller_name, " = ", str(target_dict), "\n"))) # write updated dict
if append_data: # write back the rest of the file, if truncated
f.writelines(append_data)
return target_dict
Otherwise use a temp file to write everything as you read it, except for the line matching your current script, append the new value for the current script, delete the original and rename the temp file to values.py.
So now if you store the above in, say, value_writter.py and use it in your script my_script.py as:
import value_writter
a = 2
b = 3
value_writter.write_to_values("a")
value_writter.write_to_values("b")
a = 5
value_writter.write_to_values("a")
# values.py contains: my_script = {"a": 5, "b": 3}
Same should go for any script you import it to. Now, having multiple scripts edit the same file without a locking mechanism is an accident waiting to happen, but that's a whole other story.
Also, if your values are complex the system will break (or rather printout of your dict will not look properly). Do yourself a favor and use some proper serialization, even the horrible pickle is better than this.
In my script, I'm trying to save a dictionary using cPickle. Everything works fine except the thing that loaded dictionary has modified each key.
My dictionary looks like: {'a':[45,155856,26,98536], 'b':[88,68,9454,78,4125,52]...}
When I print keys from this dictionary before saving it, it prints correct values: 'a','b'...
But when I save it and then load using cPickle, each key contains '\r' after correct char: 'a\r','b\r'...
Here is the code for saving:
def saveSuffixArrayDictA():
for i in self.creation.dictA.keys():
print len(i)
print 'STOP'
with open('dictA','w+') as f:
pickle.dump(self.creation.dictA,f)
Which prints: 1,1,1,1,1,1....STOP (with newlines of course)
Then, when I'm trying to load it using this:
#staticmethod
def dictA():
with open('ddictA','rb') as f:
dict = pickle.load(f)
for i in dict.keys():
print len(i)
print 'STOP'
return dict
It returns: 2,2,2,2,2,2,2,2...STOP (with newlines of course)
As you can see it should be the same but it isn't... where could be the problem please?
EDIT: I tried to print values and realized that each item in list (list is value) has added 'L' at the end of this item which is a number.
Per the docs:
Be sure to always open pickle files created with protocols >= 1 in binary
mode. For the old ASCII-based pickle protocol 0 you can use either text mode
or binary mode as long as you stay consistent. (my emphasis)
Therefore, do not write the pickle file in the text-mode w+, but read it in the binary mode rb.
Instead, use binary modes, wb+ and rb, for both.
When you write in text mode (e.g. w+), \n is mapped to the OS-specific end-of-line character(s). On Windows, \n is mapped to \r\n. That appears to be the source of the errant \rs appearing in the keys.
This is a very strange error and I don't know its reason. But here is another way for saving and loading data structures in python. Just convert your data structure to string using str() and write it to any file. Load the file back, read it in any variable and convert it back to data structure using ast. Demo is given below:
>>> import ast
>>> d={'a':[1,2,3,4],'b':[5,6,7,8],'c':[9,10]}
>>> saveDic=str(d)
>>> saveDic
"{'a': [1, 2, 3, 4], 'c': [9, 10], 'b': [5, 6, 7, 8]}"
# save this string to any file, load it back and convert to dictionary using ast
>>> d=ast.literal_eval(saveDic)
>>> d
{'a': [1, 2, 3, 4], 'c': [9, 10], 'b': [5, 6, 7, 8]}
I'm trying to store many variables in a file. I've tried JSON, pickle and shelve but they all seem to only take one variable
import shelve
myShelve = shelve.open('my.shelve')
myShelve.update(aasd,
basd,
casd,
dasd,
easd,
fasd,
gasd,
hasd,
iasd,
jasd)
myShelve.close()
And pickle
import pickle
with open("vars.txt", "wb") as File:
pickle.dumps(aasd,
basd,
casd,
dasd,
easd,
fasd,
gasd,
hasd,
iasd,
jasd,
File)
The errors I'm getting are along the lines of
TypeError: update() takes at most 2 positional arguments (11 given)
and
TypeError: pickle.dumps() takes at most 2 positional argument (11 given)
I'm not sure if there's any other way of storing variables except using a database, but that's a bit over what I'm currently capable of I'd say.
You can only pickle one variable at a time, but it can be a dict or other Python object. You could store your many variables in one object and pickle that object.
import pickle
class Box:
pass
vars = Box()
vars.x = 1
vars.y = 2
vars.z = 3
with open("save_vars.pickle", "wb") as f:
f.write(pickle.dumps(vars))
with open("save_vars.pickle", "rb") as f:
v = pickle.load(f)
assert vars.__dict__ == v.__dict__
using pickle, you dump one object at a time. Each time you dump to the file, you add another "record".
import pickle
with open("vars.txt", "wb") as File:
for item in (aasd,
basd,
casd,
dasd,
easd,
fasd,
gasd,
hasd,
iasd,
jasd)
pickle.dump(item,File)
Now, on when you want to get your data back, you use pickle.load to read the next "record" from the file:
import pickle
with open('vars.txt') as fin:
aasd = pickle.load(fin)
basd = pickle.load(fin)
...
Alternatively, depending on the type of data, assuming the data is stuff that json is able to serialize, you can store it in a json list:
import json
# dump to a string, but you could use json.dump to dump it to a file.
json.dumps([aasd,
basd,
casd,
dasd,
easd,
fasd,
gasd,
hasd,
iasd,
jasd])
EDIT: I just thought of a different way to store your variables, but it is a little weird, and I wonder what the gurus think about this.
You can save a file that has the python code of your variable definitions in it, for example vars.py which consists of simple statements defining your values:
x = 30
y = [1,2,3]
Then to load that into your program, just do from vars import * and you will have x and y defined, as if you had typed them in.
Original normal answer below...
There is a way using JSON to get your variables back without redefining their names, but you do have to create a dictionary of variables first.
import json
vars={} # the dictionary we will save.
LoL = [ range(5), list("ABCDE"), range(5) ]
vars['LOList'] = LoL
vars['x'] = 24
vars['y'] = "abc"
with open('Jfile.txt','w') as myfile:
json.dump(vars,myfile,indent=2)
Now to load them back:
with open('Jfile.txt','r') as infile:
D = json.load(infile)
# The "trick" to get the variables in as x,y,etc:
globals().update(D)
Now x and y are defined from their dictionary entries:
print x,y
24 abc
There is also an alternative using variable-by-variable definitions. In this way, you don't have to create the dictionary up front, but you do have to re-name the variables in proper order when you load them back in.
z=26
w="def"
with open('Jfile2.txt','w') as myfile:
json.dump([z,w],myfile,indent=2)
with open('Jfile2.txt','r') as infile:
zz,ww = json.load(infile)
And the output:
print zz,ww
26 def
A variable AA is in aaa.py. I want to use this variable in my other python file bbb.py
How do I access this variable?
You're looking for modules!
In aaa.py:
AA = 'Foo'
In bbb.py:
import aaa
print aaa.AA # Or print(aaa.AA) for Python 3
# Prints Foo
Or this works as well:
from aaa import AA
print AA
# Prints Foo
You can import it; this will execute the whole script though.
from aaa import AA
In your file bbb.py, add the following:
import sys
sys.path.append("/path/to/aaa.py/folder/")
from aaa import AA
Also would suggest reading more about Python modules and how import works. Official Documentation on Modules.
Although importing is the best approach (like poke or Haidro wrote), here is another workaround, if you're generating data with one script and want to access them in another, without executing "bbb.py". So if you're dealing with large lists/dicts, this approach works well, although it's an overkill if you simply trying to interchange a string or a number…
Besides that, you should describe your problem more detailed, since importing variables from another script is kind of hacky and may not be the way to go.
So here are two functions, the first one (dumpobj) dumps a variable (string, number, tuple, list, dict sets whatever) to a file, the second one (loadobj) reads it in. I decided to use json, since the data files are human readable.
You may want to try that, if you need it often:
import json
def dumpobj(the_object, filename) :
"""
Dumps the_object to filename via json. Use loadobj(filename)
to get it back.
>>> from tgio import dumpobj, loadobj
>>> foo = {'bar':42, 'narf':'fjoord', 23:[1,2,3,4,5]}
>>> dumpobj(foo, 'foo.var')
>>> bar = loadobj('foo.var')
>>> bar
{u'narf': u'fjoord', u'bar': 42, u'23': [1, 2, 3, 4, 5]}
"""
try:
with open(filename):
print(filename + " exists, I'll will overwrite it.")
except IOError:
print(filename + ' does not exist. Creating it...')
f = open(filename, 'w')
json.dump(the_object, f)
f.close()
def loadobj(filename) :
"""
Retrieves dumped data (via json - see dumpobj) from filename.
>>> from tgio import loadobj, dumpobj
>>> foo = {'bar':42, 'narf':'fjoord', 23:[1,2,3,4,5]}
>>> dumpobj(foo, 'foo.var')
>>> bar = loadobj('foo.var')
>>> bar
{u'narf': u'fjoord', u'bar': 42, u'23': [1, 2, 3, 4, 5]}
"""
try:
with open(filename):
print("Reading object from file " + filename)
except IOError:
print(filename + ' does not exist. Returning None.')
return None
f = open(filename, 'r')
the_object = json.load(f)
f.close
return the_object
See the usage examples in the docstrings!