I am writing a function that exports variables as a dictionary to an external file.
The problem comes when calling that function from another script. I think it has something to do with the globals() parameter.
import sys
import os
mydict = {} #'initialising" the an empty dictionary to be used locally in the function below
def writeToValues(name):
fileName = os.path.splitext(os.path.basename(sys.argv[0]))[0]
valuePrint=open("values.py","a")
def namestr(obj,namespace):
return[name for name in namespace if namespace[name] is obj]
b = namestr(name, globals())
c = "".join(str(x) for x in b)
mydict[(c)] = name
valuePrint.write(fileName)
valuePrint.write("=")
valuePrint.write(str(mydict))
valuePrint.write("\n")
valuePrint.close()
return mydict
a = 2
b = 3
writeToValues(a)
writeToValues(b)
I get the following result:
Main Junkfile={'a': 2, 'b': 3}
note the word Main Junkfile is the name of the script I ran as that is what the function first does, to get the name of the file and use that to name the dictionary.
Now help me as I cannot generate the same if I import the function from another script.
Another problem is that running the script twice generates the values in steps.
Main Junkfile={'a': 2}
Main Junkfile={'b': 3, 'a': 2}
I cannot change the file open mode from append to write since I want to store values from other scripts, too.
this is not perfect but might help as an example:
import sys
import os
mydict = {}
def namestr(obj,namespace):
return[name for name in namespace if namespace[name] is obj]
def writeto(name):
fout = 'values.py'
filename = os.path.splitext(os.path.basename(sys.argv[0]))[0]
with open (fout, 'a') as f:
b = namestr(name, globals())
c = "".join(str(x) for x in b)
mydict[(c)] = name
data = filename + '=' + str(mydict) + '\n'
f.write(data)
return mydict
a = 2
b = 3
if __name__ == '__main__':
writeto(a)
writeto(b)
First of all, to get the current executing script name, or rather the module that called your function you'll have to pick it up from the stack trace. Same goes for globals() - it will execute in the same context of writeToValues() function so it won't be picking up globals() from the 'caller'. To remedy that you can use the inspect module:
import inspect
import os
def writeToValues(name):
caller = inspect.getmodule(inspect.stack()[1][0])
caller_globals = caller.__dict__ # use this instead of globals()
fileName = os.path.splitext(os.path.basename(caller.__file__))[0]
# etc.
This will ensure that you get the name of the module that imported your script and is calling writeToValues() within it.
Keep in mind that this is a very bad idea if you intend to write usable Python files - if your script name has spaces (like in your example) it will write a variable name with spaces, which will further result in syntax error if you try to load the resulting file into a Python interpreter.
Second, why in the name of all things fluffy are you trying to do a reverse lookup to find a variable name? You are aware that:
a = 2
b = 2
ab = 5
writeToValues(b)
will write {"ab": 2}, and not {"b": 2} making it both incorrect in intent (saves the wrong var) as well as in state representation (saves a wrong value), right? You should pass a variable name you want to store/update instead to ensure you're picking up the right property.
The update part is more problematic - you need to update your file, not just merely append to it. That means that you need to find the line of your current script, remove it and then write a new dict with the same name in its place. If you don't expect your file to grow to huge proportions (i.e. you're comfortable having it partially in the working memory), you could do that with:
import os
import inspect
def writeToValues(name):
caller = inspect.getmodule(inspect.stack()[1][0])
caller_globals = caller.__dict__ # use this instead of globals()
caller_name = os.path.splitext(os.path.basename(caller.__file__))[0]
# keep 'mydict' list in the caller space so multiple callers can use this
target_dict = caller_globals['mydict'] = caller_globals.get('mydict', {})
if name not in caller_globals: # the updated value no longer exists, remove it
target_dict.pop(name, None)
else:
target_dict[name] = caller_globals[name]
# update the 'values.py':
# optionaly check if you should update - if values didn't change no need for slow I/O
with open("values.py", "a+") as f:
last_pos = 0 # keep the last non-update position
while True:
line = f.readline() # we need to use readline() for tell() accuracy
if not line or line.startswith(caller_name): # break at the matching line or EOF
break
last_pos = f.tell() # new non-update position
append_data = f.readlines() # store in memory the rest of the file content, if any
f.seek(last_pos) # rewind to the last non-update position
f.truncate() # truncate the rest of the file
f.write("".join((caller_name, " = ", str(target_dict), "\n"))) # write updated dict
if append_data: # write back the rest of the file, if truncated
f.writelines(append_data)
return target_dict
Otherwise use a temp file to write everything as you read it, except for the line matching your current script, append the new value for the current script, delete the original and rename the temp file to values.py.
So now if you store the above in, say, value_writter.py and use it in your script my_script.py as:
import value_writter
a = 2
b = 3
value_writter.write_to_values("a")
value_writter.write_to_values("b")
a = 5
value_writter.write_to_values("a")
# values.py contains: my_script = {"a": 5, "b": 3}
Same should go for any script you import it to. Now, having multiple scripts edit the same file without a locking mechanism is an accident waiting to happen, but that's a whole other story.
Also, if your values are complex the system will break (or rather printout of your dict will not look properly). Do yourself a favor and use some proper serialization, even the horrible pickle is better than this.
Related
I'm storing f-strings with function calls in a separate file (with lots of variables).
I am writing a script that has hundreds of variables which are then loaded into an HTML table. Some of the contents in the HTML table require function calls.
This works:
def add_one(a):
return a + 1
a = 1
s = f"A is {a} and next comes {add_one(a)}"
print(s)
When I store s in a file, I can use **locals() to format it and it works when I store variables in s.txt.
Contents of s.txt:
A is {a}
Contents of script that works:
a = 1
print(open('s.txt').read().format(**locals()))
However, when I try to call functions, it does not work:
Contents of s.txt:
A is {a} and next comes {add_one(a)}
Contents of script that does not work:
def add_one(a):
return a + 1
a = 1
print(open('s.txt').read().format(**locals()))
What can I do to make it work (given my actual case is hundreds of function calls and not this simple 2 variable example)?
In this example it should result in A is 1 and next comes 2.
You might want to consider using a templating language rather than f-strings if you have a complex HTML table with hundreds of variables. e.g. Jinja2.
For simplicity I've stored the a value in a dictionary as this then simplifies passing it to the Jinja2 render and also converting it to JSON for storing it in a file.
Here is your example using Jinja2 templates and storing the data to a json file:
import json
from pathlib import Path
import jinja2
json_file = Path('/tmp/test_store.json')
jinja_env = jinja2.Environment()
# Set variable values
values = {'a': 3}
# Save to json file
json_file.write_text(json.dumps(values))
# Read from json file to dictionary with new variable name
read_values = json.loads(json_file.read_text())
def add_one(a):
return a + 1
# Add custom filter to jinja environment
jinja_env.filters['add_one'] = add_one
# Define template
template = jinja_env.from_string("A is {{a}} and next comes {{a | add_one}}")
# Print rendered template
print(template.render(read_values))
This gave the output of:
A is 3 and next comes 4
The JSON file is the following:
{"a": 3}
As mentioned in the discussion e.g. here, what you want does not really work in any simple way. There is one obvious workaround: storing an f-string (e.g. f"A is {a} and next comes {add_one(a)}") in your text file and then eval'ing it:
with open('s.txt', 'r') as f:
print(eval(f.read())) # A is 1 and next comes 2
Of course, all the usual warnings about shooting yourself in the foot apply here, but your problem definition sounds exactly like this use case. You can try sandboxing your functions and whatnot, but it generally does not work well. I would say it is still a viable use case for homebrew automation, but it has a massive potential for backfiring, and the only reason I am suggesting it is because alternative solutions are likely to be about as dangerous.
Use serialization and deserialization to store data
import json
data = {
"a" : 1,
"b" : 2,
"name" : "Jack",
"bunch_of_numbers" : [1, 2, 3, 5, 6]
}
file_name = "s.txt"
with open(file_name, 'w') as file:
file.write(json.dumps(data)) #serialization
with open(file_name, 'rb') as file:
data = json.load(file) # de-serialization
print(data)
Output:
{'a': 1, 'b': 2, 'name': 'Jack', 'bunch_of_numbers': [1, 2, 3, 5, 6]}
Hello I am pretty new to python and I want to do the following:
I have a function that opens a file, reads the file, closes the file and returns the data:
def getFastaFromFile(filename):
""" Read a fasta file (filename) from disk and return
its full contents as a string"""
inf=open(filename)
data=inf.read()
inf.close()
return data
The data that is being returned are a few lines with strings.
What I want to do is have another function that uses the data from the first function and perform the .readlines(), .readline() and .count() commands
My second function:
def printTableFromFasta(fastarec):
a= data.readlines()
for i in range(a)
b= data.readline()
c= b.count('A')
print(c)
As output I would like to print the amount of times string "A" appears for every line from the data. The problem I get with this code is that the data doesn't get recognized.
First, you need to pass the data you are wanting to read into the second function, like so
def printTableFromFasta(data):
In order to get this from your first function, try returning the entire contents of the file
def getFastaFromFile(filename):
with open(filename, 'r') as inf: # handles open and close
data = inf.readlines() # Returns the entire file as a list of strings
return data
Your function call will look something like this
printTableFromFasta(getFastaFromFile(filename))
Then, in your second function, you don't need to call readlines, it's already a list.
def printTableFromFasta(data):
for line in data # look at each line
print(line.count('A')) # count 'A'
Edit:
To only read from the second function and not touch the first function
def printTableFromFasta(filename):
with open(filename, 'r') as inf: # handles open and close
for line in inf.readlines() # look at each line
print(line.count('A')) # count 'A'
Remember that the data variable in the first function is local. It cannot be accessed from outside the function it is defined in.
For example, the getName() function returns a variable which is locally called data but you access the value by calling the function.
def getName(user_id):
data = "Your name is " + str(user_id)
return data
# Throws an error, because data in undefined
name = getName("Bobby")
print(data)
# Working code, prints "Your name is Bobby"
name = getName("Bobby")
print(name)
There are no rules against calling one function from inside another.
Instead of a = data.readlines() try a = getFastaFromFile("dna.fasta') as well as changing data = inf.read() to data = inf.readlines()
I have dumped a couple of variables I need to store in a json object as shown here-
def write_json():
variable = {}
variable['p_id'] = p_id
variable['s_id'] = s_id
variable['t_id'] = t_id
variable['b_start'] = b_start
variable['b_end'] = b_end
variable['wsp_after'] = wsp_after
variable['h_id'] = h_id
variable['word'] = word
variable['norm_word'] = norm_word
variable['lemma'] = lemma
variable['pos'] = pos
variable['ner'] = ner
variable['dep'] = dep
variable['quote'] = quote
variable['ch_id'] = ch_id
with open('variable.json', 'w') as fp:
json.dump(variable,fp, sort_keys = True)
I want to write a piece of code that would automatically get these variables assigned appropriately as would happen if you wrote from 'X.py' import *
This is what I tried-
def get_variables():
with open('variable.json', 'r') as fp:
variable = json.load(fp)
for key in variable:
global key
key = variable[key]
However, when I tried to print the original value-
print t_id
NameError: name 't_id' is not defined
What do I need to do to achieve what I want rather than cumbersomely typing
t_id = variable['t_id']
.
.
.
For every variable I have dumped?
Your current code doesn't work because the following lines just bind (and rebind, for the later values in the loop) a global variable named 'key':
global key
key = variable[key]
While I'm not sure your design is really a good one, if you're sure you want all the values from your json file as global variables, I'd replace the loop with:
globals().update(variable)
The globals() function returns a dictionary that contains all the current module's global variables. Unlike the dictionary you get from locals() (which may be a copy of the local namespace, not the actual implementation of it), you can modify the globals() dictionary and use the variables like normal afterwards.
Here's the solution -
with open('variable.json', 'r') as fp:
variable = json.load(fp)
for key in variable.keys():
exec(key + " = variable['" + key + "']")
The problem before was that each 'key' was a string, not an object.
Using the exec function successfully parses each string into an object and using it as given above gets the results as was asked in the question.
This is similar to, Python creating dynamic global variable from list, but I'm still confused.
I get lots of flo data in a semi proprietary format. I've already used Python to strip the data to my needs and save the data into a json file called badactor.json and are saved in the following format:
[saddr as a integer, daddr as a integer, port, date as Julian, time as decimal number]
An arbitrary example [1053464536, 1232644361, 2222, 2014260, 15009]
I want to go through my weekly/monthly flo logs and save everything by Julian date. To start I want to go through the logs and create a list that is named according to the Julian date it happened, i.e, 2014260 and then save it to the same name 2014260.json. I have the following, but it is giving me an error:
#!/usr/bin/python
import sys
import json
import time
from datetime import datetime
import calendar
#these are varibles I've had to use throughout, kinda a boiler plate for now
x=0
templist2 = []
templist3 = []
templist4 = []
templist5 = []
bad = {}
#this is my list of "bad actors", list is in the following format
#[saddr as a integer, daddr as a integer, port, date as Julian, time as decimal number]
#or an arbitrary example [1053464536, 1232644361, 2222, 2014260, 15009]
badactor = 'badactor.json'
with open(badactor, 'r') as f1:
badact = json.load(f1)
f1.close()
for i in badact:
print i[3] #troubleshooting to verify my value is being read in
tmp = str(i[3])
print tmp#again just troubleshooting
tl=[i[0],i[4],i[1],i[2]]
bad[tmp]=bad[tmp]+tl
print bad[tmp]
Trying to create the variable is giving me the following error:
Traceback (most recent call last):
File "savetofiles.py", line 39, in <module>
bad[tmp]=bad[tmp]+tl
KeyError: '2014260'
By the time your code is executed, there is no key "2014260" in the "bad" dict.
Your problem is here:
bad[tmp]=bad[tmp]+tl
You're saying "add t1 to something that doesn't exist."
Instead, you seem to want to do:
bad[tmp]=tl
I suggest you initialize bad to be an empty collections.defaultdict instead of just regular built-in dict. i.e.
import collections
...
bad = collections.defaultdict(list)
That way, initial empty list values will be created for you automatically the first time a date key is encountered and the error you're getting from the bad[tmp]=bad[tmp]+tl statement will go away since it will effectively become bad[tmp]=list()+tl — where the list() call just creates and returns an empty list — the first time a particular date is encountered.
It's also not clear whether you really need the tmp = str(i[3]) conversion because values of any non-mutable type are valid dictionary (or defaultdict) keys, not just strings — assuming i[3] isn't a string already. Regardless, subsequent code would be more readable if you named the result something else, like julian_date = i[3] (or julian_date = str(i[3]) if the conversion really is required).
A variable AA is in aaa.py. I want to use this variable in my other python file bbb.py
How do I access this variable?
You're looking for modules!
In aaa.py:
AA = 'Foo'
In bbb.py:
import aaa
print aaa.AA # Or print(aaa.AA) for Python 3
# Prints Foo
Or this works as well:
from aaa import AA
print AA
# Prints Foo
You can import it; this will execute the whole script though.
from aaa import AA
In your file bbb.py, add the following:
import sys
sys.path.append("/path/to/aaa.py/folder/")
from aaa import AA
Also would suggest reading more about Python modules and how import works. Official Documentation on Modules.
Although importing is the best approach (like poke or Haidro wrote), here is another workaround, if you're generating data with one script and want to access them in another, without executing "bbb.py". So if you're dealing with large lists/dicts, this approach works well, although it's an overkill if you simply trying to interchange a string or a number…
Besides that, you should describe your problem more detailed, since importing variables from another script is kind of hacky and may not be the way to go.
So here are two functions, the first one (dumpobj) dumps a variable (string, number, tuple, list, dict sets whatever) to a file, the second one (loadobj) reads it in. I decided to use json, since the data files are human readable.
You may want to try that, if you need it often:
import json
def dumpobj(the_object, filename) :
"""
Dumps the_object to filename via json. Use loadobj(filename)
to get it back.
>>> from tgio import dumpobj, loadobj
>>> foo = {'bar':42, 'narf':'fjoord', 23:[1,2,3,4,5]}
>>> dumpobj(foo, 'foo.var')
>>> bar = loadobj('foo.var')
>>> bar
{u'narf': u'fjoord', u'bar': 42, u'23': [1, 2, 3, 4, 5]}
"""
try:
with open(filename):
print(filename + " exists, I'll will overwrite it.")
except IOError:
print(filename + ' does not exist. Creating it...')
f = open(filename, 'w')
json.dump(the_object, f)
f.close()
def loadobj(filename) :
"""
Retrieves dumped data (via json - see dumpobj) from filename.
>>> from tgio import loadobj, dumpobj
>>> foo = {'bar':42, 'narf':'fjoord', 23:[1,2,3,4,5]}
>>> dumpobj(foo, 'foo.var')
>>> bar = loadobj('foo.var')
>>> bar
{u'narf': u'fjoord', u'bar': 42, u'23': [1, 2, 3, 4, 5]}
"""
try:
with open(filename):
print("Reading object from file " + filename)
except IOError:
print(filename + ' does not exist. Returning None.')
return None
f = open(filename, 'r')
the_object = json.load(f)
f.close
return the_object
See the usage examples in the docstrings!