I have a class to import data from a CSV file, and a function that takes the filename and a name for the output list. I want to set the name of the self.data_name to be self.info using the setattr() function. How can I do this?
import csv
class import_data:
def import_csv(self, filename_csv, data_name):
setattr(self,data_name,0)
datafile = open(filename_csv, 'r')
datareader = csv.reader(datafile)
self.data_name = []
for row in datareader:
self.data_name.append(row)
print("finished importing data")
b = import_data()
b.import_csv('info.csv', 'info')
print(b.info)
This does not work because b.data_name is not b.info. This prints 0 instead of the imported CSV file.
Try this:
class import_data:
def import_csv(self, filename_csv, data_name):
with open(filename_csv, 'r') as f:
setattr(self, data_name, list(csv.reader(f)))
print("finished importing data")
You're going to have to replace all usages of self.data_name in the import_csv() function with calls to either setattr() or getattr() to be able to use the dynamic name.
Using self.data_name will use the member named data_name, as I suspect you've already realised, and that isn't what you want to do.
For example, try the following:
class import_data:
def import_csv(self, filename_csv, data_name):
#set dynamic named item to default value
#not required if this will happen anyway (in this example it does)
setattr(self,data_name,[])
#preparation activities
datafile = open(filename_csv, 'r')
datareader = csv.reader(datafile)
#do required work using a temporary local variable
temp = []
for row in datareader:
temp.append(row)
#copy the temporary local variable into the dynamically named one
setattr(self, data_name, temp)
#tidy up activities
datafile.close()
print("finished importing data")
Make sure you take a look at eumiro's answer, which takes a better, more compact and more Pythonic approach to your specific problem using with and list(). However, the above should hopefully make it clear how you could be using setattr() in a wider variety of cases.
Related
I am writing a function that exports variables as a dictionary to an external file.
The problem comes when calling that function from another script. I think it has something to do with the globals() parameter.
import sys
import os
mydict = {} #'initialising" the an empty dictionary to be used locally in the function below
def writeToValues(name):
fileName = os.path.splitext(os.path.basename(sys.argv[0]))[0]
valuePrint=open("values.py","a")
def namestr(obj,namespace):
return[name for name in namespace if namespace[name] is obj]
b = namestr(name, globals())
c = "".join(str(x) for x in b)
mydict[(c)] = name
valuePrint.write(fileName)
valuePrint.write("=")
valuePrint.write(str(mydict))
valuePrint.write("\n")
valuePrint.close()
return mydict
a = 2
b = 3
writeToValues(a)
writeToValues(b)
I get the following result:
Main Junkfile={'a': 2, 'b': 3}
note the word Main Junkfile is the name of the script I ran as that is what the function first does, to get the name of the file and use that to name the dictionary.
Now help me as I cannot generate the same if I import the function from another script.
Another problem is that running the script twice generates the values in steps.
Main Junkfile={'a': 2}
Main Junkfile={'b': 3, 'a': 2}
I cannot change the file open mode from append to write since I want to store values from other scripts, too.
this is not perfect but might help as an example:
import sys
import os
mydict = {}
def namestr(obj,namespace):
return[name for name in namespace if namespace[name] is obj]
def writeto(name):
fout = 'values.py'
filename = os.path.splitext(os.path.basename(sys.argv[0]))[0]
with open (fout, 'a') as f:
b = namestr(name, globals())
c = "".join(str(x) for x in b)
mydict[(c)] = name
data = filename + '=' + str(mydict) + '\n'
f.write(data)
return mydict
a = 2
b = 3
if __name__ == '__main__':
writeto(a)
writeto(b)
First of all, to get the current executing script name, or rather the module that called your function you'll have to pick it up from the stack trace. Same goes for globals() - it will execute in the same context of writeToValues() function so it won't be picking up globals() from the 'caller'. To remedy that you can use the inspect module:
import inspect
import os
def writeToValues(name):
caller = inspect.getmodule(inspect.stack()[1][0])
caller_globals = caller.__dict__ # use this instead of globals()
fileName = os.path.splitext(os.path.basename(caller.__file__))[0]
# etc.
This will ensure that you get the name of the module that imported your script and is calling writeToValues() within it.
Keep in mind that this is a very bad idea if you intend to write usable Python files - if your script name has spaces (like in your example) it will write a variable name with spaces, which will further result in syntax error if you try to load the resulting file into a Python interpreter.
Second, why in the name of all things fluffy are you trying to do a reverse lookup to find a variable name? You are aware that:
a = 2
b = 2
ab = 5
writeToValues(b)
will write {"ab": 2}, and not {"b": 2} making it both incorrect in intent (saves the wrong var) as well as in state representation (saves a wrong value), right? You should pass a variable name you want to store/update instead to ensure you're picking up the right property.
The update part is more problematic - you need to update your file, not just merely append to it. That means that you need to find the line of your current script, remove it and then write a new dict with the same name in its place. If you don't expect your file to grow to huge proportions (i.e. you're comfortable having it partially in the working memory), you could do that with:
import os
import inspect
def writeToValues(name):
caller = inspect.getmodule(inspect.stack()[1][0])
caller_globals = caller.__dict__ # use this instead of globals()
caller_name = os.path.splitext(os.path.basename(caller.__file__))[0]
# keep 'mydict' list in the caller space so multiple callers can use this
target_dict = caller_globals['mydict'] = caller_globals.get('mydict', {})
if name not in caller_globals: # the updated value no longer exists, remove it
target_dict.pop(name, None)
else:
target_dict[name] = caller_globals[name]
# update the 'values.py':
# optionaly check if you should update - if values didn't change no need for slow I/O
with open("values.py", "a+") as f:
last_pos = 0 # keep the last non-update position
while True:
line = f.readline() # we need to use readline() for tell() accuracy
if not line or line.startswith(caller_name): # break at the matching line or EOF
break
last_pos = f.tell() # new non-update position
append_data = f.readlines() # store in memory the rest of the file content, if any
f.seek(last_pos) # rewind to the last non-update position
f.truncate() # truncate the rest of the file
f.write("".join((caller_name, " = ", str(target_dict), "\n"))) # write updated dict
if append_data: # write back the rest of the file, if truncated
f.writelines(append_data)
return target_dict
Otherwise use a temp file to write everything as you read it, except for the line matching your current script, append the new value for the current script, delete the original and rename the temp file to values.py.
So now if you store the above in, say, value_writter.py and use it in your script my_script.py as:
import value_writter
a = 2
b = 3
value_writter.write_to_values("a")
value_writter.write_to_values("b")
a = 5
value_writter.write_to_values("a")
# values.py contains: my_script = {"a": 5, "b": 3}
Same should go for any script you import it to. Now, having multiple scripts edit the same file without a locking mechanism is an accident waiting to happen, but that's a whole other story.
Also, if your values are complex the system will break (or rather printout of your dict will not look properly). Do yourself a favor and use some proper serialization, even the horrible pickle is better than this.
I have dumped a couple of variables I need to store in a json object as shown here-
def write_json():
variable = {}
variable['p_id'] = p_id
variable['s_id'] = s_id
variable['t_id'] = t_id
variable['b_start'] = b_start
variable['b_end'] = b_end
variable['wsp_after'] = wsp_after
variable['h_id'] = h_id
variable['word'] = word
variable['norm_word'] = norm_word
variable['lemma'] = lemma
variable['pos'] = pos
variable['ner'] = ner
variable['dep'] = dep
variable['quote'] = quote
variable['ch_id'] = ch_id
with open('variable.json', 'w') as fp:
json.dump(variable,fp, sort_keys = True)
I want to write a piece of code that would automatically get these variables assigned appropriately as would happen if you wrote from 'X.py' import *
This is what I tried-
def get_variables():
with open('variable.json', 'r') as fp:
variable = json.load(fp)
for key in variable:
global key
key = variable[key]
However, when I tried to print the original value-
print t_id
NameError: name 't_id' is not defined
What do I need to do to achieve what I want rather than cumbersomely typing
t_id = variable['t_id']
.
.
.
For every variable I have dumped?
Your current code doesn't work because the following lines just bind (and rebind, for the later values in the loop) a global variable named 'key':
global key
key = variable[key]
While I'm not sure your design is really a good one, if you're sure you want all the values from your json file as global variables, I'd replace the loop with:
globals().update(variable)
The globals() function returns a dictionary that contains all the current module's global variables. Unlike the dictionary you get from locals() (which may be a copy of the local namespace, not the actual implementation of it), you can modify the globals() dictionary and use the variables like normal afterwards.
Here's the solution -
with open('variable.json', 'r') as fp:
variable = json.load(fp)
for key in variable.keys():
exec(key + " = variable['" + key + "']")
The problem before was that each 'key' was a string, not an object.
Using the exec function successfully parses each string into an object and using it as given above gets the results as was asked in the question.
I need to create a function that creates a list from a named text file and can be given any name.
def createalist(filename,listname):
with open(filename, mode="r") as myfile:
listname=myfile.read().splitlines()
myfile.close()
When I type
createalist("snames.txt",mynewlist) or variants with brackets etc., I get name newarray is not defined
This is just a simple misunderstanding of how variables work in Python.
What I am assuming you want to do is this:
def create_a_list(filename):
with open(filename, mode="r") as my_file:
my_list = my_file.read().splitlines()
my_file.close()
return my_list
anyName = create_a_list("snakes.txt")
The differences in my code versus yours is that I return the list and assign it to a variable rather than trying to pass a variable name/callback in.
When you want to give it any variable name, simply assign the variable equal to that function and make sure that the function returns the value you want the variable to have.
I'm trying to store many variables in a file. I've tried JSON, pickle and shelve but they all seem to only take one variable
import shelve
myShelve = shelve.open('my.shelve')
myShelve.update(aasd,
basd,
casd,
dasd,
easd,
fasd,
gasd,
hasd,
iasd,
jasd)
myShelve.close()
And pickle
import pickle
with open("vars.txt", "wb") as File:
pickle.dumps(aasd,
basd,
casd,
dasd,
easd,
fasd,
gasd,
hasd,
iasd,
jasd,
File)
The errors I'm getting are along the lines of
TypeError: update() takes at most 2 positional arguments (11 given)
and
TypeError: pickle.dumps() takes at most 2 positional argument (11 given)
I'm not sure if there's any other way of storing variables except using a database, but that's a bit over what I'm currently capable of I'd say.
You can only pickle one variable at a time, but it can be a dict or other Python object. You could store your many variables in one object and pickle that object.
import pickle
class Box:
pass
vars = Box()
vars.x = 1
vars.y = 2
vars.z = 3
with open("save_vars.pickle", "wb") as f:
f.write(pickle.dumps(vars))
with open("save_vars.pickle", "rb") as f:
v = pickle.load(f)
assert vars.__dict__ == v.__dict__
using pickle, you dump one object at a time. Each time you dump to the file, you add another "record".
import pickle
with open("vars.txt", "wb") as File:
for item in (aasd,
basd,
casd,
dasd,
easd,
fasd,
gasd,
hasd,
iasd,
jasd)
pickle.dump(item,File)
Now, on when you want to get your data back, you use pickle.load to read the next "record" from the file:
import pickle
with open('vars.txt') as fin:
aasd = pickle.load(fin)
basd = pickle.load(fin)
...
Alternatively, depending on the type of data, assuming the data is stuff that json is able to serialize, you can store it in a json list:
import json
# dump to a string, but you could use json.dump to dump it to a file.
json.dumps([aasd,
basd,
casd,
dasd,
easd,
fasd,
gasd,
hasd,
iasd,
jasd])
EDIT: I just thought of a different way to store your variables, but it is a little weird, and I wonder what the gurus think about this.
You can save a file that has the python code of your variable definitions in it, for example vars.py which consists of simple statements defining your values:
x = 30
y = [1,2,3]
Then to load that into your program, just do from vars import * and you will have x and y defined, as if you had typed them in.
Original normal answer below...
There is a way using JSON to get your variables back without redefining their names, but you do have to create a dictionary of variables first.
import json
vars={} # the dictionary we will save.
LoL = [ range(5), list("ABCDE"), range(5) ]
vars['LOList'] = LoL
vars['x'] = 24
vars['y'] = "abc"
with open('Jfile.txt','w') as myfile:
json.dump(vars,myfile,indent=2)
Now to load them back:
with open('Jfile.txt','r') as infile:
D = json.load(infile)
# The "trick" to get the variables in as x,y,etc:
globals().update(D)
Now x and y are defined from their dictionary entries:
print x,y
24 abc
There is also an alternative using variable-by-variable definitions. In this way, you don't have to create the dictionary up front, but you do have to re-name the variables in proper order when you load them back in.
z=26
w="def"
with open('Jfile2.txt','w') as myfile:
json.dump([z,w],myfile,indent=2)
with open('Jfile2.txt','r') as infile:
zz,ww = json.load(infile)
And the output:
print zz,ww
26 def
I have this class that consists of 3 functions. Each function is in charge of one part of the whole process.
.load() loads up two files, re-formats their content and writes them to two new files.
.compare() takes two files and prints out their differences in a specific format.
.final() takes the result of .compare() and creates a file for every set of values.
Please ignore the Frankenstein nature of the logic as it is not my main concern at the moment. I know it can be written a thousand times better and that's fine by me for now as i am still new to Python and programing in general. I do have some theoretical experience but very limited technical practice and that is something i am working on.
Here is the code:
from collections import defaultdict
from operator import itemgetter
from itertools import groupby
from collections import deque
import os
class avs_auto:
def load(self, fileIn1, fileIn2, fileOut1, fileOut2):
with open(fileIn1+'.txt') as fin1, open(fileIn2+'.txt') as fin2:
frame_rects = defaultdict(list)
for row in (map(str, line.split()) for line in fin1):
id, frame, rect = row[0], row[2], [row[3],row[4],row[5],row[6]]
frame_rects[frame].append(id)
frame_rects[frame].append(rect)
for row in (map(str, line.split()) for line in fin2):
id, frame, rect = row[0], row[2], [row[3],row[4],row[5],row[6]]
frame_rects[frame].append(id)
frame_rects[frame].append(rect)
with open(fileOut1+'.txt', 'w') as fout1, open(fileOut2+'.txt', 'w') as fout2:
for frame, rects in sorted(frame_rects.iteritems()):
fout1.write('{{{}:{}}}\n'.format(frame, rects))
fout2.write('{{{}:{}}}\n'.format(frame, rects))
def compare(self, f1, f2):
with open(f1+'.txt', 'r') as fin1:
with open(f2+'.txt', 'r') as fin2:
lines1 = fin1.readlines()
lines2 = fin2.readlines()
diff_lines = [l.strip() for l in lines1 if l not in lines2]
diffs = defaultdict(list)
with open(f1+'x'+f2+'Result.txt', 'w') as fout:
for line in diff_lines:
d = eval(line)
for k in d:
list_ids = d[k]
for i in range(0, len(d[k]), 2):
diffs[d[k][i]].append(k)
for id_ in diffs:
diffs[id_].sort()
for k, g in groupby(enumerate(diffs[id_]), lambda (i, x): i - x):
group = map(itemgetter(1), g)
fout.write('{0} {1} {2}\n'.format(id_, group[0], group[-1]))
def final(self):
with open('hw1load3xhw1load2Result.txt', 'r') as fin:
lines = (line.split() for line in fin)
for k, g in groupby(lines, itemgetter(0)):
fst = next(g)
lst = next(iter(deque(g, 1)), fst)
with open('final/{}.avs'.format(k), 'w') as fout:
fout.write('video0=ImageSource("MovieName\original\%06d.jpeg", {}, {}, 15)\n'.format(fst[1], lst[2]))
Now to my question, how do i make it so each of the functions passes it's output files as values to the next function and calls it?
So for an example:
running .load() should output two files, call the .compare() function passing it those two files.
Then when .compare() is done, it should pass .final() the output file and calls it.
So .final() will open whatever file is passed to it from .compare() and not "test123.txt" as it is defined above.
I hope this all makes sense. Let me know if you need clarification. Any criticism is welcome concerning the code itself. Thanks in advance.
There are a couple of ways to do this, but I would write a master function that calls the other three in sequence. Something like:
def load_and_compare(self, input_file1, input_file2, output_file1, output_file2, result_file):
self.load(input_file1, input_file2, output_file1, output_file2)
self.compare(output_file1, output_file2)
self.final(result_file)
Looking over your code, I think you have a problem in load. You only declare a single dictionary, then load the contents of both files into it and write those same contents out to two files. Because each file has the same content, compare won't do anything meaningful.
Also, do you really want to write out the file contents and then re-read it into memory? I would keep the frame definitions in memory for use in compare after loading rather than reading them back in.
I don't really see a reason for this to be a class at all rather than just a trio of functions, but maybe if you have to read multiple files with mildly varying formats you could get some benefit of using class attributes to define the format while inheriting the general logic.
Do you mean call with the name of the two files? Well you defined a class, so you can just do:
def load(self, fileIn1, fileIn2, fileOut1, fileOut2):
... // do stuff here
// when done
self.compare( fileOut1, fileOut2 )
And so on.
I might be totally off here, but why don't you do it exactly as you're saying?
Just call self.compare() out of your load() method.
You can also add return statements to load() and return a tuple with the files.
Then add a 4th method to your class, which then collects the returned files and pipes them to the compare() method.
Best Regards!
One of the more powerful aspects of Python is that you can return something called a tuple. To answer this in a more generic Python sense consider this code:
>>> def load(file1, file2):
return file1+'.txt',file2+'.txt'
>>> def convert(file1, file2):
return 'converted_'+file1,'converted_'+file2
>>> convert(*load("Java", "C#"))
('converted_Java.txt', 'converted_C#.txt')
Each function takes two named arguments, but the returned tuple of the first can be "unpacked" into the input arguments of the second by adding a * in front of it.