I need to save a complex piece of data:
list = ["Animals", {"Cats":4, "Dogs":5}, {"x":[], "y":[]}]
I was planning on saving several of these lists within the same file, and I was also planning on using the pickle module to save this data. I also want to be able to access the pickled data and add items to the lists in the 2nd dictionary. So after I unpickle the data and edit, the list might look like this:
list = ["Animals", {"Cats":4, "Dogs":5}, {"x"=[1, 2, 3], "y":[]}]
Preferable, I want to be able to save this list (using pickle) in the same file I took that piece of data from. However, if I simply re-pickle the data to the same file (lets say I originally saved it to "File"), I'll end up with two copies of the same list in that file:
a = open("File", "ab")
pickle.dump(list, a)
a.close()
Is there a way to replace the edited list in the file using pickle rather than adding a second (updated) copy? Or, is there another method I should consider for saving this data?
I think you want the shelve module. It creates a file (uses pickle under the hood) that contains the contents of a variable accessible by key (think persistent dictionary).
You could open the file for writing instead of appending -- then the changes would overwrite previous data. This is however a problem if there is more data stored in that file. If what you want really is to selectively replace data in a pickled file, I'm afraid this won't work with pickle. If this is a common operation, check if something like a sqlite database helps you to this end.
Related
Now I use keyword
Append Data
${list}= Create List Test1 Test2
${data}= create list ${list}
Append To Csv File ${File_Path} ${list}
but it cannot specific the data's position that I want to update, In my test case I have to update new data everytimes after finished case to use new data in next case. (I kept the test data is in CSV file)
Looks like you are already making use of CSVLibrary
in this library you have only the following KWS, what we can notice from here is that, we do not have replace CSV line/file anything, hence, we need to come up with our own procedure.
Append To Csv File
Csv File From Associative
Empty Csv File
Read Csv File To Associative
Read Csv File To List
APPROACH#1
In my test case I have to update new data everytimes after finished
case to use new data in next case.
One of the ways which can be employed to solve your problem, is by converting all of the csv file data into list of dicts.
Read the cvs into list of dicts using Read Csv File To
Associative
make a copy of the original list of dicts
Start of Testcase#1
make the modification to the list of dicts, just in case you would like to go back in time for a quick referral
End of Testcase#1
Start of Testcase#2
make and use the modified content of list of dists from Testcase#1
End of Testcase#2
So on for the rest of the test cases.
Here no need to use CSV library.
If we want to create new csv file with new data always then we can use Create File keyword from OperatingSystem library
Create File filename.csv content=content_added_in_csvFile
e.g. Create File ${CURDIR}/Demo.csv content=675432561
If we want to add multiple data in CSV then
Create File ${CURDIR}/Demo.csv content=68868686,85757464,5757474
Here when we will run this code then old file will be replace by new file with provided content .
Hope It will resolve this issue
Which is the best way to store dictionary of strings in file(as they are big) and load it partially in python. Dictionary of strings here means, keyword would be a string and the value would be a list of strings.
Dictionary storing in appended form to check keys, if available not update or else update. Then use keys for post processing.
Usually a dictionary is stored in JSON.
I'll leave here a link:
Convert Python dictionary to JSON array
You could simply write the dictionary to a text file, and then create a new dictionary that only pulls certain keys and values from that text file.
But you're probably best off exploring the json module.
Here's a straighforward way to write a dict called "sample" to a file with the json module:
import json
with open('result.json', 'w') as fp:
json.dump(sample, fp)
On the loading side, we'd need to know more about how you want to choose which keys to load from the JSON file.
The above answers are great, but i hate using JSON, i have had issues with pickle before that corrupted my data, so what i do is, i use numpy's save and load
To save np.save(filename,dict)
to load dict = np.load(filename).item()
really simple and works well, as far as loading partially goes, you could always split the dictionary into multiple smaller dictionaries and save them as individual files, maybe not a very concrete solution but it could work
to split the dictionary you could do something like this
temp_dict = {}
for i,k in enumerate(dict.keys()):
if i%1000 == 0:
np.save("records-"+str(i-1000)+"-"+str(i)+".npy",temp_dict)
temp_dict = {}
temp_dict[k]=dict[k].value()
then for loading just do something like
my_dict={}
all_files = glob.glob("*.npy")
for f in all_files:
dict = np.load(filename).item()
my_dict.update(dict)
If this is for some sort of database type use then save yourself the headache and use TinyDB. It uses JSON format when saving to disc and will provide you the "partial" loading that you're looking for.
I only recommend TinyDB as this seems to be the closest to what you're looking to achieve, maybe try googling for other databases if this isn't your fancy there's TONS of them out there!
I am using tkinter to manage the GUI for a note retrieval program. I can pull my notes by typing a key word and hitting Enter in a text field but I would like to move my dictionary to a file so that my code space is not filled up with a massive dictionary.
I have been looking around but I am not sure how I would go about doing this.
I have the file in my directory. I know I can use open(“filename”, “mode”) to open said file for reading but how do I call each section of the notes.
For example what I do now is just call a keyword from my dictionary and have it write the definition for that keyword to a text box in my GUI. Can I do the same from the file?
How would I go about reading from the file the keyword and returning the definition to a variable or directly to the text box? For now I just need to figure out how to read the data. I think once I know that I can figure out how to write new notes or edit existing notes.
This is how I am set up now.
To call my my function
root.bind('<Return>', kw_entry)
How I return my definition to my text box
def kw_entry(event=None):
e1Current = keywordEntry.get().lower()
if e1Current in notes:
root.text.delete(1.0, END)
root.text.insert(tkinter.END, notes[e1Current])
root.text.see(tkinter.END)
else:
root.text.delete(1.0, END)
root.text.insert(tkinter.END, "Not a Keyword")
root.text.see(tkinter.END)
Sound's like you'd need to load the dictionary to memory at init time, and use it like a normal dictionary.
I am assuming your dictionary is a standard python dict of strings, so I recommend using the python json lib.
Easiest way to do this is to export the dictionary as json once to a file using something like:
with open(filename, 'w') as fp:
json.dump(dictionary, fp)
and then change your code to load the dict at init time using:
with open(filename) as fp:
dictionary = json.load(fp)
Alternatively, if your data is more complex than text, you can use python shelve which is a persistent, dictionary-like object to which you can pass any pickle-able object. Note that shelve has its drawbacks so read the attached doc.
sqlitedict is a project providing a persistent dictionary using sqlite in the background. You can use it like a normal dictionary e.g. by assigning arbitrary (picklable) objects to it.
If you access an element from the dictionary, only the value you requested is loaded from disk.
I am working with cPickle for the purpose to convert the structure data into datastream format and pass it to the library. The thing i have to do is to read file contents from manually written file name "targetstrings.txt" and convert the contents of file into that format which Netcdf library needs in the following manner,
Note: targetstrings.txt contains latin characters
op=open("targetstrings.txt",'rb')
targetStrings=cPickle.load(op)
The Netcdf library take the contents as strings.
While loading a file it stuck with the following error,
cPickle.UnpicklingError: invalid load key, 'A'.
Please tell me how can I rectify this error, I have googled around but did not find an appropriate solution.
Any suggestions,
pickle is not for reading/writing generic text files, but to serialize/deserialize Python objects to file. If you want to read text data you should use Python's usual IO functions.
with open('targetstrings.txt', 'r') as f:
fileContent = f.read()
If, as it seems, the library just wants to have a list of strings, taking each line as a list element, you just have to do:
with open('targetstrings.txt', 'r') as f:
lines=[l for l in f]
# now in lines you have the lines read from the file
As stated - Pickle is not meant to be used in this way.
If you need to manually edit complex Python objects taht are to be read and passed as Python objects to another function, there are plenty of other formats to use - for example XML, JSON, Python files themselves. Pickle uses a Python specific protocol, that while note being binary (in the version 0 of the protocol), and not changing across Python versions, is not meant for this, and is not even the recomended method to record Python objects for persistence or comunication (although it can be used for those purposes).
I want to copy the contents of a JSON file in another JSON file, with Python
Any ideas ?
Thank you :)
Given the lack of research effort, I normally wouldn't answer, but given the poor suggestions in comments, I'll bite and give a better option.
Now, this largely depends on what you mean, do you wish to overwrite the contents of one file with another, or insert? The latter can be done like so:
with open("from.json", "r") as from, open("to.json", "r") as to:
to_insert = json.load(from)
destination = json.load(to)
destination.append(to_insert) #The exact nature of this line varies. See below.
with open("to.json", "w") as to:
json.dump(to, destination)
This uses python's json module, which allows us to do this very easily.
We open the two files for reading, then open the destination file again in writing mode to truncate it and write to it.
The marked line depends on the JSON data structure, here I am appending it to the root list element (which could not exist), but you may want to place it at a particular dict key, or somesuch.
In the case of replacing the contents, it becomes easier:
with open("from.json", "r") as from, open("to.json", "w") as to:
to.write(from.read())
Here we literally just read the data out of one file and write it into the other file.
Of course, you may wish to check the data is JSON, in which case, you can use the JSON methods as in the first solution, which will throw exceptions on invalid data.
Another, arguably better, solution to this could also be shutil's copy methods, which would avoid actually reading or writing the file contents manually.
Using the with statement gives us the benefit of automatically closing our files - even if exceptions occur. It's best to always use them where we can.
Note that in versions of Python before 2.7, multiple context managers are not handled by the with statement, instead you will need to nest them:
with open("from.json", "r") as from:
with open("to.json", "r+") as to:
...