I've had a search around but can't find anything regarding this...
I'm looking for a way to save a dictionary to file and then later be able to load it back into a variable at a later date by reading the file.
The contents of the file don't have to be "human readable" it can be as messy as it wants.
Thanks
- Hyflex
EDIT
import cPickle as pickle
BDICT = {}
## Automatically generated START
name = "BOB"
name_title = name.title()
count = 5
BDICT[name_title] = count
name = "TOM"
name_title = name.title()
count = 5
BDICT[name_title] = count
name = "TIMMY JOE"
name_title = name.title()
count = 5
BDICT[name_title] = count
## Automatically generated END
if BDICT:
with open('DICT_ITEMS.txt', 'wb') as dict_items_save:
pickle.dump(BDICT, dict_items_save)
BDICT = {} ## Wiping the dictionary
## Usually in a loop
firstrunDICT = True
if firstrunDICT:
with open('DICT_ITEMS.txt', 'rb') as dict_items_open:
dict_items_read = dict_items_open.read()
if dict_items_read:
BDICT = pickle.load(dict_items_open)
firstrunDICT = False
print BDICT
Error:
Traceback (most recent call last):
File "C:\test3.py", line 35, in <module>
BDICT = pickle.load(dict_items_open)
EOFError
A few people have recommended shelve - I haven't used it, and I'm not knocking it. I have used pickle/cPickle and I'll offer the following approach:
How to use Pickle/cPickle (the abridged version)...
There are many reasons why you would use Pickle (or its noticable faster variant, cPickle). Put tersely Pickle is a way to store objects outside of your process.
Pickle not only gives you the options to store objects outside your python process, but also does so in a serialized fashion. Meaning, First In, First Out behavior (FIFO).
import pickle
## I am making up a dictionary here to show you how this works...
## Because I want to store this outside of this single run, it could be that this
## dictionary is dynamic and user based - so persistance beyond this run has
## meaning for me.
myMadeUpDictionary = {"one": "banana", "two": "banana", "three": "banana", "four": "no-more"}
with open("mySavedDict.txt", "wb") as myFile:
pickle.dump(myMadeUpDictionary, myFile)
So what just happened?
Step1: imported a module named 'pickle'
Step2: created my dictionary object
Step3: used a context manager to handle the opening/closing of a new file...
Step4: dump() the contents of the dictionary (which is referenced as 'pickling' the object) and then write it to a file (mySavedDict.txt).
If you then go into the file that was just created (located now on your filesystem), you can see the contents. It's messy - ugly - and not very insightlful.
nammer#crunchyQA:~/workspace/SandBox/POSTS/Pickle & cPickle$ cat mySavedDict.txt
(dp0
S'four'
p1
S'no-more'
p2
sS'three'
p3
S'banana'
p4
sS'two'
p5
g4
sS'one'
p6
g4
s.
So what's next?
To bring that BACK into our program we simply do the following:
import pickle
with open("mySavedDict.txt", "rb") as myFile:
myNewPulledInDictionary = pickle.load(myFile)
print myNewPulledInDictionary
Which provides the following return:
{'four': 'no-more', 'one': 'banana', 'three': 'banana', 'two': 'banana'}
cPickle vs Pickle
You won't see many people use pickle these days - I can't think off the top of my head why you would want to use the first implementation of pickle, especially when there is cPickle which does the same thing (more or less) but a lot faster!
So you can be lazy and do:
import cPickle as pickle
Which is great if you have something already built that uses pickle... but I argue that this is a bad recommendation and I fully expect to get scolded for even recommending that! (you should really look at your old implementation that used the original pickle and see if you need to change anything to follow cPickle patterns; if you have legacy code or production code you are working with, this saves you time refactoring (finding/replacing all instances of pickle with cPickle).
Otherwise, just:
import cPickle
and everywhere you see a reference to the pickle library, just replace accordingly. They have the same load() and dump() method.
Warning Warning I don't want to write this post any longer than it is, but I seem to have this painful memory of not making a distinction between load() and loads(), and dump() and dumps(). Damn... that was stupid of me! The short answer is that load()/dump() does it to a file-like object, wheres loads()/dumps() will perform similar behavior but to a string-like object (read more about it in the API, here).
Again, I haven't used shelve, but if it works for you (or others) - then yay!
RESPONSE TO YOUR EDIT
You need to remove the dict_items_read = dict_items_open.read() from your context-manager at the end. The file is already open and read in. You don't read it in like you would a text file to pull out strings... it's storing pickled python objects. It's not meant for eyes! It's meant for load().
Your code modified... works just fine for me (copy/paste and run the code below and see if it works). Notice near the bottom I've removed your read() of the file object.
import cPickle as pickle
BDICT = {}
## Automatically generated START
name = "BOB"
name_title = name.title()
count = 5
BDICT[name_title] = count
name = "TOM"
name_title = name.title()
count = 5
BDICT[name_title] = count
name = "TIMMY JOE"
name_title = name.title()
count = 5
BDICT[name_title] = count
## Automatically generated END
if BDICT:
with open('DICT_ITEMS.txt', 'wb') as dict_items_save:
pickle.dump(BDICT, dict_items_save)
BDICT = {} ## Wiping the dictionary
## Usually in a loop
firstrunDICT = True
if firstrunDICT:
with open('DICT_ITEMS.txt', 'rb') as dict_items_open:
BDICT = pickle.load(dict_items_open)
firstrunDICT = False
print BDICT
Python has the shelve module for this. It can store many objects in a file that can be opened up later and read in as objects, but it's operating system-dependent.
import shelve
dict1 = #dictionary
dict2 = #dictionary
#flags:
# c = create new shelf; this can't overwrite an old one, so delete the old one first
# r = read
# w = write; you can append to an old shelf
shelf = shelve.open("filename", flag="c")
shelf['key1'] = dict1
shelf['key2'] = dict2
shelf.close()
#reading:
shelf = shelve.open("filename", flag='r')
for key in shelf.keys():
newdict = shelf[key]
#do something with it
shelf.close()
You can also use Pickle for this task. Here's a blog post that explains how to do it.
What you are looking for is shelve.
Two functions which create a text file for saving a dictionary and loading a dictionary (which was already saved before) for use again.
import pickle
def SaveDictionary(dictionary,File):
with open(File, "wb") as myFile:
pickle.dump(dictionary, myFile)
myFile.close()
def LoadDictionary(File):
with open(File, "rb") as myFile:
dict = pickle.load(myFile)
myFile.close()
return dict
These functions can be called through :
SaveDictionary(mylib.Members,"members.txt") # saved dict. in a file
members = LoadDictionary("members.txt") # opened dict. of members
Related
I just started studying dictionaries and I encountered the following issue. I have the variable dictionary that is modified various times through some other functions, and every time that happens if dumps itself to a file using the pickle module (obligatory to use pickle). dictionary is originally defined as dictionary = {} so everytime I run the program it cleans the file. I thought of defining it usingpickle.load but it really isn't working. Does anyone know how could I modify it so it loads the file and, just in case its empty it adds the empty dictionary.
Here's the part of the code that is giving me issues:
dictionary = {}
toys = open('toysFile','wb')
pickle.dump(dictionary,toys)
toys.close()
toysRB = open('deporte,','rb')
toysRBLoad = pickle.load(toysRB)
print(toysRB)
You can try it
import pickle
dictionary = {"test": "1"}
# save to pickle file
toys = open('toysFile.pkl','ab')
pickle.dump(dictionary,toys)
toys.close()
# load pickle file
toysRB = open('toysFile.pkl','rb')
toysRBLoad = pickle.load(toysRB)
print(toysRBLoad)
Check to see if the file already exists, and if not, create it:
try:
dictionary = pickle.load(open('toysFile",'rb'))
except Exception as e:
dictionary = {}
toys = open('toysFile','wb')
pickle.dump(dictionary,toys)
(Untested code should work)
I'm trying to create a family tree in python, and I want to create a dictionary of Persons, where the key is the person's first name and the value is a Person object for that name.
I can create the dictionary fine, and I can save it fine using the below code.
import os, ast
myFile = open( FILE, "r+" ) # Opens the file for reading and writing
tree = myFile.read()
if tree == "":
tree = {}
else:
tree = ast.literal_eval(tree)
def save():
myFile.write(str(tree))
myFile.close()
However, when I reload my program and try to read in the dictionary, I get the following error:
File "<unknown>", line 1
{'Charlie': <__main__.Person object at 0x00000000032DB860>}
^
SyntaxError: invalid syntax
I suspect that the evaluator can't recognize the object because once the program closes the pointer no longer exists. Is there a way I can save my dictionary so that I can reload it and have access to all my Person objects' data without losing it each time my program closes?
ast.literal_eval is not meant for evaluating and understanding custom objects. It's used to evaluate strings and literals safely. You need to serialize your data by using something like pickle.
>>> import pickle
>>> class Person(object):
... def __init__(self, name):
... self.name = name
...
>>> persons = {'Charlie': Person('Charlie')}
>>> with open(FILE, "wb") as my_file:
... pickle.dump(persons, my_file)
...
>>> with open(FILE, "rb") as f:
... result = pickle.load(f)
...
...
>>> result
{'Charlie': <__main__.Person object at 0x1598bd0>}
When you have large (or many) objects, you can use cPickle in which the pickling is done in C instead of python thus providing a major speed increase.
If object serialization is something you have not heard about before, please read this.
When I use pickle, it works fine and I can dump any load.
The problem is if I close the program and try to dump again, it replaces the old file data with the new dumping. Here is my code:
import pickle
import os
import time
dictionary = dict()
def read():
with open('test.txt', 'rb') as f:
a = pickle.load(f)
print(a)
time.sleep(2)
def dump():
chs = raw_input('name and number')
n = chs.split()
dictionary[n[0]] = n[1]
with open('test.txt', 'wb') as f:
pickle.dump(dictionary, f)
Inpt = raw_input('Option : ')
if Inpt == 'read':
read()
else:
dump()
When you open a file in w mode (or wb), that tells it to write a brand-new file, erasing whatever was already there.
As the docs say:
The most commonly-used values of mode are 'r' for reading, 'w' for writing (truncating the file if it already exists), and 'a' for appending…
In other words, you want to use 'ab', not 'wb'.
However, when you append new dumps to the same file, you end up with a file made up of multiple separate values. If you only call load once, it's just going to load the first one. If you want to load all of them, you need to write code that does that. For example, you can load in a loop until EOFError.
Really, it looks like what you're trying to do is not to append to the pickle file, but to modify the existing pickled dictionary.
You could do that with a function that loads and merges all of the dumps together, like this:
def Load():
d = {}
with open('test.txt', 'rb') as f:
while True:
try:
a = pickle.load(f)
except EOFError:
break
else:
d.update(a)
# do stuff with d
But that's going to get slower and slower the more times you run your program, as you pile on more and more copies of the same values. To do that right you need to load the old dictionary, modify that, and then dump the modified version. And for that, you want w mode.
However, a much better way to persist a dictionary, at least if the keys are strings, is to use dbm (if the values are also strings) or shelve (otherwise) instead of a dictionary in the first place.
Opening a file in "wb" mode truncates the file -- that is, it deletes the contents of the file, and then allows you to work on it.
Usually, you'd open the file in append ("ab") mode to add data at the end. However, Pickle doesn't support appending, so you'll have to save your data to a new file (come up with a different file name -- ask the user or use a command-line parameter such as -o test.txt?) each time the program is run.
On a related topic, don't use Pickle. It's unsafe. Consider using JSON instead (it's in the standard lib -- import json).
I have gone through this website and many others but no one seems to give me the simplest possible answer. In the scrip bellow there are 2 different variables that need to be placed into a single pickle (aka 'test1' and 'test2'); but I am wholly unable to get even the simpler one of the two to load. There are no error messages or anything, and it does appear that something is being written to the pickle but then I close the 'program', re open it, try to load the pickle but the value of 'test1' does not change.
The second question is how to save both to the same pickle? at first i tried using the allStuff variable to store both test1 and test2 then dumping allStuff...the dump seems to be a success but loading does jack. Ive tried a variation where you list each file that should be loaded but this just caused a whole lot of errors and caused me to assault my poor old keyboard...
Please Help.
import pickle
class testing():
test1 = 1000
test2 = {'Dogs' : 0, 'Cats' : 0, 'Birds' : 0, 'Mive' : 0}
def saveload():
check = int(input(' 1. Save : 2. Load : 3. Print : 4. Add'))
allStuff = testing.test1, testing.test2
saveFile = 'TestingSaveLoad.data'
if check == 1:
f = open(saveFile, 'wb')
pickle.dump(testing.test1, f)
f.close()
print()
print('Saved.')
testing.saveload()
elif check == 2:
f = open(saveFile, 'rb')
pickle.load(f)
print()
print('Loaded.')
testing.saveload()
elif check == 3:
print(allStuff)
testing.saveload()
else:
testing.test1 += 234
testing.saveload()
testing.saveload()
The pickle.load documentation states:
Read a pickled object representation from the open file object file and return the reconstituted object hierarchy specified therein.
So you would need something like this:
testing.test1 = pickle.load(f)
However, to save and load multiple objects, you can use
# to save
pickle.dump(allStuff, f)
# to load
allStuff = pickle.load(f)
testing.test1, testing.test2 = allStuff
Dump them as a tuple, and when loading, unpack the result back into the two variables.
pickle.dump((testing.test1,testing.test2), f)
and
testing.test1, testing.test2 = pickle.load(f)
Then change the print to print the two items and forget about allStuff, since you would have to keep updating allStuff every time you loaded/reassigned (depending on the type of item you are storing).
print(testing.test1, testing.test2)
I'd also remove the recursive call to saveLoad() and wrap whatever should be repeated in a while loop with an option to exit
if check == 0:
break
You aren't saving the reconstituted pickled object currently. The documentation states that pickle.load() returns the reconstituted object.
You should have something like:
f = open(saveFile, 'rb')
testing.test1 = pickle.load(f)
To save multiple objects, use the approach recommended in this answer:
If you need to save multiple objects, you can simply put them in a single list, or tuple
Also, I recommend using the with keyword to open the file. That will ensure the file is closed even if something goes wrong. An example of a final output:
with open(saveFile, 'wb') as f:
pickle.dump((testing1, testing2), f)
...
with open(saveFile, 'rb') as f:
testing1, testing2 = pickle.load(f) # Implicit unpacking of the tuple
You might also want a while loop instead of the multiple calls to saveload() - it will be a bit cleaner. Note that right now you have no way out of your loop, short of quitting the program.
I have been working on the same Battleship game for quite a while now and am getting to the end stages. Now I need to have the game save the top five scores in a text file using the function def saveScore. I then need it to read the file that I just created and load the scores into the Python code using try and except for file open and close. I don't know how to get Python to recognize my variable score because I believe it's only local. Here's what I have. I do not know how to use pickle.
def main():
board=createBoard()
printBoard(board)
s = [[21,22,23,24,25],
[45,55,65,75],
[1,2,3],
[85,86,87],
[5,15],
[46,56]]
playBattleship(s,board)
main()
Using pickle is a lower level way to serialize a python object out to a file, and then read the format back again into an object. If you want a bit of a higher level interface that might be easier for you to use naturally, try looking at the shelve module: http://docs.python.org/library/shelve.html#example
You can treat it like a dictionary, and just append and save your scores. It will save to a file by pickling under the hood.
import shelve
# open a shelve file. writeback=True makes it save
# on the fly
d = shelve.open('temp.file', writeback=True)
if not d.has_key('scores'):
d['scores'] = []
print d['scores']
# []
# add some score values
d['scores'].append(10)
d['scores'].append(20)
d.close()
# next time, open the file again. It will have
# the 'scores' key. Though you should probably check
# for it each time in case its a first run.
d = shelve.open('temp.file', writeback=True)
print d['scores']
#[10, 20]
# sort the list backwards and take the first 5 top scores
topScores = sorted(d['scores'], reverse=True)[:5]
Probably the easiest way for you to do this would be to use Pickle. Using the "load" and "dump" functions you can easily save/load the score object.
http://docs.python.org/library/pickle.html
import pickle
def saveScore(score):
pickle.dump(score, 'topfive2.txt')
def loadScore():
return pickle.load('topfive2.txt')
Reading and writing files in Python is pretty straightforward:
# Opening a file for writing will return the file handle f
f = open('/tmp/workfile', 'w')
# You can then write to the file using the 'write' method
f.write('Hello world!\n')
# To read your data back you can use the 'read' or 'readlines' methods
# Read the entire file
str = f.read()
# Read the file one line at a time
line = f.readline()
# Read the file into a list
list = f.readlines()
If you want to store more data than just the last score, you might consider creating a SQLite3 database. Python has good built-in support for SQLite3. This is a cross-platform filesystem database. The database is just a regular text file on disk but it supports many of the SQL operations you'd expect from a database.