When I use pickle, it works fine and I can dump any load.
The problem is if I close the program and try to dump again, it replaces the old file data with the new dumping. Here is my code:
import pickle
import os
import time
dictionary = dict()
def read():
with open('test.txt', 'rb') as f:
a = pickle.load(f)
print(a)
time.sleep(2)
def dump():
chs = raw_input('name and number')
n = chs.split()
dictionary[n[0]] = n[1]
with open('test.txt', 'wb') as f:
pickle.dump(dictionary, f)
Inpt = raw_input('Option : ')
if Inpt == 'read':
read()
else:
dump()
When you open a file in w mode (or wb), that tells it to write a brand-new file, erasing whatever was already there.
As the docs say:
The most commonly-used values of mode are 'r' for reading, 'w' for writing (truncating the file if it already exists), and 'a' for appending…
In other words, you want to use 'ab', not 'wb'.
However, when you append new dumps to the same file, you end up with a file made up of multiple separate values. If you only call load once, it's just going to load the first one. If you want to load all of them, you need to write code that does that. For example, you can load in a loop until EOFError.
Really, it looks like what you're trying to do is not to append to the pickle file, but to modify the existing pickled dictionary.
You could do that with a function that loads and merges all of the dumps together, like this:
def Load():
d = {}
with open('test.txt', 'rb') as f:
while True:
try:
a = pickle.load(f)
except EOFError:
break
else:
d.update(a)
# do stuff with d
But that's going to get slower and slower the more times you run your program, as you pile on more and more copies of the same values. To do that right you need to load the old dictionary, modify that, and then dump the modified version. And for that, you want w mode.
However, a much better way to persist a dictionary, at least if the keys are strings, is to use dbm (if the values are also strings) or shelve (otherwise) instead of a dictionary in the first place.
Opening a file in "wb" mode truncates the file -- that is, it deletes the contents of the file, and then allows you to work on it.
Usually, you'd open the file in append ("ab") mode to add data at the end. However, Pickle doesn't support appending, so you'll have to save your data to a new file (come up with a different file name -- ask the user or use a command-line parameter such as -o test.txt?) each time the program is run.
On a related topic, don't use Pickle. It's unsafe. Consider using JSON instead (it's in the standard lib -- import json).
Related
myFile = open('high scores.py', 'w')
if player1_total > player2_total :
myFile.write(player1_total)
else :
myFile.write(player2_total)
myFile.close
The file write method only expects strings (or bytestrings, if the file is open in binary mode). The max function can save you a conditional, too. Try something like:
with open('high_scores.py', 'w') as myFile:
myFile.write(str(max(player1_total, player2_total)))
You would then be able to read this back with
with open('high_scores.py') as f:
high_score = int(f.read())
Note that the use of the with statements ensures that files are always correctly closed no matter what the outcome of the with block.
Personally, since the file isn't a Python program file I'd use a different extension in its name. For storing a larger set of values consider using the shelve module.
myFile = open('high scores.py', 'w')
if player1_total > player2_total :
myFile.write(str(player1_total))
else :
myFile.write(str(player2_total))
myFile.close()
The issue is that you need to cast the integer to string before writing. easiest way is str(player2_total)
Also close the file once done documentation
When you’re done with a file, call f.close() to close it and free up
any system resources taken up by the open file.
But a concise way to write it is give in this answer.
More info on using the context manager with open(): can be found in PEP-0343 & read up on this blog post as well
cast your values to strings before writing to file:
myFile.write(str(player1_total))
Started Python a week ago and I have some questions to ask about reading and writing to the same files. I've gone through some tutorials online but I am still confused about it. I can understand simple read and write files.
openFile = open("filepath", "r")
readFile = openFile.read()
print readFile
openFile = open("filepath", "a")
appendFile = openFile.write("\nTest 123")
openFile.close()
But, if I try the following I get a bunch of unknown text in the text file I am writing to. Can anyone explain why I am getting such errors and why I cannot use the same openFile object the way shown below.
# I get an error when I use the codes below:
openFile = open("filepath", "r+")
writeFile = openFile.write("Test abc")
readFile = openFile.read()
print readFile
openFile.close()
I will try to clarify my problems. In the example above, openFile is the object used to open file. I have no problems if I want write to it the first time. If I want to use the same openFile to read files or append something to it. It doesn't happen or an error is given. I have to declare the same/different open file object before I can perform another read/write action to the same file.
#I have no problems if I do this:
openFile = open("filepath", "r+")
writeFile = openFile.write("Test abc")
openFile2 = open("filepath", "r+")
readFile = openFile2.read()
print readFile
openFile.close()
I will be grateful if anyone can tell me what I did wrong here or is it just a Pythong thing. I am using Python 2.7. Thanks!
Updated Response:
This seems like a bug specific to Windows - http://bugs.python.org/issue1521491.
Quoting from the workaround explained at http://mail.python.org/pipermail/python-bugs-list/2005-August/029886.html
the effect of mixing reads with writes on a file open for update is
entirely undefined unless a file-positioning operation occurs between
them (for example, a seek()). I can't guess what
you expect to happen, but seems most likely that what you
intend could be obtained reliably by inserting
fp.seek(fp.tell())
between read() and your write().
My original response demonstrates how reading/writing on the same file opened for appending works. It is apparently not true if you are using Windows.
Original Response:
In 'r+' mode, using write method will write the string object to the file based on where the pointer is. In your case, it will append the string "Test abc" to the start of the file. See an example below:
>>> f=open("a","r+")
>>> f.read()
'Test abc\nfasdfafasdfa\nsdfgsd\n'
>>> f.write("foooooooooooooo")
>>> f.close()
>>> f=open("a","r+")
>>> f.read()
'Test abc\nfasdfafasdfa\nsdfgsd\nfoooooooooooooo'
The string "foooooooooooooo" got appended at the end of the file since the pointer was already at the end of the file.
Are you on a system that differentiates between binary and text files? You might want to use 'rb+' as a mode in that case.
Append 'b' to the mode to open the file in binary mode, on systems
that differentiate between binary and text files; on systems that
don’t have this distinction, adding the 'b' has no effect.
http://docs.python.org/2/library/functions.html#open
Every open file has an implicit pointer which indicates where data will be read and written. Normally this defaults to the start of the file, but if you use a mode of a (append) then it defaults to the end of the file. It's also worth noting that the w mode will truncate your file (i.e. delete all the contents) even if you add + to the mode.
Whenever you read or write N characters, the read/write pointer will move forward that amount within the file. I find it helps to think of this like an old cassette tape, if you remember those. So, if you executed the following code:
fd = open("testfile.txt", "w+")
fd.write("This is a test file.\n")
fd.close()
fd = open("testfile.txt", "r+")
print fd.read(4)
fd.write(" IS")
fd.close()
... It should end up printing This and then leaving the file content as This IS a test file.. This is because the initial read(4) returns the first 4 characters of the file, because the pointer is at the start of the file. It leaves the pointer at the space character just after This, so the following write(" IS") overwrites the next three characters with a space (the same as is already there) followed by IS, replacing the existing is.
You can use the seek() method of the file to jump to a specific point. After the example above, if you executed the following:
fd = open("testfile.txt", "r+")
fd.seek(10)
fd.write("TEST")
fd.close()
... Then you'll find that the file now contains This IS a TEST file..
All this applies on Unix systems, and you can test those examples to make sure. However, I've had problems mixing read() and write() on Windows systems. For example, when I execute that first example on my Windows machine then it correctly prints This, but when I check the file afterwards the write() has been completely ignored. However, the second example (using seek()) seems to work fine on Windows.
In summary, if you want to read/write from the middle of a file in Windows I'd suggest always using an explicit seek() instead of relying on the position of the read/write pointer. If you're doing only reads or only writes then it's pretty safe.
One final point - if you're specifying paths on Windows as literal strings, remember to escape your backslashes:
fd = open("C:\\Users\\johndoe\\Desktop\\testfile.txt", "r+")
Or you can use raw strings by putting an r at the start:
fd = open(r"C:\Users\johndoe\Desktop\testfile.txt", "r+")
Or the most portable option is to use os.path.join():
fd = open(os.path.join("C:\\", "Users", "johndoe", "Desktop", "testfile.txt"), "r+")
You can find more information about file IO in the official Python docs.
Reading and Writing happens where the current file pointer is and it advances with each read/write.
In your particular case, writing to the openFile, causes the file-pointer to point to the end of file. Trying to read from the end would result EOF.
You need to reset the file pointer, to point to the beginning of the file before through seek(0) before reading from it
You can read, modify and save to the same file in python but you have actually to replace the whole content in file, and to call before updating file content:
# set the pointer to the beginning of the file in order to rewrite the content
edit_file.seek(0)
I needed a function to go through all subdirectories of folder and edit content of the files based on some criteria, if it helps:
new_file_content = ""
for directories, subdirectories, files in os.walk(folder_path):
for file_name in files:
file_path = os.path.join(directories, file_name)
# open file for reading and writing
with io.open(file_path, "r+", encoding="utf-8") as edit_file:
for current_line in edit_file:
if condition in current_line:
# update current line
current_line = current_line.replace('john', 'jack')
new_file_content += current_line
# set the pointer to the beginning of the file in order to rewrite the content
edit_file.seek(0)
# delete actual file content
edit_file.truncate()
# rewrite updated file content
edit_file.write(new_file_content)
# empties new content in order to set for next iteration
new_file_content = ""
edit_file.close()
So I basically just want to have a list of all the pixel colour values that overlap written in a text file so I can then access them later.
The only problem is that the text file is having (set([ or whatever written with it.
Heres my code
import cv2
import numpy as np
import time
om=cv2.imread('spectrum1.png')
om=om.reshape(1,-1,3)
om_list=om.tolist()
om_tuple={tuple(item) for item in om_list[0]}
om_set=set(om_tuple)
im=cv2.imread('RGB.png')
im=cv2.resize(im,(100,100))
im= im.reshape(1,-1,3)
im_list=im.tolist()
im_tuple={tuple(item) for item in im_list[0]}
ColourCount= om_set & set(im_tuple)
File= open('Weedlist', 'w')
File.write(str(ColourCount))
Also, if I run this program again but with a different picture for comparison, will it append the data or overwrite it? It's kinda hard to tell when just looking at numbers.
If you replace these lines:
im=cv2.imread('RGB.png')
File= open('Weedlist', 'w')
File.write(str(ColourCount))
with:
import sys
im=cv2.imread(sys.argv[1])
open(sys.argv[1]+'Weedlist', 'w').write(str(list(ColourCount)))
you will get a new file for each input file and also you don't have to overwrite the RGB.png every time you want to try something new.
Files opened with mode 'w' will be overwritten. You can use 'a' to append.
You opened the file with the 'w' mode, write mode, which will truncate (empty) the file when you open it. Use 'a' append mode if you want data to be added to the end each time
You are writing the str() conversion of a set object to your file:
ColourCount= om_set & set(im_tuple)
File= open('Weedlist', 'w')
File.write(str(ColourCount))
Don't use str to convert the whole object; format your data to a string you find easy to read back again. You probably want to add a newline too if you want each new entry to be added on a new line. Perhaps you want to sort the data too, since a set lists items in an ordered determined by implementation details.
If comma-separated works for you, use str.join(); your set contains tuples of integer numbers, and it sounds as if you are fine with the repr() output per tuple, so we can re-use that:
with open('Weedlist', 'a') as outputfile:
output = ', '.join([str(tup) for tup in sorted(ColourCount)])
outputfile.write(output + '\n')
I used with there to ensure that the file object is automatically closed again after you are done writing; see Understanding Python's with statement for further information on what this means.
Note that if you plan to read this data again, the above is not going to be all that efficient to parse again. You should pick a machine-readable format. If you need to communicate with an existing program, you'll need to find out what formats that program accepts.
If you are programming that other program as well, pick a format that other programming language supports. JSON is widely supported for example (use the json module and convert your set to a list first; json.dump(sorted(ColourCount), fileobj), then `fileobj.write('\n') to produce newline-separated JSON objects could do).
If that other program is coded in Python, consider using the pickle module, which writes Python objects to a file efficiently in a format the same module can load again:
with open('Weedlist', 'ab') as picklefile:
pickle.dump(ColourCount, picklefile)
and reading is as easy as:
sets = []
with open('Weedlist', 'rb') as picklefile:
while True:
try:
sets.append(pickle.load(output))
except EOFError:
break
See Saving and loading multiple objects in pickle file? as to why I use a while True loop there to load multiple entries.
How would you like the data to be written? Replace the final line by
File.write(str(list(ColourCount)))
Maybe you like that more.
If you run that program, it will overwrite the previous content of the file. If you prefer to apprend the data open the file with:
File= open('Weedlist', 'a')
I am trying to create a basic mathematical quiz and need to be able to store the name of the user next to their score. To ensure that I could edit the data dynamically regardless of the length of the user's name or the number of digits in their score, I decided to split up the name and score with a comma and use the split function. I'm new to file handling in python so don't know if I am using the wrong mode ("r+") but when I complete the quiz, my score is not recorded at all, nothing is added to the file. Here is my code:
for line in class_results.read():
if student_full_name in line:
student = line.split(",")
student[1] = correct
line.replace(line, "{},{}".format(student_full_name, student[1]))
else:
class_results.write("{},{}".format(student_full_name, correct))
Please let me know how I can get this system to work. Thank you in advance.
Yes r+ opens the file for both reading and writing and to summarize:
r when the file will only be read
w for only writing (an existing file with the same name will be erased)
a opens the file for appending; any data written to the file is automatically added to the end.
I will recommend instead of comma separation to benifit from json or yaml syntax, it fits better in this case.
scores.json:
{
"student1": 12,
"student2": 798
}
The solution:
import json
with open(filename, "r+") as data:
scores_dict = json.loads(data.read())
scores_dict[student_full_name] = correct # if already exist it will be updated otherwise it will be added
data.seek(0)
data.write(json.dumps(scores_dict))
data.truncate()
scores.yml will looks as follow:
student1: 45
student2: 7986
Solution:
import yaml
with open(filename, "r+") as data:
scores_dict = yaml.loads(data.read())
scores_dict[student_full_name] = correct # if already exist it will be updated otherwise it will be added
data.seek(0)
data.write(yaml.dump(scores_dict, default_flow_style=False))
data.truncate()
to instal yaml python package: pip install pyyaml
Modifying a file in place is generally a poor way to do this. It risks errors causing the resulting file to be half new data, half old, with the split point being corrupted. The usual pattern is to write to a new file, then atomically replace the old file with the new file, so either you have the entire original old file and a partial new file, or the new file, not a mish-mash of both.
Given your example code, here is how you would fix it up to do that:
import csv
import os
from tempfile import NamedTemporaryFile
origfile = '...'
origdir = os.path.dirname(origfile)
# Open original file for read, and tempfile in same directory for write
with open(origfile, newline='') as inf, NamedTemporaryFile('w', dir=origdir, newline='') as outf:
old_results = csv.reader(inf)
new_results = csv.writer(outf)
for name, oldscore in old_results:
if name == student_full_name:
# Found our student, replace their score
new_results.writerow((name, correct))
# The write out the rest of the lines unchanged
new_results.writerows(old_results)
# and we're done
break
else:
new_results.writerow((name, oldscore))
else:
# else block on for loop executes if loop ran without break-ing
new_results.writerow((student_full_name, correct))
# If we got here, no exceptions, so let's keep the new data to replace the old
outf.delete = False
# Atomically replaces the original file with the temp file with updated data
os.replace(outf.name, origfile)
I have been working on the same Battleship game for quite a while now and am getting to the end stages. Now I need to have the game save the top five scores in a text file using the function def saveScore. I then need it to read the file that I just created and load the scores into the Python code using try and except for file open and close. I don't know how to get Python to recognize my variable score because I believe it's only local. Here's what I have. I do not know how to use pickle.
def main():
board=createBoard()
printBoard(board)
s = [[21,22,23,24,25],
[45,55,65,75],
[1,2,3],
[85,86,87],
[5,15],
[46,56]]
playBattleship(s,board)
main()
Using pickle is a lower level way to serialize a python object out to a file, and then read the format back again into an object. If you want a bit of a higher level interface that might be easier for you to use naturally, try looking at the shelve module: http://docs.python.org/library/shelve.html#example
You can treat it like a dictionary, and just append and save your scores. It will save to a file by pickling under the hood.
import shelve
# open a shelve file. writeback=True makes it save
# on the fly
d = shelve.open('temp.file', writeback=True)
if not d.has_key('scores'):
d['scores'] = []
print d['scores']
# []
# add some score values
d['scores'].append(10)
d['scores'].append(20)
d.close()
# next time, open the file again. It will have
# the 'scores' key. Though you should probably check
# for it each time in case its a first run.
d = shelve.open('temp.file', writeback=True)
print d['scores']
#[10, 20]
# sort the list backwards and take the first 5 top scores
topScores = sorted(d['scores'], reverse=True)[:5]
Probably the easiest way for you to do this would be to use Pickle. Using the "load" and "dump" functions you can easily save/load the score object.
http://docs.python.org/library/pickle.html
import pickle
def saveScore(score):
pickle.dump(score, 'topfive2.txt')
def loadScore():
return pickle.load('topfive2.txt')
Reading and writing files in Python is pretty straightforward:
# Opening a file for writing will return the file handle f
f = open('/tmp/workfile', 'w')
# You can then write to the file using the 'write' method
f.write('Hello world!\n')
# To read your data back you can use the 'read' or 'readlines' methods
# Read the entire file
str = f.read()
# Read the file one line at a time
line = f.readline()
# Read the file into a list
list = f.readlines()
If you want to store more data than just the last score, you might consider creating a SQLite3 database. Python has good built-in support for SQLite3. This is a cross-platform filesystem database. The database is just a regular text file on disk but it supports many of the SQL operations you'd expect from a database.