I am trying to create a file "varstore.dat" (that does not exist prior to running this) that should initially contain the value 0. Then every time I execute the script, I want to increment this value by 1.
So essentially I am trying to create the file one time, read from the file, and then rewrite(or overwrite) the file upon each execution. However, my problem is that each time I run the program, it will always initialize to 0 and output 1. I am trying to rewrite into varstore.dat and for the new value to become the old value the next time I execute the script.
def get_var_value(filename="varstore.dat"):
with open(filename, "a+") as f:
val = int(f.read() or 0) + 1
f.seek(0)
f.truncate()
f.write(str(val))
return val
your_counter = get_var_value()
print("This script has been run {} times.".format(your_counter))
You just need to do f.seek(0) before you get the val.
def get_var_value(filename="varstore.dat"):
with open(filename, "a+") as f:
f.seek(0)
val = int(f.read() or 0) + 1
f.seek(0)
f.truncate()
f.write(str(val))
return val
The original code actually works in Python 2. Python2 opens the file(in a+ mode) at 0 whereas Python3 opens it at the end(hence "append" mode). It appears that the reason it's different is because Python2 is based on C's stdio.h: https://linux.die.net/man/3/fopen
http://bugs.python.org/msg229514
When you open the file in "a+" mode (or any a mode), it puts the file position at the end of the file. To read it again you'd have to seek back to byte zero, as Cory Madden's answer suggests.
However, I don't understand why you're opening in "a+" mode if you're reading first, then overwriting. Just make two open calls.
with open(filename, 'r') as f:
val = int(f.read() or 0) + 1
with open(filename, 'w') as f:
f.write(str(val))
The issue is with the 'a+' mode that's being used to open the file. In Python 3, that mode starts you at the end of the file (where there's nothing left to read).
I'm not sure why it didn't work that way in Python 2. Instead it starts you at position 0 and warps you to the end of the file whenever you try to write to it (at least, that's how it works on my Windows system, the documentation suggests it may be OS dependent).
To make your code work correctly, you should use mode 'r+' to open the file. That will allow you to both read and write the file from the beginning. And in fact, it might let you get rid of the truncate() call, since the new number you're going to be writing will always be at least as long as the number that was already in the file.
Related
Started Python a week ago and I have some questions to ask about reading and writing to the same files. I've gone through some tutorials online but I am still confused about it. I can understand simple read and write files.
openFile = open("filepath", "r")
readFile = openFile.read()
print readFile
openFile = open("filepath", "a")
appendFile = openFile.write("\nTest 123")
openFile.close()
But, if I try the following I get a bunch of unknown text in the text file I am writing to. Can anyone explain why I am getting such errors and why I cannot use the same openFile object the way shown below.
# I get an error when I use the codes below:
openFile = open("filepath", "r+")
writeFile = openFile.write("Test abc")
readFile = openFile.read()
print readFile
openFile.close()
I will try to clarify my problems. In the example above, openFile is the object used to open file. I have no problems if I want write to it the first time. If I want to use the same openFile to read files or append something to it. It doesn't happen or an error is given. I have to declare the same/different open file object before I can perform another read/write action to the same file.
#I have no problems if I do this:
openFile = open("filepath", "r+")
writeFile = openFile.write("Test abc")
openFile2 = open("filepath", "r+")
readFile = openFile2.read()
print readFile
openFile.close()
I will be grateful if anyone can tell me what I did wrong here or is it just a Pythong thing. I am using Python 2.7. Thanks!
Updated Response:
This seems like a bug specific to Windows - http://bugs.python.org/issue1521491.
Quoting from the workaround explained at http://mail.python.org/pipermail/python-bugs-list/2005-August/029886.html
the effect of mixing reads with writes on a file open for update is
entirely undefined unless a file-positioning operation occurs between
them (for example, a seek()). I can't guess what
you expect to happen, but seems most likely that what you
intend could be obtained reliably by inserting
fp.seek(fp.tell())
between read() and your write().
My original response demonstrates how reading/writing on the same file opened for appending works. It is apparently not true if you are using Windows.
Original Response:
In 'r+' mode, using write method will write the string object to the file based on where the pointer is. In your case, it will append the string "Test abc" to the start of the file. See an example below:
>>> f=open("a","r+")
>>> f.read()
'Test abc\nfasdfafasdfa\nsdfgsd\n'
>>> f.write("foooooooooooooo")
>>> f.close()
>>> f=open("a","r+")
>>> f.read()
'Test abc\nfasdfafasdfa\nsdfgsd\nfoooooooooooooo'
The string "foooooooooooooo" got appended at the end of the file since the pointer was already at the end of the file.
Are you on a system that differentiates between binary and text files? You might want to use 'rb+' as a mode in that case.
Append 'b' to the mode to open the file in binary mode, on systems
that differentiate between binary and text files; on systems that
don’t have this distinction, adding the 'b' has no effect.
http://docs.python.org/2/library/functions.html#open
Every open file has an implicit pointer which indicates where data will be read and written. Normally this defaults to the start of the file, but if you use a mode of a (append) then it defaults to the end of the file. It's also worth noting that the w mode will truncate your file (i.e. delete all the contents) even if you add + to the mode.
Whenever you read or write N characters, the read/write pointer will move forward that amount within the file. I find it helps to think of this like an old cassette tape, if you remember those. So, if you executed the following code:
fd = open("testfile.txt", "w+")
fd.write("This is a test file.\n")
fd.close()
fd = open("testfile.txt", "r+")
print fd.read(4)
fd.write(" IS")
fd.close()
... It should end up printing This and then leaving the file content as This IS a test file.. This is because the initial read(4) returns the first 4 characters of the file, because the pointer is at the start of the file. It leaves the pointer at the space character just after This, so the following write(" IS") overwrites the next three characters with a space (the same as is already there) followed by IS, replacing the existing is.
You can use the seek() method of the file to jump to a specific point. After the example above, if you executed the following:
fd = open("testfile.txt", "r+")
fd.seek(10)
fd.write("TEST")
fd.close()
... Then you'll find that the file now contains This IS a TEST file..
All this applies on Unix systems, and you can test those examples to make sure. However, I've had problems mixing read() and write() on Windows systems. For example, when I execute that first example on my Windows machine then it correctly prints This, but when I check the file afterwards the write() has been completely ignored. However, the second example (using seek()) seems to work fine on Windows.
In summary, if you want to read/write from the middle of a file in Windows I'd suggest always using an explicit seek() instead of relying on the position of the read/write pointer. If you're doing only reads or only writes then it's pretty safe.
One final point - if you're specifying paths on Windows as literal strings, remember to escape your backslashes:
fd = open("C:\\Users\\johndoe\\Desktop\\testfile.txt", "r+")
Or you can use raw strings by putting an r at the start:
fd = open(r"C:\Users\johndoe\Desktop\testfile.txt", "r+")
Or the most portable option is to use os.path.join():
fd = open(os.path.join("C:\\", "Users", "johndoe", "Desktop", "testfile.txt"), "r+")
You can find more information about file IO in the official Python docs.
Reading and Writing happens where the current file pointer is and it advances with each read/write.
In your particular case, writing to the openFile, causes the file-pointer to point to the end of file. Trying to read from the end would result EOF.
You need to reset the file pointer, to point to the beginning of the file before through seek(0) before reading from it
You can read, modify and save to the same file in python but you have actually to replace the whole content in file, and to call before updating file content:
# set the pointer to the beginning of the file in order to rewrite the content
edit_file.seek(0)
I needed a function to go through all subdirectories of folder and edit content of the files based on some criteria, if it helps:
new_file_content = ""
for directories, subdirectories, files in os.walk(folder_path):
for file_name in files:
file_path = os.path.join(directories, file_name)
# open file for reading and writing
with io.open(file_path, "r+", encoding="utf-8") as edit_file:
for current_line in edit_file:
if condition in current_line:
# update current line
current_line = current_line.replace('john', 'jack')
new_file_content += current_line
# set the pointer to the beginning of the file in order to rewrite the content
edit_file.seek(0)
# delete actual file content
edit_file.truncate()
# rewrite updated file content
edit_file.write(new_file_content)
# empties new content in order to set for next iteration
new_file_content = ""
edit_file.close()
I'm stuck in a very basic problem of I/O in python. I'd like to insert some line in existing file (called ofe, output file), extracted from an source file (called ife, input file) according to arguments passed by user as stored in an list called lineRange (which has an index idx and values lineNumber).
This is the result:
for ifeidx,ifeline in enumerate(ife,1): #for each line of the input file...
with open(outFile,'r+') as ofe:
for idx,lineNumber in enumerate(lineRange,1): #... check if it's present in desired list of lines...
if (ifeidx == lineNumber): #...if found...
ofeidx = 0
for ofeidx, ofeline in enumerate(ofe,1):
if (ofeidx == idx): #...just scroll the the output file and find which is the exact position in desired list...
ofe.write(ifeline) #...put the desired line in correct order. !!! This is always appending at the end of out file!!!!
break
Problem is, the write() method is always pointing to the end of file, appending the lines instead of inserting them when scrolling the output file.
I really don't understand what's happening since the file is open in read+write (r+) mode, neither append (a) nor read+append (r+a) mode, .
I'm also aware that code will (should) overwrite the output file lines. Additional information are the OS WIndow7, Python version 2.7 and development tool is Eclipse with PyDev 3.7.1.xx
Any suggestion on what I'm doing wrong?
You can start by reading the whole file with readlines(), which will return a list. After that you just need to do list.insert(index, value) and write it again back to the file.
with open(outFile, "r") as f:
data = f.readlines()
data.insert(index, value)
with open(outFile, "w+") as f:
f.write(data)
Of course you should change this approach if you are dealing with a huge file.
By the way, if you are not using the with statement you should close the file in the end.
In python, there are a few flags you can supply when opening a file for operation. I am a bit baffled at finding a combination that allow me to do random write without truncating. The behavior I am looking for is equivalent to C: create it if it doesn't exist, otherwise, open for write (not truncating)
open(filename, O_WRONLY|O_CREAT)
Python's document is confusing (to me): "w" will truncate the file first, "+" is supposed to mean updating, but "w+" will truncate it anyway. Is there anyway to achieve this without resorting to the low-level os.open() interface?
Note: the "a" or "a+" doesn't work either (please correct if I am doing something wrong here)
cat test.txt
eee
with open("test.txt", "a+") as f:
f.seek(0)
f.write("a")
cat test.txt
eeea
Is that so the append mode insist on writing to the end?
You can do it with os.open:
import os
f = os.fdopen(os.open(filename, os.O_RDWR | os.O_CREAT), 'rb+')
Now you can read, write in the middle of the file, seek, and so on. And it creates the file. Tested on Python 2 and 3.
You should try reading the file then open writing mode, as seen here:
with open("file.txt") as reading:
r = reading.read()
with open("file.txt", "w") as writing:
writing.write(r)
According to the discussion Difference between modes a, a+, w, w+, and r+ in built-in open function, the open with a mode will always write to the end of file irrespective of any intervening fseek(3) or similar.
If you only want to use python built-in function. I guess the solution is to first check if the file exist, and then open with r+ mode.
For Example:
import os
filepath = "test.txt"
if not os.path.isfile(filepath):
f = open(filepath, "x") # open for exclusive creation, failing if the file already exists
f.close()
with open(filepath, "r+") as f: # random read and write
f.seek(1)
f.write("a")
You need to use "a" to append, it will create the file if it does not exist or append to it if it does.
You cannot do what you want with append as the pointer automatically moves to the end of the file when you call the write method.
You could check if the file exists then use fileinput.input with inplace=True inserting a line on whichever line number you want.
import fileinput
import os
def random_write(f, rnd_n, line):
if not os.path.isfile(f):
with open(f, "w") as f:
f.write(line)
else:
for ind, line in enumerate(fileinput.input(f, inplace=True)):
if ind == rnd_n:
print("{}\n".format(line) + line, end="")
else:
print(line, end="")
http://linux.die.net/man/3/fopen
a+
Open for reading and appending (writing at end of file). The file is created if it does not exist. The initial file position for reading is at the beginning of the file, but output is always appended to the end of the file.
fileinput makes a f.bak copy of the file you pass in and it is deleted when the output is closed. If you specify a backup extension backup=."foo" the backup file will be kept.
I have two versions of similar code, one works, the other doesn't:
version 1
#some code
with open('file', 'w') as f:
f.writelines(new_lines)
with open('file') as f:
i = 0
for line in f:
i = i + 1
if i != 5 and i != 18 and i != 27:
row.append(line)
version 2
# some code
with open('file', 'w') as f:
f.writelines(new_lines)
i = 0
for line in f:
i = i + 1
if i != 5 and i != 18 and i != 27:
row.append(line)
When executing the version 2 code, I got an error which says that the file is not open. I got confused because obviously the code is still in the with statement. Does the file get closed automatically right after the f.writelines() method?
The issue you're having has to do with the mode you're using to open the file. In the first example, you first open it for writing with the "w" mode. Then you close it a reopen it with the default "r" mode. This makes a difference!
To start with, you're not allowed to read from a file that is open with the regular "w" mode. You can use "w+" to allow both reading and writing, or "r+" if you don't want it to truncate the file immediately after you open it.
The other issue is the location you're at in the file. When you write your data to the file, you'll end up at the end of the file's data, and so just reading it from there will not get you anything. You will need to call f.seek(0) to go back to the start of the file, so you can read it again.
But, it's a bit silly to write the file and then read back the lines you just wrote (unless there's other complicated logic involved). I'd suggest making your for line in f loop be for line in new_lines instead, which avoids reading the file unnecessarily.
You're iterating over the file while it's in writing mode. Change it to with open('file', 'r+') as f:
After f.writelines(new_lines), you also need to go back to the beginning of the file in order to iterate through each line. Put f.seek(0) in before the loop
The code I have
i = 0
while i < len(newsymbolslist):
time = 102030
data = 895.233
array = [time], [data]
with open('StockPrice.csv', 'wb') as file:
file_writer = csv.writer(file)
file_writer.writerow(array)
file.close()
i += 1
I'm fairly new to Python so I'm not 100% sure why the previous code only enters data into the top row. My guess is that because I'm opening and the file each iteration it doesn't know that its not suppose to override. I know how to fix it in theory (if that is the problem). I'm just having trouble with syntax.
My guess: use the iterations (var i) to count how many rows down the file should write.
with open('StockPrice.csv', 'wb') as f:
file_writer = csv.writer(f)
for s in newsymbolslist:
time = 102030
data = 895.233
array = [time], [data]
file_writer.writerow(array)
Your first guess is correct: Every time you open the file in 'wb' mode, the file is effectively deleted (if it existed) and a new empty file is created. So only the contents written during the last iteration through the while-loop affects the contents of the file.
The solution is to open the file once (before the loop begins).
Note that opening the file with the with-statement guarantees that the file will be closed when Python leaves the with-block. So there is no need to call f.close() yourself.
From the documentation:
The most commonly-used values of mode are 'r' for reading, 'w' for writing (truncating the file if it already exists), and 'a' for appending (...)
If you want to write to the end of an existing file, open it in append mode, with 'a'. (Though in this case, yes, restructuring your loop is the better answer.)