We Keep Coding

An external increment?

I am trying to create a file "varstore.dat" (that does not exist prior to running this) that should initially contain the value 0. Then every time I execute the script, I want to increment this value by 1.
So essentially I am trying to create the file one time, read from the file, and then rewrite(or overwrite) the file upon each execution. However, my problem is that each time I run the program, it will always initialize to 0 and output 1. I am trying to rewrite into varstore.dat and for the new value to become the old value the next time I execute the script.
def get_var_value(filename="varstore.dat"):
with open(filename, "a+") as f:
val = int(f.read() or 0) + 1
f.seek(0)
f.truncate()
f.write(str(val))
return val
your_counter = get_var_value()
print("This script has been run {} times.".format(your_counter))

You just need to do f.seek(0) before you get the val.
def get_var_value(filename="varstore.dat"):
with open(filename, "a+") as f:
f.seek(0)
val = int(f.read() or 0) + 1
f.seek(0)
f.truncate()
f.write(str(val))
return val
The original code actually works in Python 2. Python2 opens the file(in a+ mode) at 0 whereas Python3 opens it at the end(hence "append" mode). It appears that the reason it's different is because Python2 is based on C's stdio.h: https://linux.die.net/man/3/fopen
http://bugs.python.org/msg229514

When you open the file in "a+" mode (or any a mode), it puts the file position at the end of the file. To read it again you'd have to seek back to byte zero, as Cory Madden's answer suggests.
However, I don't understand why you're opening in "a+" mode if you're reading first, then overwriting. Just make two open calls.
with open(filename, 'r') as f:
val = int(f.read() or 0) + 1
with open(filename, 'w') as f:
f.write(str(val))

The issue is with the 'a+' mode that's being used to open the file. In Python 3, that mode starts you at the end of the file (where there's nothing left to read).
I'm not sure why it didn't work that way in Python 2. Instead it starts you at position 0 and warps you to the end of the file whenever you try to write to it (at least, that's how it works on my Windows system, the documentation suggests it may be OS dependent).
To make your code work correctly, you should use mode 'r+' to open the file. That will allow you to both read and write the file from the beginning. And in fact, it might let you get rid of the truncate() call, since the new number you're going to be writing will always be at least as long as the number that was already in the file.

Writing lines to file python

I have a very simple script in python that runs a user defined function (hetero) that joins sequences (strings of text) together over very large files, 2 sequences (rows) at a time. Anyway, As I have it written, it prints out to the screen, but I would like to write all output to a single file.
f = open ("new", "r")
while True:
line1 = f.readline()
line1a = line1.split()
line2 = f.readline()
line2a =line2.split()
if not line2: break
tri="".join ([hetero(b1, b2) for (b1, b2) in zip(line1a[2], line2a[2])])
print line1a[1]+"_"+line1a[0],tri
This simply prints to the terminal the results of the script. So I tried to write the results (from the print command, "line1a[1]+.....") to an another file opened for writing (appended to the end of the script):
out_file = open ("out.txt", "w")
out_file.write(line1a[1]+"_"+line1a[0],tri)
out_file.close()
But of course it does not work. I don't understand why though...Do I need to open the file to write along with the file for reading, so that its outside teh While loop? The thing that is tricky is that the script reads in two lines at a time over the entire file, and prints the ID info and the sequence in a single line, each time -- I want to print all those results to a single file.
This is a simple fix I'm sure, but I don't use python that often and always find the file system i/o difficult to deal with.

Every time you open the file for writing it gets truncated. If you want to append, you can open it at the beginning and keep it open, or open in append mode instead (a instead of w).
Also, you should be using the with statement:
with open('new', 'r') as f, open('out.txt', 'w') as out:
while True:
...
That will call close automatically for you after the block ends.
You can also clean up your "read a pair of lines and split them" code. Instead of while True:
from itertools import izip
pairs = ((l1.split(), l2.split()) for l1, l2 in izip(f, f))
for line1a, line2a in pairs:
tri = ...
Note that you want to use izip instead of zip or it'll just read the whole file into memory right away.

Not sure where you put your out_file code but you likely put that in the loop and it opened and closed every pass. Try something like
with open('out.txt', 'w') as outfile, open("new", "r") as f:
while True:
line1 = f.readline()
line1a = line1.split()
line2 = f.readline()
line2a =line2.split()
if not line2: break
tri="".join ([hetero(b1, b2) for (b1, b2) in zip(line1a[2], line2a[2])])
#print line1a[1]+"_"+line1a[0],tri
out_file.write(line1a[1]+"_"+line1a[0],tri)
EDIT You'll notice I opened the file using a context, I am fan of this because you don't have to worry about closing it later and it seems clearer to me how long the file is open

You are using this code
out_file = open ("out.txt", "w")
out_file.write(line1a[1]+"_"+line1a[0],tri)
out_file.close()
at every iteration. Note the 'w' flag: this means you are opening again the file at each iteration and overwriting it from start. If you want instead to append to it you can use the flag 'a'.
But there is more: this code
out_file = open ("out.txt", "w")
[while ...]
out_file.close()
should be outside the while loop, since you only need to open and close this file once.

You can only open the file inside the loop if you open it like:
out_file = open ("out.txt", "a")
Notice the "a" for appending mode.
If you open it using "w" it will be overwritten every iteration of the loop.
You can check this Python files reference to learn more about.

open file for random write without truncating?

In python, there are a few flags you can supply when opening a file for operation. I am a bit baffled at finding a combination that allow me to do random write without truncating. The behavior I am looking for is equivalent to C: create it if it doesn't exist, otherwise, open for write (not truncating)
open(filename, O_WRONLY|O_CREAT)
Python's document is confusing (to me): "w" will truncate the file first, "+" is supposed to mean updating, but "w+" will truncate it anyway. Is there anyway to achieve this without resorting to the low-level os.open() interface?
Note: the "a" or "a+" doesn't work either (please correct if I am doing something wrong here)
cat test.txt
eee
with open("test.txt", "a+") as f:
f.seek(0)
f.write("a")
cat test.txt
eeea
Is that so the append mode insist on writing to the end?

You can do it with os.open:
import os
f = os.fdopen(os.open(filename, os.O_RDWR | os.O_CREAT), 'rb+')
Now you can read, write in the middle of the file, seek, and so on. And it creates the file. Tested on Python 2 and 3.

You should try reading the file then open writing mode, as seen here:
with open("file.txt") as reading:
r = reading.read()
with open("file.txt", "w") as writing:
writing.write(r)

According to the discussion Difference between modes a, a+, w, w+, and r+ in built-in open function, the open with a mode will always write to the end of file irrespective of any intervening fseek(3) or similar.
If you only want to use python built-in function. I guess the solution is to first check if the file exist, and then open with r+ mode.
For Example:
import os
filepath = "test.txt"
if not os.path.isfile(filepath):
f = open(filepath, "x") # open for exclusive creation, failing if the file already exists
f.close()
with open(filepath, "r+") as f: # random read and write
f.seek(1)
f.write("a")

You need to use "a" to append, it will create the file if it does not exist or append to it if it does.
You cannot do what you want with append as the pointer automatically moves to the end of the file when you call the write method.
You could check if the file exists then use fileinput.input with inplace=True inserting a line on whichever line number you want.
import fileinput
import os
def random_write(f, rnd_n, line):
if not os.path.isfile(f):
with open(f, "w") as f:
f.write(line)
else:
for ind, line in enumerate(fileinput.input(f, inplace=True)):
if ind == rnd_n:
print("{}\n".format(line) + line, end="")
else:
print(line, end="")
http://linux.die.net/man/3/fopen
a+
Open for reading and appending (writing at end of file). The file is created if it does not exist. The initial file position for reading is at the beginning of the file, but output is always appended to the end of the file.
fileinput makes a f.bak copy of the file you pass in and it is deleted when the output is closed. If you specify a backup extension backup=."foo" the backup file will be kept.

Python: Open a file, search then append, if not exist

I am trying to append a string to a file, if the string doesn't exit in the file. However, opening a file with a+ option doesn't allow me to do at once, because opening the file with a+ will put the pointer to the end of the file, meaning that my search will always fail. Is there any good way to do this other than opening the file to read first, close and open again to append?
In code, apparently, below doesn't work.
file = open("fileName", "a+")
I need to do following to achieve it.
file = open("fileName", "r")
... check if a string exist in the file
file.close()
... if the string doesn't exist in the file
file = open("fileName", "a")
file.write("a string")
file.close()

To leave the input file unchanged if needle is on any line or to append the needle at the end of the file if it is missing:
with open("filename", "r+") as file:
for line in file:
if needle in line:
break
else: # not found, we are at the eof
file.write(needle) # append missing data
I've tested it and it works on both Python 2 (stdio-based I/O) and Python 3 (POSIX read/write-based I/O).
The code uses obscure else after a loop Python syntax. See Why does python use 'else' after for and while loops?

You can set the current position of the file object using file.seek(). To jump to the beginning of a file, use
f.seek(0, os.SEEK_SET)
To jump to a file's end, use
f.seek(0, os.SEEK_END)
In your case, to check if a file contains something, and then maybe append append to the file, I'd do something like this:
import os
with open("file.txt", "r+") as f:
line_found = any("foo" in line for line in f)
if not line_found:
f.seek(0, os.SEEK_END)
f.write("yay, a new line!\n")

There is a minor bug in the previous answers: often, the last line in a text file is missing an ending newline. If you do not take that that into account and blindly append some text, your text will be appended to the last line.
For safety:
needle = "Add this line if missing"
with open("filename", "r+") as file:
ends_with_newline = True
for line in file:
ends_with_newline = line.endswith("\n")
if line.rstrip("\n\r") == needle:
break
else: # not found, we are at the eof
if not ends_with_newline:
file.write("\n")
file.write(needle + "\n") # append missing data

Writing to csv file in Python

The code I have
i = 0
while i < len(newsymbolslist):
time = 102030
data = 895.233
array = [time], [data]
with open('StockPrice.csv', 'wb') as file:
file_writer = csv.writer(file)
file_writer.writerow(array)
file.close()
i += 1
I'm fairly new to Python so I'm not 100% sure why the previous code only enters data into the top row. My guess is that because I'm opening and the file each iteration it doesn't know that its not suppose to override. I know how to fix it in theory (if that is the problem). I'm just having trouble with syntax.
My guess: use the iterations (var i) to count how many rows down the file should write.

with open('StockPrice.csv', 'wb') as f:
file_writer = csv.writer(f)
for s in newsymbolslist:
time = 102030
data = 895.233
array = [time], [data]
file_writer.writerow(array)
Your first guess is correct: Every time you open the file in 'wb' mode, the file is effectively deleted (if it existed) and a new empty file is created. So only the contents written during the last iteration through the while-loop affects the contents of the file.
The solution is to open the file once (before the loop begins).
Note that opening the file with the with-statement guarantees that the file will be closed when Python leaves the with-block. So there is no need to call f.close() yourself.

From the documentation:
The most commonly-used values of mode are 'r' for reading, 'w' for writing (truncating the file if it already exists), and 'a' for appending (...)
If you want to write to the end of an existing file, open it in append mode, with 'a'. (Though in this case, yes, restructuring your loop is the better answer.)