We Keep Coding

Reading a string from an opened text file [duplicate]

I have seen these two ways to process a file:
file = open("file.txt")
for line in file:
#do something
file = open("file.txt")
contents = file.read()
for line in contents:
# do something
I know that in the first case, the file will act like a list, so the for loop iterates over the file as if it were a list. What exactly happens in the second case, where we read the file and then iterate over the contents? What are the consequences of taking each approach, and how should I choose between them?

In the first one you are iterating over the file, line by line. In this scenario, the entire file data is not read into the memory at once; instead, only the current line is read into memory. This is useful for handling very large files, and good for robustness if you don't know if the file is going to be large or not.
In the second one, file.read() returns the complete file data as a string. When you are iterating over it, you are actually iterating over the file's data character by character. This reads the complete file data into memory.
Here's an example to show this behavior.
a.txt file contains
Hello
Bye
Code:
>>> f = open('a.txt','r')
>>> for l in f:
... print(l)
...
Hello
Bye
>>> f = open('a.txt','r')
>>> r = f.read()
>>> print(repr(r))
'Hello\nBye'
>>> for c in r:
... print(c)
...
H
e
l
l
o
B
y
e

The second case reads in the contents of the file into one big string. If you iterate over a string, you get each character in turn. If you want to get each line in turn, you can do this:
for line in contents.split('\n'):
# do something
Or you can read in the contents as a list of lines using readlines() instead of read().
with open('file.txt','r') as fin:
lines = fin.readlines()
for line in lines:
# do something

Insert a line into the middle of a text file in Python

I want to insert a line into the middle of a text file in Python, so I tried
with open(erroredFilepath, 'r+t') as erroredFile:
fileContents = erroredFile.read()
if 'insert_here' in fileContents:
insertString.join(fileContents.rsplit('insert_here'))
erroredFile.truncate()
erroredFile.write(insertString)
However, insertString got written at the end of the file. Why?
As an aside, I tried to simply things, by just using strings, instead of files.
'123456789'.join('qwertyuiop'.split('y'))
gives
'qwert123456789uiop'
what happened to the 'y' ?

If you want to write in the middle of the file use the fileinput module.
import fileinput
for line in fileinput.input(erroredFilepath, inplace=True):
print("something", end ="")
from the docs
if the keyword argument inplace=True is passed to fileinput.input() or to the FileInput constructor, the file is moved to a backup file and standard output is directed to the input file (if a file of the same name as the backup file already exists, it will be replaced silently).
Whatever you print will go in the file. So you have to read and print every line and modify whichever you want to replace. Also, when printing existing lines, use end="" as it will prevent print from adding an extra newline.

Although OS-level details of files vary, in general, when you have a file open in r+ mode and do some read or write operation, the "current position" is left after the last read or write.
When you did:
fileContents = erroredFile.read()
the stream erroredFile was read to the end, so the current position is now "at the end".
The truncate function defaults to using the current position as the size to which to truncate. Assume the file is 100 bytes long, so that the current position "at the end" is byte 100. Then:
erroredFile.truncate()
means "make the file 100 bytes long"—which it already is.
The current position remains at the end of the file, so the subsequent write appends.
Presumably you wanted to seek back to the beginning of the file, and/or use truncate(0) (note that just truncate(0) will, at least on Unix-like systems, leave the seek position at the end of the file so that the next write leaves a hole where the original data used to be). You could also be slightly more clever: if you're inserting, just overwrite-and-extend in place (no truncate is required at all).
(Joel Hinz already answered the second question, I see.)

test.txt
a
b
c
d
e
1. Read into a list then overwrite
def match_then_insert(filename, match, content):
lines = open(filename).read().splitlines()
index = lines.index(match)
lines.insert(index, content)
open(filename, mode='w').write('\n'.join(lines))
match_then_insert('test.txt', match='c', content='123')
Result
a
b
123
c
d
e
2. FileInput
from fileinput import FileInput
def match_then_insert(filename, match, content):
for line in FileInput(filename, inplace=True):
if match in line:
line = content + '\n' + line
print(line, end='') # Redirect to the original file
match_then_insert('test.txt', match='c', content='123')
3. seek
def match_then_insert(filename, match, content):
with open(filename, mode='rb+') as f:
while True:
try:
line = f.readline()
except IndexError:
break
line_str = line.decode().splitlines()[0]
if line_str == match:
f.seek(-len(line), 1)
rest = f.read()
f.seek(-len(rest), 1)
f.truncate()
content = content + '\n'
f.write(content.encode())
f.write(rest)
break
match_then_insert('test.txt', match='c', content='123')
Compare
Method
Time/s
Read into a list then overwrite
54.42
FileInput
121.59
seek
3.53
from timeit import timeit
from fileinput import FileInput
def init_txt():
open('test.txt', mode='w').write('\n'.join(['a', 'b', 'c', 'd', 'e']))
def f1(filename='test.txt', match='c', content='123'):
lines = open(filename).read().splitlines()
index = lines.index(match)
lines.insert(index, content)
open(filename, mode='w').write('\n'.join(lines))
def f2(filename='test.txt', match='c', content='123'):
for line in FileInput(filename, inplace=True):
if match in line:
line = content + '\n' + line
print(line, end='')
def f3(filename='test.txt', match='c', content='123'):
with open(filename, mode='rb+') as f:
while True:
try:
line = f.readline()
except IndexError:
break
line_str = line.decode().splitlines()[0]
if line_str == match:
f.seek(-len(line), 1)
rest = f.read()
f.seek(-len(rest), 1)
f.truncate()
content = content + '\n'
f.write(content.encode())
f.write(rest)
break
init_txt()
print(timeit(f1, number=1000))
init_txt()
print(timeit(f2, number=1000))
init_txt()
print(timeit(f3, number=1000))

Not a Python answer but it may widen your horizon. Use sed:
$ cat input.txt
foo
bar
baz
INSERT HERE
qux
quux
$ sed '/INSERT HERE/anew stuff' < input.txt
foo
bar
baz
INSERT HERE
new stuff
qux
quux
The command a will append the text on a new line. If you want to insert the text before the match, use the command i:
$ sed '/INSERT HERE/inew stuff' < input.txt
foo
bar
baz
new stuff
INSERT HERE
qux
quux

Why do you not try a two steps solution? At first, you read and fix the string, at the second step, you rewrite the file. Probably it's not the most efficient algorithm, but I think it works.
with open(erroredFilepath, 'r') as erroredFile:
fileContents = erroredFile.read()
fileContents.replace('insert_here', 'insert_string')
with open(erroredFilePath, 'w') as fixingFile:
fixingFile.write(fileContents)

readlines() cannot read lines after using readline()

The following simple code reads a CSV file and returns the number of lines of the file. As you can see in the output, the file has 501 lines.
>>> import codecs
>>> f = codecs.open("tmp.csv", "r", "utf_8")
>>> print len(f.readlines())
501
But if I insert a readline() before using readlines(), the latter does not reach at the end of the file.
>>> import codecs
>>> f = codecs.open("tmp.csv", "r", "utf_8")
>>> f.readline()
>>> print len(f.readlines())
1
Is there any basic mistake in my code? How can I mix readline() and readlines()? (actually I don't need to mix these two functions in my real program, but I am just curious...)
You can download the file at
https://dl.dropboxusercontent.com/u/16653989/tmp/tmp.csv

This has something to do with the codecs module. Because when you do the same thing with the regular python open statement, it works as expected:
f = open('tmp.csv')
f.readline()
>>> print len(f.readlines())
500

Open() and codecs.open() in Python 2.7 behave strangely different

I have a text file with first line of unicode characters and all other lines in ASCII.
I try to read the first line as one variable, and all other lines as another. However, when I use the following code:
# -*- coding: utf-8 -*-
import codecs
import os
filename = '1.txt'
f = codecs.open(filename, 'r3', encoding='utf-8')
print f
names_f = f.readline().split(' ')
data_f = f.readlines()
print len(names_f)
print len(data_f)
f.close()
print 'And now for something completely differerent:'
g = open(filename, 'r')
names_g = g.readline().split(' ')
print g
data_g = g.readlines()
print len(names_g)
print len(data_g)
g.close()
I get the following output:
<open file '1.txt', mode 'rb' at 0x01235230>
28
7
And now for something completely differerent:
<open file '1.txt', mode 'r' at 0x017875A0>
28
77
If I don't use readlines(), whole file reads, not only first 7 lines both at codecs.open() and open().
Why does such thing happen?
And why does codecs.open() read file in binary mode, despite the 'r' parameter is added?
Upd: This is original file: http://www1.datafilehost.com/d/0792d687

Because you used .readline() first, the codecs.open() file has filled a linebuffer; the subsequent call to .readlines() returns only the buffered lines.
If you call .readlines() again, the rest of the lines are returned:
>>> f = codecs.open(filename, 'r3', encoding='utf-8')
>>> line = f.readline()
>>> len(f.readlines())
7
>>> len(f.readlines())
71
The work-around is to not mix .readline() and .readlines():
f = codecs.open(filename, 'r3', encoding='utf-8')
data_f = f.readlines()
names_f = data_f.pop(0).split(' ') # take the first line.
This behaviour is really a bug; the Python devs are aware of it, see issue 8260.
The other option is to use io.open() instead of codecs.open(); the io library is what Python 3 uses to implement the built-in open() function and is a lot more robust and versatile than the codecs module.

Read files in python

Python learner. So please excuse me.
I am following: http://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files
I want to read a file; here is my file:
# cat test
line1 word1
line2 word2
line3 word3
line4 word4
and here it my code:
>>> f = open ('test')
>>> for line in f:
... print f
...
<open file 'test', mode 'r' at 0xb7729180>
<open file 'test', mode 'r' at 0xb7729180>
<open file 'test', mode 'r' at 0xb7729180>
<open file 'test', mode 'r' at 0xb7729180>
How and why i am getting above output? I was hoping that it will print each line per line.
What am I missing here. looking at the link mentioned above, my syntax seems to be OK but the output is not
Thanks.

During the iteration, you are printing f instead of the line variable.
>>> f = open ('test')
>>> for line in f:
... print line

You are printing the file handle, replace print f with print line:
f = open ('test')
for line in f:
print line

for line in open('test')
print line

You have to read the lines with a readline function. Then it turns each line into a list value. That is when you can do your for statement.

for line in f:
This means 'iterate through the contents of f, assigning each item to the variable named 'line''. Not the converse. f won't change.. it's what you are getting the data from, not the variable you are assigning it to.
That said, the other answer I saw.. from someone whose name starts with 'D'.. is correct: you want to print (line), not print(f). I write that with ()'s because it's a good habit to get into, it makes your code compatible with both Python 2.x and Python 3.x.