We Keep Coding

how to read a large text file in a data frame from a .txt file in python

I have a large text file which has names and long paragraphs of statements made by several different people. the file format is .txt, I am trying to separate the name and the statement into two different columns of a data frame.
Data is in this format-
Harvey: I’m inclined to give you a shot. But what if I decide to go the other way?
Mike: I’d say that’s fair. Sometimes I like to hang out with people who aren’t that bright, you know, just to see how the other half lives.
Mike in the club
(mike speaking to jessica.)
Jessica: How are you mike?
Mike: good!
.....
....
and so on
the length of text file is 4million.
in the output I need a dataframe with one name column having the name of speaker and another statement column with that persons respective statement.

if: the format is always "name: one-liner-no-colon"
you could try:
df = pd.read_csv('untitled.txt',sep=': ', header=None)
or go manually:
f = open("untitled.txt", "r")
file_contents = []
current_name = ""
current_dialogue = ""
for line in f:
splitted_line = line.split(": ")
if len(splitted_line) > 1:
# you are on a row with name: on it
# first stop the current dialogue - save it
if current_name:
file_contents.append([current_name, current_dialogue])
# then update the name encountered
current_name = splitted_line.pop(0)
current_dialogue = ""
current_dialogue += ": ".join(splitted_line)
# add the last dialogue line
file_contents.append([current_name, current_dialogue])
f.close()
df = pd.DataFrame(file_contents)
df

If you read the file line-by-line, you can use something like this to split the speaker from the spoken text, without using regex.
def find_speaker_and_text_from_line(line):
split = line.split(": ")
name = split.pop(0)
rest = ": ".join(split)
return name, rest

How to replace a specific data according to lines not by string? Python, Files

import linecache
f = open("database.txt" , "r+")
password = input("What is your password? ")
for loop in range(3):
line = f.readline()
data = line.split(",")
if data[1] == password:
print(data[1]) #These are too help me understand the code
print(data[0])
b = data[0]
newpass = "Hi"
a = data[1]
line1 = linecache.getline("database.txt" ,loop+1)
print(line)
print("Password Valid\n")
write = (line1.replace(a, newpass))
write = f.write(line1.replace(a, newpass))
f.close()
I have a file which stores data separated by columns. Each new line is a different users data:
username,password,Recovery1,Answer1,Recovery2,Answer2,Recovery3,Answer3,Recovery4,Answer4,
Recovery5,Answer5,o,o,o,o,o,o,o,o,o,o,
happy,bye,o,o,o,o,o,o,o,o,o,o,
bye,happy,o,o,o,o,o,o,o,o,o,o,
THIS IS THE WANTED FILE:
username,password,Recovery1,Answer1,Recovery2,Answer2,Recovery3,Answer3,Recovery4,Answer4,
Recovery5,Answer5,o,o,o,o,o,o,o,o,o,o,
happy,Hi,o,o,o,o,o,o,o,o,o,o,
bye,happy,o,o,o,o,o,o,o,o,o,o,
I want the user to enter their password and then be able to change it. Which is the second column (e.g. data[1]). The program should then change the users password to a different one. "Hi" in this case. However, IT MUST NOT AFFECT ANOTHER USERS USERNAME AS THE PASSWORDS AND USERNAMES COULD BE THE SAME. EG. changing bye to HI in third line, should not change bye in the fourth line to Hi.
The code can append the new line, but the previous line still exists. Is this the write way to go about it? Could you help? The code should be able to change other lines password according to the users input.
Thanks in advance.

How do I add a user inputted number into my text file list?

I am not able to add a number to my list that i have in a text file and don't know how to.
Code so far:
def add_player_points():
# Allows the user to add a points onto the players information.
L = open("players.txt","r+")
name = raw_input("\n\tPlease enter the name of the player whose points you wish to add: ")
for line in L:
s = line.strip()
string = s.split(",")
if name == string[0]:
opponent = raw_input("\n\t Enter the name of the opponent: ")
points = raw_input("\n\t Enter how many points you would like to add?: ")
new_points = string[7] + points
L.close()
This is a sample of a key in the text file. There are about 100 in the file:
Joe,Bloggs,J.bloggs#anemailaddress.com,01269 512355, 1, 0, 0, 0,
^
The value that i would like this number to be added to is the 0 besides the number already in there, indicated by an arrow below it. The text file is called players.txt as shown.
A full code answer would be helpful.

I didn't like what i wrote earlier, and the use case isn't optimal for fileinput. I took a similar piece of code from the sources and suited it for your needs.
Notice that for each line you modify, you are re-writing an entire file. I strongly suggest changing the way you handle data if performance is a concern.
This code works tho.
from tempfile import mkstemp
from shutil import move
from os import remove, close
def add_player_points():
file_path = "test.txt"
name = raw_input("\n\tPlease enter the name of the player whose points you wish to add: ")
#Create temp file
fh, abs_path = mkstemp()
with open(abs_path,'w') as new_file:
with open(file_path) as old_file:
for line in old_file:
stripped_line = line.strip()
split_string = stripped_line.split(",")
print name == split_string[0]
if name == split_string[0]:
opponent = raw_input("\n\t Enter the name of the opponent: ")
points = raw_input("\n\t Enter how many points you would like to add?: ")
temp = int(split_string[5]) + int(points) # fool proofing the code
split_string[5] = str(temp)
stripped_line = ','.join(split_string)# line you shove back into the file.
print stripped_line
new_file.write(stripped_line +'\n')
else:
new_file.write(line)
close(fh)
#Remove original file
remove(file_path)
#Move new file
move(abs_path, file_path)
Search and replace a line in a file in Python
Editing specific line in text file in python
You wouldn't expect it to be that big of an issue, but it is.
Another tip: might wanna check the module csv - it might be smarter for file editing than what i showed here.

2 issues, first you're never saving your changes to the file. You need to build the string and then save it at the end with L.write("your new string"). Second, you need to cast the points to ints before adding them, change
new_points = string[7] + points
to
new_points = int(string[7]) + int(points)
Edit: Fixed the syntax as mentioned in the comments

How do I search for text within a tab delimited file and print this information?

I need to search for something in a tab delimited text file. The user is supposed to input both the file and the thing that needs searching for. The programme is then supposed to return the whole line that the user inputted word is in. I have got two models so far because I've been coming at this problem from different angles. The first programme goes as follows:
import csv
searchfile = raw_input ('Which file do you want to search? ')
try:
input_file = open (searchfile, 'rU')
except:
print "Invalid file. Please enter a correct file"
csv_file_object = csv.reader(open(searchfile, 'rb'))
header = csv_file_object.next()
data=[]
for row in csv_file_object:
data.append(row)
searchA = raw_input ('which author?')
author_search = data[0::,0] == searchA
if author_search in searchfile:
print author_search
The problem with the first programme is that this error pops up:
TypeError: list indices must be integers, not tuple
I therefore attempted this method:
import csv
searchfile = raw_input ('Which file do you want to search? ')
try:
input_file = open (searchfile, 'rU')
except:
print "Invalid file. Please enter a correct file"
with open(searchfile) as f:
reader = csv.reader(f, delimiter="\t")
d = list(reader)
searchtype = raw_input ('Search on author or journal/conference or [Q = quit]')
if searchtype == 'author':
searchdataA = raw_input ("Input author name")
if searchdataA in input_file:
print line
elif searchtype == 'journal' or 'conference' or 'journal/conference':
searchdataJ = raw_input ("input journal/conference name")
if searchdataJ in d:
print line
elif searchtype == 'Q':
print "Program left"
else:
print "please choose either author or journal/conference"
This is unable to get beyond inputting the search parameters.
Any help on where to go with either programme would be much appreciated, or if I'm completely on the wrong track then links to useful material would be great.

I think you're making this a bit more complicated than it needs to be. Since you want to print the whole line that the target word appeared on, you don't really need the CSV module. You're not doing any of the sophisticated parsing it is capable of.
searchfile = raw_input ('Which file do you want to search? ')
searchA = raw_input ('which author?')
with open(searchfile) as infile:
for line in infile:
if searchA in line:
print(' '.join(line.split()))
break # remove this if you want to print all matches instead of
# just the first one
Notice that in the when printing the line, I first split the line (which splits on whitespace by default), then rejoin the fields with two spaces between them. I think doing something like this would be a good way to go for you since you're printing tab-separated fields on the console. Reducing that extra space will make your prints a bit easier to read, but using two spaces still makes it easy to distinguish the columns from each other.
You can generalize it by prompting your user for any search term, instead of specifying "author". This may be the way to go, since your second code snippet suggests that you may want to search for other fields, like "journal" or "conference":
target_term = raw_input("Which term or phrase would you like to find?")
Since this method searches in and prints the entire line, there's no need to deal with the separate columns and different kinds of search terms. It just looks at the whole row at once and prints a matching line.

Since you are not really using a different search method, depending on if you are searching for author, journal, conference or journal/conference. So you could actually do a full text search on the line. Therefore it is wise to collect all data you need from user BEFORE processing file, so you can output just the matching lines. If user passes a rather large CSV file, then your way would take up far too much memory.
with open(searchfile, 'r') as f:
for line in f:
if line.find(searchA) > -1:
print line
This way you are looping through the file as fast as possible and prints out all matching lines.
The .find() function returns the index to where in the string he found the match and otherwise -1 if the string was not found. So from the value you could "estimate" on where the match was made, but if you really want to differentiate between author, journal, etc. then you will have to split the line. In my sample i will assume the author field to be the sixth field in the CSV line:
with open(searchfile, 'r') as f:
for line in f:
fields = line.split("\t")
if len(fields) > 5: # check length of fields array
if fields[5].find(searchA) > -1: # search straight in author field
print line # return full line

why not simply
fname = raw_input("Enter Filename")
author = raw_input("Enter Author Name:")
if author in open(fname,"rb").read():
print "match found"
if you want to see the lines you could do
print re.findall(".*%s.*"%(author),open(fname,"rb").read())
as people point out it is better form to do
with open(fname,"rb") as f:
data = print re.findall(".*%s.*"%(author),f.read())
although in CPython it will be garbage collected immediatly so its not really a problem....

The first thing that came to my mind is simply:
def check_file(file_name, author_name):
with open(file_name) as f:
content = f.readlines()
for line in content:
if author_name in line:
print "Found: ", line
Hope it can be useful.

Deleting and inserting a score into a text file form python

I have created a text based game in python 3.3 were users pick a class for their character. I want the game to store three scores so an average can be taken. My problem is that i am unsure how to get the program to search for a name in the file and delete there oldest score, This is what the file where the scores are saved looks like:
Bennie
33
62
94
Josh
82
55
31
Jackie
10
4
3
My current code that sees if they have done the game before and if not writes there score to the file if they have i have got the code to split the lines and read them. It needs to delete the score closet to their name and insert and new score just before the next name but i am unsure how to do this. This is my current code
class_choice = input('Enter Class one, Class two or Class three.')
if class_choice == "One":
text_file = 'class1.txt'
elif class_choice == "Two":
text_file = 'class2.txt'
elif class_choice == "Three":
text_file = 'class3.txt'
else:
False
first_time = input('Is this the first you have completed this game: Yes or No?')
if first_time == 'Yes':
with open(text_file, "a") as file:
file.write("{}\n".format(name))
file.write("0\n")
file.write("0\n")
file.write("{}\n".format(score))
sys.exit()
else:
file = open(text_file, 'r')
lines = file.read().splitlines()

giving a sample python method to add a score to the file (pseudo-code) modify as needed to suit your needs (your requirement is not the most efficient method):
def add_score(user, score, text_file):
lines = text_file.splitlines()
count = len(lines)
user_exists = False
user_index = -1
num_scores = 3
user_num_scores = 0
for i in range(count-1):
line = lines[i]
if user == line:
# user previous scores start after here
user_exists = True
user_index = i
break
if not user_exists:
# user does not exist, create by appending
lines.append(user)
lines.append(str(score))
else: # user exists, fix the scores
j=1
while j <= num_scores:
line = lines[user_index+j]
j += 1
if line.isdigit():
# user score line
user_num_scores +=1
if user_num_scores == num_scores:
for i in range(1,num_scores-1): lines[user_index+i] = lines[user_index+i+1] # shift up
lines[user_index+num_scores] = str(score) # add the latest score
else: # just append/insert the score, as the previous scores are less than num_scores
lines.insert(user_index+user_num_scores, str(score))
return "\n".join(lines) # return the new text_file back
use like this:
text = file.read()
updated_text = add_score('UserName', 32, text)
Note the function given will not change the file it will just operate on the given file contents (as text_file argument). This is on purpose since if the function itself manipulated the files it would limit its use, since one can read or write the file in any convenient instant for the whole application. So this function only operates on strings. This is not the most efficient method, for example one can use a file per user and a much simpler format, but since this what is asked for this answer covers only that.
When you have finished adding scores and manipulating the files, you can update the file by writing the updated_text back to it. (Probably if the file is already opened in read mode, you will have to close it and re-open it in write mode).
To write (update) the file with the new contents use sth like this:
file.write(updated_text)
for example see here for python file I/O operations

It is not possible to change a single line inside a file. You can only append content at the end of the file or override the entire content.
If you wat to store the scores on a file, you need to read the file line by line storing each line in an auxiliar variable (concatenating lines) and when you reach the line you need to modify, concatenate the modified line into the auxiliar variable. Then keep reading the file line by line and storing it into the auxiliar variable.
When done, write down the content of the auxiliar variable in the file.

Why dont you try creating an .ini file and store your data in the attributes under sections. It's framework suits your requirements. It kind-of works like an xml file, but since your input and output parameters are limited (as you mentioned; 3 scores), it seems like the best option to opt for. Simply go through ini file manipulation using python in python docs and you will know what to do. Cheers !

My problem is that i am unsure how to get the program to search for a name in the file and delete there oldest score
I would create several .txt (or .dat) files with the name "< class >_stats[.txt]" (without the spaces, obviously). From there:
class_choice = raw_input("Choose your class")
# if needing stats
f = open("%s_stats.txt" % class_choice, "r+")
lines = f.readlines()
f.close()
stats = [float(i) for i in lines] # i.e. [3, 5.5, 4]
# rest of game
# overwrite with new stats
new_stat = get_new_stat()
f = open("%s_stats.txt" % class_choice, "w")
f.write("\n".join([str(i) for i in stats]))
However, I would recommend just keeping the stats, you may want them later, and text is cheap. Instead of reading all lines, simply open the file for appending, read the last three, and append the new stat to the end when you get your new stat, i.e.
f = open("%s_stats.txt")
lines = f.readlines[-3:] # reads last 3
f.close()
# stuff
f = open("%s_stats.txt", "a")
f.write(get_new_stat())
f.close()