We Keep Coding

Extracting specific data from multiple text files and writing them into columns in csv

I'm trying to write a code that will search for specific data from multiple report files, and write them into columns in a single csv.
The report file lines i'm looking for aren't always on the same line, so i'm looking for the data associated on the lines below:
Estimate file: pog_example.bef
Estimate ID: o1_p1
61078 (100.0%) estimated.
And I want to write the data from each text file into columns in a csv as below:
example.bef, o1_p1, 61078 (100.0%) estimated
So far I have this script which will list out the first of my criteria, but I can't figure out how to loop it through to find my second and third lines to populate the second and third columns
from glob import glob
import fileinput
import csv
with open('percentage_estimated.csv', 'w', newline='') as est_report:
writer = csv.writer(est_report)
for line in fileinput.input(glob('*.bef*')):
if 'Estimate file' in line:
writer.writerow([line.split('pog_')[1].strip()])
I'm pretty new to python so any help would be appreciated!

I think I see what you're trying to do, but I'm not sure.
I think your BEF file might look something like this:
a line
another line
Estimate file: pog_example.bef
Estimate ID: o1_p1
61078 (100.0%) estimated.
still more lines
If that's true, then once you find a line with 'Estimate file', you need to take control from the for-loop and start manually iterating the lines because you know which lines are coming up.
This is a very simple example script which opens my mock BEF file (above) and automatically iterates the lines till it finds 'Estimate file'. From there it processes each line specifically, using next(bef_file) to iterate to the next line, expecting them to have the correct text:
import csv
all_rows = []
bef_file = open('input.bef')
for line in bef_file:
if 'Estimate file' in line:
fname = line.split('pog_')[1].strip()
line = next(bef_file)
est_id = line.split('Estimate ID:')[1].strip()
line = next(bef_file)
value = line.strip()
row = [fname, est_id, value]
all_rows.append(row)
break # stop iterating lines in this file
csv_out = open('output.csv', 'w', newline='')
writer = csv.writer(csv_out)
writer.writerow(['File name', 'Est ID', 'Est Value'])
writer.writerows(all_rows)
When I run that I get this for output.csv:
File name,Est ID,Est Value
example.bef,o1_p1,61078 (100.0%) estimated.
If there are blank lines in your data between the lines you care about, manually step over them with next(bef_file) statements.

if anyone wants to see what finally worked for me
from glob import glob
import csv
all_rows = []
with open('percentage_estimated.csv', 'w', newline='') as bef_report:
writer = csv.writer(bef_report)
writer.writerow(['File name', 'Est ID', 'Est Value'])
for file in glob('*.bef*'):
with open(file,'r') as f:
for line in f:
if 'Estimate file' in line:
fname = line.split('pog_')[1].strip()
line = next(f)
est_id = line.split('Estimate ID:')[1].strip()
line = next(f)
line = next(f)
line = next(f)
line = next(f)
line = next(f)
line = next(f)
line = next(f)
value = line.strip()
row = [fname, est_id, value]
all_rows.append(row)
break
writer.writerows(all_rows)

Converting a Text .txt document to CSV .csv Using a Delimiter

I'd like to create a CSV from a TXT file. I have a text file with lines (300 lines+) separated by backslashes. I'd like each line to be a separate row, and each backslash to be a separate new column.
The text file looks like:
example 1\example 2\example 3\example 4
test 1\test 2\test 3\test 4
I'd like the CSV to look like:
Example 1
Example 2
Example 3
Example 4
Test 1
Test 2
Test 3
Test 4
So far I have:
import csv
with open('Report.txt') as report:
report_txt = report.read()
with open('Report.csv','w',newline='') as csvfile:
writer = csv.writer(csvfile)
writer.writerow(report_txt)
I know I need to use \ as a delimiter, but I'm not sure how. Thanks for any help!

Define your delimiter like this (escape the \):
reader = csv.reader(open("Report.csv"), delimiter="\\")
Code:
import csv
with open('Report.txt') as report:
reader = csv.reader(report, delimiter="\\")
with open('Report_output.csv', 'w', newline='') as csvfile:
writer = csv.writer(csvfile)
for line in reader:
writer.writerow(line)

First you got to split the string based on the delimeter. You can achieve this by using the split operator or regex.
import csv
with open('file.txt', 'r') as in_file:
stripped = (line.strip() for line in in_file)
lines = (line.split("\\") for line in stripped if line)
Then pretty much write it to the csv.
with open('report.csv', 'w') as out_file:
writer = csv.writer(out_file)
writer.writerows(lines)
Tweak your code accordingly. The concept is pretty much the same. Note the double backslash is to account for the escape character.

If you are just trying to convert that text into CSV, you can just replace every "\" character with ";" and you'll have a valid CSV file.
Else, if you want to do something with the parsed data before reexporting to CSV, you can read the file line by line and use the split() Method with "\", then rejoin and write line by line, like here:
with open('in.txt') as input_file:
with open('out.csv','a') as output_file:
txt_line = input_file.readline()
while txt_line:
cells = txt_line.split("\\")
# Do something with each cell...
csv_line = ";".join(cells)
output_file.write(csv_line)
txt_line = input_file.readline()

Inserting a comma in between columns in text tile

The problem is I have this text, csv file which is missing commas and I would like to insert it in order to run the file on LaTex and make a table. I have a MWE of a code from another problem which I ran and it did not work. Is it possible someone could guide me on how to change it.
I have used a Python code which provides a blank file, and another one which provides a blank document, and another which removes the spaces.
import fileinput
input_file = 'C:/Users/Light_Wisdom/Documents/Python Notes/test.txt'
output= open('out.txt','w+')
with open('out.txt', 'w+') as output:
for each_line in fileinput.input(input_file):
output.write("\n".join(x.strip() for x in each_line.split(',')))
text file contains more numbers but its like this
0 2.58612
0.00616025 2.20018
0.0123205 1.56186
0.0184807 0.371172
0.024641 0.327379
0.0308012 0.368863
0.0369615 0.322228
0.0431217 0.171899
Outcome
0.049282, -0.0635003
0.0554422, -0.110747
0.0616025, 0.0701394
0.0677627, 0.202381
0.073923, 0.241264
0.0800832, 0.193697
Renewed Attempt:
with open("CSV.txt","r") as file:
new = list(map(lambda x: ''.join(x.split()[0:1]+[","]+x.split()[0:2]),file.readlines()))
with open("New_CSV.txt","w+") as output:
for i in new:
output.writelines(i)
output.writelines("\n")

This can be using .split and .join by splitting the line into a list and then joining the list separated by commas. This enables us to handle several subsequent spaces in the file:
f1 = open(input_file, "r")
with open("out.txt", 'w') as f2:
for line in f1:
f2.write(",".join(line.split()) + "\n")
f1.close()
You can also use csv to handle the writing automatically:
import csv
f1 = open(input_file, "r")
with open("out.txt", 'w') as f2:
writer = csv.writer(f2)
for line in f1:
writer.writerow(line.split())
f1.close()

add a line in the “n” position of a csv loaded

I have not managed to achieve it. but what confuses me the most is how to add an element in a line "n" of my csv, for example I want to add a line in the line 2 of my csv.
mycsv.csv
name,last name
yeison, smith
lola, boa
elmo, spitia
anderson, exneider
juan, ortega
this is my code:
with open('mycsv.csv', 'w') as f:
#I need add "barney, cubides" on position [2] of my csv
f.write("barney, cubides") #not works properly..
how can do it?

You have to write to file after you read it. So read the whole csv and save each line as a list, insert your new line where you want to, and then re-write the whole file.
index_to_insert = 2
new_csv = []
new_line = "barney, cubides\n"
with open("mycsv.csv", "r") as f:
new_csv = f.readlines()
new_csv.insert(index_to_insert, new_line)
with open("mycsv.csv", "w") as f:
for line in new_csv:
f.write(line)
ps. You might want to get rid of the whitespaces before and after the commas in your csv file.

Start reading and writing on specific line on CSV with Python

I have a CSV file that looks like this:
COL_A,COL_B
12345,A=1$B=2$C=3$
How do I read that file and wrote it back to a new file but just the second row (line)?
I want the output file to contain:
12345,A=1$B=2$C=3$
Thanks!

The following reads your csv, extracts the second row, then writes that second row to another file.
with open('file.csv') as file:
second_line = list(file)[1]
with open('out.csv', mode = 'w') as file:
file.write(second_line)

outfile = open(outfilename,"w")
with open(filename) as f:
for line in f:
print >> outfile , line.split()[-1]
outfile.close()
as long as the lines actually look like the line you posted in the OP