We Keep Coding

Rewriting a txt file in python, creating new lines where there is a certain string

I have converted a PDF bank statement to a txt file. Here is a snippet of the .txt file:
15 Apr 20DDOPEN 100.00DDBENNON WATER SRVCS29.00DDBG BUSINESS106.00BPC BOB PETROL MINISTRY78.03BPC BARBARA STREAMING DATA30.50CRPAYPAL Z4J22FR450.00CRPAYNAL AAWDL4Z4J22222KHMG30.0019,028.4917 Apr 20CRCASH IN AT HSBC BANK
What is the easiest way of re-writing the text file in python to create a new line at certain points. i.e. after a number ‘xx.xx’ there in a new date such as ‘xx APR’
For example the text to become:
15 Apr 20DDOPEN 100.00
BENNON WATER SRVCS29.00
DDBG BUSINESS106.00...(etc)
I am just trying to make a PDF more readable and useful when working amongst my other files.
If you know of another PDF to txt python converter which works better, I would also be interested.
Thanks for your help

First step would be getting the text file into Python
with open(“file.txt”) as file:
data = file.read()
This next part, initially, I thought you wouldn't be able to do, but in your example, each part contains a number XX.XX The important thing to notice here is that there is a '.' in each number.
Using Python's string find command, you can iteratively look for that '.' and add a newline character two characters later. You can change my indices below to remove the DD as well if you want.
index = 0
while(index != -1):
index = data.find('.', index)
if index != -1:
data = data[:index+3] + '\n' + data[index+3:]
Then you need to write the new data back to the file.
file = open('ValidEmails.txt','w')
file.write(data)

For the given input the following should work:
import re
counter = 0
l = "15 Apr 20DDOPEN 100.00DDBENNON WATER SRVCS29.00DDBG BUSINESS106.00BPC BOB PETROL MINISTRY78.03BPC BARBARA STREAMING DATA30.50CRPAYPAL Z4J22FR450.00CRPAYNAL AAWDL4Z4J22222KHMG30.0019,028.4917 Apr 20CRCASH IN AT HSBC BANK"
nums = re.finditer("[\d]+[\.][\d]+", l)
for elem in nums:
idx = elem.span()[1] + counter
l = l[:idx] + '\n' + l[idx:]
counter += 1
print(l)
The output is:
15 Apr 20DDOPEN 100.00
DDBENNON WATER SRVCS29.00
DDBG BUSINESS106.00
BPC BOB PETROL MINISTRY78.03
BPC BARBARA STREAMING DATA30.50
CRPAYPAL Z4J22FR450.00
CRPAYNAL AAWDL4Z4J22222KHMG30.0019
,028.4917
Apr 20CRCASH IN AT HSBC BANK
Then you should easily able to write line by line to a file.

Filtering and parsing text over Solar Region Summary files

I was trying to filter some .txt files that are named after a date in YYYYMMDD format and contain some data about active regions in the Sun. I made a code that, given a date in YYYYMMDD format, can list the files that are within a time range which I expect the active region I am looking for to be and parse the information based on that entry. An example of these txts can be seen below and more information about it (if you feel curious) can be seen on SWPC website.
:Product: 0509SRS.txt
:Issued: 2012 May 09 0030 UTC
# Prepared jointly by the U.S. Dept. of Commerce, NOAA,
# Space Weather Prediction Center and the U.S. Air Force.
#
Joint USAF/NOAA Solar Region Summary
SRS Number 130 Issued at 0030Z on 09 May 2012
Report compiled from data received at SWO on 08 May
I. Regions with Sunspots. Locations Valid at 08/2400Z
Nmbr Location Lo Area Z LL NN Mag Type
1470 S19W68 284 0030 Cro 02 02 Beta
1471 S22W60 277 0120 Cso 05 03 Beta
1474 N14W13 229 0010 Axx 00 01 Alpha
1476 N11E35 181 0940 Fkc 17 33 Beta-Gamma-Delta
1477 S22E73 144 0060 Hsx 03 01 Alpha
IA. H-alpha Plages without Spots. Locations Valid at 08/2400Z May
Nmbr Location Lo
1472 S28W80 297
1475 N05W05 222
II. Regions Due to Return 09 May to 11 May
Nmbr Lat Lo
1460 N16 126
1459 S16 110
The code I am using to parse over these txt files is:
import glob
def seeker(noaa_number, t_start, path = None):
'''
This function will open an SRS file
and look for each line if the given AR
(specified by its NOAA number) is there.
If so, this function should grab the
entries and return them.
'''
#defaulting path if none is given
if path is None:
#assigning
path = 'defaultpath'
#listing the items within the directory
files = sorted(glob.glob(path+'*.txt'))
#finding the index in the list of
#the starting time
index = files.index(path+str(t_start)+'SRS.txt')
#looping over each file
for file in files[index: index+20]:
#opening file
f = open(file, 'r')
#reading the lines
text = f.readlines()
#looping over each line in the text
for line in text:
#checking if the noaa number is mentioned
#in the given line
if noaa_number in line:
#test print
print('Original line: ', line)
#slicing the text to get the column values
nbr = line[:4]
Location = line[5:11]
Lo = line[14:18]
Area = line[19:23]
Z = line[24:28]
LL = line[29:31]
NN = line[34:36]
MagType = line[37:]
#test prints
print('nbr: ', nbr)
print('location: ', Location)
print('Lo: ', Lo)
print('Area: ', Area)
print('Z: ', Z)
print('LL: ', LL)
print('NN: ', NN)
print('MagType: ', MagType)
return
I tested this and it is working but I fell a bit dumb for two reasons:
Despite these files being made following a standard, one extra space is all it takes to crash the code considering the way I am slicing the arrays by index. Is there a better option to that?
The information on tables IA and II are not relevant for me so, ideally, I would like to prevent my code to scan them. Since the number of lines on the first column varies, is it possible to tell the code when to stop reading a giving document?
Thanks for your time!

Robustness:
Instead of slicing by absolute position you could split the lines up into a list using the .split() method. This will be robust against extra spaces.
So instead of
Location = line[5:11]
Lo = line[14:18]
Area = line[19:23]
Z = line[24:28]
LL = line[29:31]
NN = line[34:36]
You could use
Location = line.split()[1]
Lo = line.split()[2]
Area = line.split()[3]
Z = line.split()[4]
LL = line.split()[5]
NN = line.split()[6]
If you wanted it to be faster you could split the list once and then just pull the relevant data from the same list rather than splitting it every time:
data = line.split()
Location = data[1]
Lo = data[2]
Area = data[3]
Z = data[4]
LL = data[5]
NN = data[6]
Stopping:
To stop it from continuing reading the file after it's passed the relevant data you could just have something that exits the loop once it no longer finds the noaa_number in the line
# In the file function but before looping through the lines.
started_reading = False ## Set this to false so
## that it doesn't exit
## before it gets to the
## relevant data
for line in text:
if noaa_number in line:
started_reading = True
## Parsing stuff
elif started_reading is True:
break # exits the loop

Filling tabs until the maximum length of column

I have a tab-delimited txt that looks like
11 22 33 44
53 25 36 25
74 89 24 35 and
But there is no "tab" after 44 and 25. So the 1st and 2nd rows have 4 columns, 3rd row has 5 columns.
To rewrite it so that tabs are shown,
11\t22\t33\t44
53\t25\t36\t25
74\t89\t24\t35\tand
I need to have a tool to mass-add tabs where there are no entries.
If the maximum length of column is n (n=5 in the above example), then I want to fill tabs until that nth column for all rows to make
11\t22\t33\t44\t
53\t25\t36\t25\t
74\t89\t24\t35\tand
I tried to do it by notepad++, and python by using replacer code like
map_dict = {'':'\t'}
but it seems I need more logic to do it.

I am assuming your file also contains newlines so it would actually look like this:
11\t22\t33\t44\n
53\t25\t36\t25\n
74\t89\t24\t35\tand\n
If you know for sure that the maximum length of your columns is 5, you can do it like this:
with open('my_file.txt') as my_file:
y = lambda x: len(x.strip().split('\t'))
a = [line if y(line) == 5 else '%s%s\n' % (line.strip(), '\t'*(5 - y(line)))
for line in my_file.readlines()]
# ['11\t22\t33\t44\t\n', '53\t25\t36\t25\t\n', '74\t89\t24\t35\tand\n']
This will add ending tabs until you reach 5 columns. You will get a list of lines that you need to write back to a file (i have 'my_file2.txt' but you can write back to the original one if you want).
with open('my_file2.txt', 'w+') as out_file:
for line in a:
out_file.write(line)

If I understood it correctly, you can achieve this in Notepad++ only using following:
And yes, if you have several files on which you want to perform this, you can record this as a macro and bind it on to key as a shortcut

Complex parsing query

I have a very complex parsing problem. Any thoughts would be appreciated here. I have a test.dat file.The file to be parsed looks like this:
* Number = 40
Time = 0
1 10.13 10 10.11 12 13
.
.
Time = n
1 10 10 10 12.50 13
.
.
There are N time blocks and each block has 40 lines like shown above. What I would like to do is add e.g. the 1st line of first block , then 1st line in block #2 .. and so on to to a new file -test_1.dat. Similarly, 2nd line of every block to test_2.datand so on.The lines in the block should be written as is to the new _n.dat file. Is there any way to do this? The number I have assumed here is 40, so if the * number = 40 there will be 40 lines under each time block.
regards,
Ris

You can read the file in as a list of strings (call it fileList), where each string is a different line:
f = open('filename')
fileList = f.readlines()
Then, remove the "header" part of your file with
fileList.pop(0)
fileList.pop(0)
Then, do
outFileContents = {} # This will be a dict, where number -> content of test_number.dat
for outFileName in range(1,41): #outFileName will be the number going after the _ in your filename
outFileContents[outFileName] = []
for n in range(40): # Counting through the time blocks
currentRowIndex = (42 * n) + outFileName # 42 to account for the Time = and blank row
outFileContents[outFileName].append(fileList[currentRowIndex])
Finally you can loop through outFileContents and write the contents of each value to separate files.

Writing CSV values to file results in values being separated in single characters (Python)

Kinda knew to Python:
I have the following code:
def printCSV(output, values, header):
63 """
64 prints the output data as comma-separated values
65 """
66
67 try:
68 with open(output, 'w') as csvFile:
69 #print headers
70 csvFile.write(header)
71
72 for value in values:
73 #print value, "\n"
74 csvFile.write(",".join(value))
75 csvFile.write("\n")
76 except:
77 print "Error occured while writing CSV file..."
Values is a list constructed somewhat like this:
values = []
for i in range(0,5):
row = "A,%s,%s,%s" % (0,stringval, intval)
values.append(row)
When I open the file created by the above function, I expect to see something like this:
Col1,Col2,Col3,Col4
A,0,'hello',123
A,0,'foobar',42
Instead, I am seeing data like this:
Col1,Col2,Col3,Col4
A,0,'h','e','l','l','o',1,2,3
A,0,'f','o','o','b','a','r',4,2
Anyone knows what is causing this?
I even tried to use fopen and fwrite() directly, still the same problem exists.
Whats causing this?

The problem you're encountering is that you're doing ",".join(value) with value being a string. Strings act like a collection of characters, so the command translates to "Join each character with a comma."
What you could do instead is use a tuple instead of a string for your row values you pass to printCSV, like this:
values = []
for i in range(0,5):
row = ('A', 0, stringval, intval)
values.append(row)