We Keep Coding

Find all strings in text file fitting either of two formats

So I know similar questions have been asked before, but every method I have tried is not working...
Here is the ask: I have a text file (which is a log file) that I am parsing for any occurrence of "app.task2". The following are the 2 scenarios that can occur (As they appear in the text file, independent of my code):
Scenario 1:
Mar 23 10:28:24 dasd[116] <Notice>: app.task2.refresh:556A2D:[
{name: ApplicationPolicy, policyWeight: 50.000, response: {Decision: Can Proceed, Score: 0.45}}
] sumScores:68.785000, denominator:96.410000, FinalDecision: Can Proceed FinalScore: 0.713463}
Scenario 2:
Mar 23 10:35:56 dasd[116] <Notice>: 'app.task2.refresh:C6C2FE' CurrentScore: 0.636967, ThresholdScore: 0.410015 DecisionToRun:1
The problem I am facing is that my current code below, I am not getting the entire log entry for the first case, and it is only pulling the first line in the log, not the remainder of the log entry, and it appears to be stopping at the new line escape character, which is occurring after ":[".
My Code:
all = []
with open(path_to_log) as f:
for line in f:
if "app.task2" in line:
all.append(line)
print all
How can I get the entire log entry for the first case? I tried stripping escape characters with no luck. From here I should be able to parse the list of results returned for what I truly need, but this will help! ty!
OF NOTE: I need to be able to locate these types of log entries (which will then give us either scenario 1 or scenario 2) by the string "app.task2". So this needs to be incorporated, like in my example...

Before adding the line to all, check if it ends with [. If it does, keep reading and merge the lines until you get to ].
import re
all = []
with open(path_to_log) as f:
for line in f:
if "app.task2" in line:
if re.search(r'\[\s*$', line): # start of multiline log message
for line2 in f:
line += line2
if re.search(r'^\s*\]', line2): # end of multiline log message
break
all.append(line)
print(all)

You are iterating over each each line individually which is why you only get the first line in scenario 1.
Either you can add a counter like this:
all = []
count = -1
with open(path_to_log) as f:
for line in f:
if count > 0:
all.append(line)
if count == 1:
tmp = all[-count:]
del all[-count:]
all.append("\n".join(tmp))
count -= 1
continue
if "app.task2" in line:
all.append(line)
if line.endswith('[\n'):
count = 3
print all
In this case i think Barmar solution would work just as good.
Or you can (preferably) when storing the log file have some distinct delimiter between each log entry and just split the log file by this delimiter.

I like #Barmar's solution with nested loops on the same file object, and may use that technique in the future. But prior to seeing I would have done it with a single loop, which may or may not be more readable:
all = []
keep = False
for line in open(path_to_log,"rt"):
if "app.task2" in line:
all.append(line)
keep = line.rstrip().endswith("[")
elif keep:
all.append(line)
keep = not line.lstrip().startswith("]")
print (all)
or, you can print it nicer with:
print(*all,sep='\n')

Replacement for isAlpha() to include underscores?

I am processing data using Python3 and I need to read a results file that looks like this:
ENERGY_BOUNDS
1.964033E+07 1.733253E+07 1.491825E+07 1.384031E+07 1.161834E+07 1.000000E+07 8.187308E+06 6.703200E+06
6.065307E+06 5.488116E+06 4.493290E+06 3.678794E+06 3.011942E+06 2.465970E+06 2.231302E+06 2.018965E+06
GAMMA_INTERFACE
0
EIGENVALUE
1.219034E+00
I want to search the file for a specific identifier (in this case ENERGY_BOUNDS), begin reading the numeric values after this identifier but not the identifier itself, and stop when I reach the next identifier. However, my problem is that I was using isAlpha to find the next identifier, and some of them contain underscores. Here is my code:
def read_data_from_file(file_name, identifier):
with open(file_name, 'r') as read_obj:
list_of_results = []
# Read all lines in the file one by one
for line in read_obj:
# For each line, check if line contains the string
if identifier in line:
# If yes, read the next line
nextValue = next(read_obj)
while(not nextValue.strip().isalpha()): # Keep on reading until next identifier appears
list_of_results.extend(nextValue.split())
nextValue = next(read_obj)
return(list_of_results)
I think I need to use regex, but I am stuck regarding how to phrase it. Any help would be much appreciated!

take = False
with open('path/to/input') as infile:
for line in input:
if line.strip() == "ENERGY_BOUNDS":
take = True
continue # we don't actually want this line
if all(char.isalpha() or char=="_" for char in line.strip()): # we've hit the next section
take = False
if take:
print(line) # or whatever else you want to do with this line

Here's an option for you.
Just iterate over the file until you hit the identifier.
Then iterate over it in another for loop until the next identifier causes a ValueError.
def read_data_from_file(file_name, identifier):
with open(file_name, 'r') as f:
list_of_results = []
for line in f:
if identifier in line:
break
for line in f:
try:
list_of_results.extend(map(float, line.split()))
except ValueError:
break
return list_of_results

You can use this regex: ^[A-Z]+(?:_[A-Z]+)*$
Additionally, you can modify the regex to match strings of custom length, like this: ^[A-Z]{2,10}+(?:_[A-Z]+)*$, where {2, 10} is {MIN, MAX} length:
You can find this demo here: https://regex101.com/r/9jESAH/35
See this answer for more details.

Here is a simple function to verify a string has alpha, uppercase and lowercase and underscore:
RE_PY_VAR_NAME="^[a-zA-Z_]+$"
def isAlphaUscore(s:str) -> bool:
assert not s is None, "s cannot be None"
return re.search(RE_PY_VAR_NAME, s)

extract the dimensions from the head lines of text file

Please see following attached image showing the format of the text file. I need to extract the dimensions of data matrix indicated by the first line in the file, here 49 * 70 * 1 for the case shown by the image. Note that the length of name "gd_fac" can be varying. How can I extract these numbers as integers? I am using Python 3.6.

Specification is not very clear. I am assuming that the information you want will always be in the first line, and always be in parenthesis. After that:
with open(filename) as infile:
line = infile.readline()
string = line[line.find('(')+1:line.find(')')]
lst = string.split('x')
This will create the list lst = [49, 70, 1].
What is happening here:
First I open the file (you will need to replace filename with the name of your file, as a string. The with ... as ... structure ensures that the file is closed after use. Then I read the first line. After that. I select only the parts of that line that fall after the open paren (, and before the close paren ). Finally, I break the string into parts, with the character x as the separator. This creates a list that contains the values in the first line of the file, which fall between parenthesis, and are separated by x.

Since you have mentioned that length of 'gd_fac' van be variable, best solution will be using Regular Expression.
import re
with open("a.txt") as fh:
for line in fh:
if '(' in line and ')' in line:
dimension = re.findall(r'.*\((.*)\)',line)[0]
break
print dimension
Output:
'49x70x1'

What this does is it looks for "gd_fac"
then if it's there is removes all the unneeded stuff and replaces it with just what you want.
with open('test.txt', 'r') as infile:
for line in infile:
if("gd_fac" in line):
line = line.replace("gd_fac", "")
line = line.replace("x", "*")
line = line.replace("(","")
line = line.replace(")","")
print (line)
break
OUTPUT: "49x70x1"

Issues using .startswith() for a specific location in a string

I have a text file, which has many lines of data in it. I need to check each line of this text file and process the data contained within the line accordingly (i.e. save to a separate, tabulated .txt for analysis)
The text file is in the following format:
Number 1 or 0 (denoting relevance of data)
An ID for each line (referring to what the data is)
The data itself (contained in rest of line)
So this is what two example lines may look like:
1 ID:K-95 list of data
0 ID:D-56 list of other data
Such that the first line had relevant data to ID K-95 and the second had irrelevant data to ID D-56.
I want to parse the text file, and sort the data contained within each line based on the relevance (0 or 1) and the data ID. I.e. save each line with the same ID in order of relevance (first all the lines with 1 and then with 0). Lines can have the same ID, but different data. Lines are also always of a fixed length.
To do this I came up with:
idtag = input('Enter ID:')
with open("example.txt", 'r') as f:
for line in f.readlines():
if line.startswith('1') and line.startswith(idtag, 5, 3):
print line
Having trouble with this however. Specifically around the second condition after the and operator. I can print/select lines based on whether there is a 0 or 1, no problem. However, using the .startswith() method with a defined position seems to return nothing: no error, no printing - it simply executes and returns nothing.
Any ideas? Maybe a better way of parsing this data to meet my objective?

The start and end are interpreted as absolute positions (specifically: end is not interpreted relative to start) for str.startswith:
str.startswith(prefix[, start[, end]])
Return True if string starts with the prefix, otherwise return False. prefix can also be a tuple of prefixes to look for. With optional start, test string beginning at that position. With optional end, stop comparing string at that position.
So instead of
line.startswith(idtag, 5, 3)
you need to use
line.startswith(idtag, 5, 5+4)
The two parameters are equivalent to slicing notation:
line[5: 5+4].startswith(idtag)
For example:
>>> a = 'abcdefg'
>>> a.startswith('c', 2, 1)
False
>>> a[2:1]
''
>>> a.startswith('c', 2)
True
>>> a[2:]
'cdefg'
>>> a.startswith('c', 2, 3)
True
>>> a[2:3]
'c'

I realise there's already an answer, but as an alternative you could also just check if idtag exists in the line:
idtag = input('Enter ID:')
with open("example.txt", 'r') as f:
for line in f.readlines():
if line.startswith('1') and idtag in line:
print line

list comprehension not obeying if statement

This stub of my program
with open(fi) as f:
lines = f.readlines()
lines = [line.split('!')[0] for line in lines if not line.startswith('!') and line.strip()] # Removing line and inline comments
for line in lines:
print line, len(line), bool(not line.startswith('!') and line.strip())
gives me output such as
conduction-guess double_array optional 68 True
valence character optional 68 True
valence-guess double_array optional 68 True
68 False
saturated-bond-length-factor double required 68 True
Shouldn't the line whose bool value is False not be included? I'm lost.
I thought it might be short-circuiting, but flipping the expression doesn't help either.
To clarify, I want my list to be a list of the lines that end with 'True' in the above section of code.

The first not line.startswith('!') and line.strip() is operating on a different set of line values than the second one. The list comprehension causes lines to be replaced with a list of the first !-delimited field of each string in the original value of lines, and it is this new lines that the code then prints out. For example, if lines originally contains the string " !foo", this string will pass the conditional in the comprehension, but only the part before the ! — a single space — will be saved into the new lines list, and a single space will not cause line.strip() to return true.

You could see something like that if you had a line in your file which had an exclamation point somewhere other than in the first column. e.g.:
line = " ! "

line.strip() returns a new string, it's not a boolean.
Updated code:
with open(fi) as f:
lines = [ line.strip() for line in f ]
# remove exclamation points
lines = [ line.split('!')[0] for line in lines
if not line.startswith('!')
]
# 3rd col will always be True
for line in lines:
print line, len(line), bool(not line.startswith('!'))