We Keep Coding

How to extract every string between two substring in a paragraph?

After web-scrapping, I get the following:
[<p>xxx<p>, <p>1.apple</p>, <p>aaa</p>, <p>xxxxx</p>, <p>xxxxx</p>, <p>2.orange</p>, <p>aaa</p>, <p>xxxxx</p>,<p>3.banana</p>, <p>aaa</p>, <p>xxxxx</p>]
From the list, "xxxx" are those useless values. I can see the pattern that the result I want is between two substrings. Substring1 = "<p>1" / "<p>2" / "<p>3" ; Substring2 = "</p>, <p>aaa".
Assume this pattern repeats hundreds of times. How do I get the result by python? Many thanks !!
My target result is :
apple
orange
banana
I have tried to use split and tried [sub1:sub2] but it doesn't work

From what I INFER from your question (assuming the words you're looking for follow a beacon of format <p>number. ), a regex would do the job:
import re
print(re.findall(r'<p>\d+.([^<]+)', html_string)
# ['apple', 'orange', 'banana']

Python regex to match Ledger/hledger account journal entry

I am writing a program in Python to parse a Ledger/hledger journal file.
I'm having problems coming up with a regex that I'm sure is quite simple. I want to parse a string of the form:
expenses:food:food and wine 20.99
and capture the account sections (between colons, allowing any spaces), regardless of the number of sub-accounts, and the total, in groups. There can be any number of spaces between the final character of the sub-account name and the price digits.
expenses:food:wine:speciality 19.99 is also allowable (no space in sub-account).
So far I've got (\S+):|(\S+ \S+):|(\S+ (?!\d))|(\d+.\d+) which is not allowing for any number of sub-accounts and possible spaces. I don't think I want to have OR operators in there either as this is going to concatenated with other regexes with .join() as part of the parsing function.
Any help greatly appreciated.
Thanks.

You can use the following:
((?:[^\s:]+)(?:\:[^\s:]+)*)\s*(\d+\.\d+)
Now we can use:
s = 'expenses:food:wine:speciality 19.99'
rgx = re.compile(r'((?:[^\s:]+)(?:\:[^\s:]+)*)\s*(\d+\.\d+)')
mat = rgx.match(s)
if mat:
categories,price = mat.groups()
categories = categories.split(':')
Now categories will be a list containing the categories, and price a string with the price. For your sample input this gives:
>>> categories
['expenses', 'food', 'wine', 'speciality']
>>> price
'19.99'

You don't need regex for such a simple thing at all, native str.split() is more than enough:
def split_ledger(line):
entries = line.split(":") # first split all the entries
last = entries.pop() # take the last entry
return entries + last.rsplit(" ", 1) # split on last space and return all together
print(split_ledger("expenses:food:food and wine 20.99"))
# ['expenses', 'food', 'food and wine ', '20.99']
print(split_ledger("expenses:food:wine:speciality 19.99"))
# ['expenses', 'food', 'wine', 'speciality ', '19.99']
Or if you don't want the leading/trailing whitespace in any of the entries:
def split_ledger(line):
entries = [e.strip() for e in line.split(":")]
last = entries.pop()
return entries + [e.strip() for e in last.rsplit(" ", 1)]
print(split_ledger("expenses:food:food and wine 20.99"))
# ['expenses', 'food', 'food and wine', '20.99']
print(split_ledger("expenses:food:wine:speciality 19.99"))
# ['expenses', 'food', 'wine', 'speciality', '19.99']

How to perform pattern matching between two strings?

I want to have a good pattern matching code which can exactly match between both strings.
x = "Apple iPhone 6(Silver, 16 GB)"
y = "Apple iPhone 6 64 GB GSM Mobile Phone (Silver)"
Approach 1:
tmp_body = " ".join("".join([" " if ch in string.punctuation else ch.lower() for ch in y]).split())
tmp_body_1 = " ".join("".join([" " if ch in string.punctuation else ch.lower() for ch in x]).split())
if tmp_body in tmp_body_1:
print "true"
In my problem x will always be a base string and y will change
Approach 2:
Fuzzy logic --> But was not getting good results through it
Approach 3:
Using regex which I don't know
I am still figuring out ways to solve it with regex.
Removal of special characters from both base and incoming string
Matches the GB and Color
Splitting the GB from the number for good matching
These things I have figured out.

How about the following approach. Split each into words, lowercase each word and store in a set. x must then be a subset of y. So for your example it will fail as 16 does not match 64:
x = "Apple iPhone 6(Silver, 16 GB)"
y = "Apple iPhone 6 64 GB GSM Mobile Phone (Silver)"
set_x = set([item.lower() for item in re.findall("([a-zA-Z0-9]+)", x)])
set_y = set([item.lower() for item in re.findall("([a-zA-Z0-9]+)", y)])
print set_x
print set_y
print set_x.issubset(set_y)
Giving the following results:
set(['apple', '16', 'gb', '6', 'silver', 'iphone'])
set(['apple', 'mobile', 'phone', '64', 'gb', '6', 'gsm', 'silver', 'iphone'])
False
If 64 is changed to 16 then you get:
set(['apple', '16', 'gb', '6', 'silver', 'iphone'])
set(['apple', '16', 'mobile', 'phone', 'gb', '6', 'gsm', 'silver', 'iphone'])
True

Looks like you're trying to do longest common substring here ofntwo unknown strings.
Find common substring between two strings
Regex only works when you have a known pattern to your strings. You could use LCS to derive a pattern that you could use to test additional strings, but I don't think that's what you want.
If you are wanting to extract the capacity, model, and other information from these strings, you may want to use multiple patterns to find each piece of information. Some information may not be available. Your regular expressions will need to flex in order to handle a wider input (hard for me to assume all variations given a sample size of 2).
capacity = re.search(r'(\d+)\s*GB', useragent)
model = re.search(r'Apple iPhone ([A-Za-z0-9]+)', useragent)
These patterns won't make much sense to you unless you read the Python re module documentation. Basically, for capacity, I'm searching for 1 or more digits followed by 0 or more whitespace followed by GB. If I find a match, the result is a match object and I can get the capacity with match.group(). Similar story for finding iPhone version, though my pattern doesn't work for "6 Plus".
Since you have no control over the generation of these strings, if this is a script that you plan on using 3 years from now, expect to be a slave to it, updating the regular expression patterns as new string formats become available. Hopefully this is a one-off number crunching exercise that can be scrapped as soon as you answered your question.

How to strip variable spaces in each line of a text file based on special condition - one-liner in Python?

I have some data (text files) that is formatted in the most uneven manner one could think of. I am trying to minimize the amount of manual work on parsing this data.
Sample Data :
Name Degree CLASS CODE EDU Scores
--------------------------------------------------------------------------------------
John Marshall CSC 78659944 89989 BE 900
Think Code DB I10 MSC 87782 1231 MS 878
Mary 200 Jones CIVIL 98993483 32985 BE 898
John G. S Mech 7653 54 MS 65
Silent Ghost Python Ninja 788505 88448 MS Comp 887
Conditions :
More than one spaces should be compressed to a delimiter (pipe better? End goal is to store these files in the database).
Except for the first column, the other columns won't have any spaces in them, so all those spaces can be compressed to a pipe.
Only the first column can have multiple words with spaces (Mary K Jones). The rest of the columns are mostly numbers and some alphabets.
First and second columns are both strings. They almost always have more than one spaces between them, so that is how we can differentiate between the 2 columns. (If there is a single space, that is a risk I am willing to take given the horrible formatting!).
The number of columns varies, so we don't have to worry about column names. All we want is to extract each column's data.
Hope I made sense! I have a feeling that this task can be done in a oneliner. I don't want to loop, loop, loop :(
Muchos gracias "Pythonistas" for reading all the way and not quitting before this sentence!

It still seems tome that there's some format in your files:
>>> regex = r'^(.+)\b\s{2,}\b(.+)\s+(\d+)\s+(\d+)\s+(.+)\s+(\d+)'
>>> for line in s.splitlines():
lst = [i.strip() for j in re.findall(regex, line) for i in j if j]
print(lst)
[]
[]
['John Marshall', 'CSC', '78659944', '89989', 'BE', '900']
['Think Code DB I10', 'MSC', '87782', '1231', 'MS', '878']
['Mary 200 Jones', 'CIVIL', '98993483', '32985', 'BE', '898']
['John G. S', 'Mech', '7653', '54', 'MS', '65']
['Silent Ghost', 'Python Ninja', '788505', '88448', 'MS Comp', '887']
Regex is quite straightforward, the only things you need to pay attention to are the delimiters (\s) and the word breaks (\b) in case of the first delimiter. Note that when the line wouldn't match you get an empty list as lst. That would be a read flag to bring up the user interaction described below. Also you could skip the header lines by doing:
>>> file = open(fname)
>>> [next(file) for _ in range(2)]
>>> for line in file:
... # here empty lst indicates issues with regex
Previous variants:
>>> import re
>>> for line in open(fname):
lst = re.split(r'\s{2,}', line)
l = len(lst)
if l in (2,3):
lst[l-1:] = lst[l-1].split()
print(lst)
['Name', 'Degree', 'CLASS', 'CODE', 'EDU', 'Scores']
['--------------------------------------------------------------------------------------']
['John Marshall', 'CSC', '78659944', '89989', 'BE', '900']
['Think Code DB I10', 'MSC', '87782', '1231', 'MS', '878']
['Mary 200 Jones', 'CIVIL', '98993483', '32985', 'BE', '898']
['John G. S', 'Mech', '7653', '54', 'MS', '65']
another thing to do is simply allow user to decide what to do with questionable entries:
if l < 3:
lst = line.split()
print(lst)
iname = input('enter indexes that for elements of name: ') # use raw_input in py2k
idegr = input('enter indexes that for elements of degree: ')
Uhm, I was all the time under the impression that the second element might contain spaces, since it's not the case you could just do:
>>> for line in open(fname):
name, _, rest = line.partition(' ')
lst = [name] + rest.split()
print(lst)

Variation on SilentGhost's answer, this time first splitting the name from the rest (separated by two or more spaces), then just splitting the rest, and finally making one list.
import re
for line in open(fname):
name, rest = re.split('\s{2,}', line, maxsplit=1)
print [name] + rest.split()

This answer was written after the OP confessed to changing every tab ("\t") in his data to 3 spaces (and not mentioning it in his question).
Looking at the first line, it seems that this is a fixed-column-width report. It is entirely possible that your data contains tabs that if expanded properly might result in a non-crazy result.
Instead of doing line.replace('\t', ' ' * 3) try line.expandtabs().
Docs for expandtabs are here.
If the result looks sensible (columns of data line up), you will need to determine how you can work out the column widths programatically (if that is possible) -- maybe from the heading line.
Are you sure that the second line is all "-", or are there spaces between the columns?
The reason for asking is that I once needed to parse many different files from a database query report mechanism which presented the results like this:
RecordType ID1 ID2 Description
----------- -------------------- ----------- ----------------------
1 12345678 123456 Widget
4 87654321 654321 Gizmoid
and it was possible to write a completely general reader that inspected the second line to determine where to slice the heading line and the data lines. Hint:
sizes = map(len, dash_line.split())
If expandtabs() doesn't work, edit your question to show exactly what you do have i.e. show the result of print repr(line) for the first 5 or so lines (including the heading line). It might also be useful if you could say what software produces these files.

how to show chinese word , not unicode word

this is my code:
from whoosh.analysis import RegexAnalyzer
rex = RegexAnalyzer(re.compile(ur"([\u4e00-\u9fa5])|(\w+(\.?\w+)*)"))
a=[(token.text) for token in rex(u"hi 中 000 中文测试中文 there 3.141 big-time under_score")]
self.render_template('index.html',{'a':a})
and it show this on the web page:
[u'hi', u'\u4e2d', u'000', u'\u4e2d', u'\u6587', u'\u6d4b', u'\u8bd5', u'\u4e2d', u'\u6587', u'there', u'3.141', u'big', u'time', u'under_score']
but i want to show chinese word , so i change this:
a=[(token.text).encode('utf-8') for token in rex(u"hi 中 000 中文测试中文 there 3.141 big-time under_score")]
and it show :
['hi', '\xe4\xb8\xad', '000', '\xe4\xb8\xad', '\xe6\x96\x87', '\xe6\xb5\x8b', '\xe8\xaf\x95', '\xe4\xb8\xad', '\xe6\x96\x87', 'there', '3.141', 'big', 'time', 'under_score']
so how to show chinese word in my code,
thanks

By default, printing a larger built-in structure gives the repr() of each of the elements. If you want the str()/unicode() instead then you need to iterate over the sequence yourself.
a = u"['" + u"', '".join(token.text for token in ...) + u"']"
print a