I'm scraping some data. One of the data points is tournament prize pools. There are many different currencies in the data. I'd like to extract the amount and currency from each value, so that I can use Google to convert these to a base currency. However, it's been a while since I've used regular expressions, so I'm rusty to say the least. Possible formats of the data are as follows:
$534
$22,136.20
3,200,000 Ft HUF
12,500 kr DKK
50,000 kr SEK
$3,800 AUD
$10,000 NZD
€4,500 EUR
¥100,000 CNY
₹7,000,000 INR
R$39,000 BRL
Below is the first regular expression I came up with.
[0-9,.]+(.+)[A-Z]{3}
But that obviously doesn't capture the amount and currency, so I changed it.
([0-9,.]+).+([A-Z]{3})
However, there are issues with this regular expression that I can't figure out.
([0-9,.]+) by itself works fine to capture just the amount.
When I add .+ to that expression, for some reason it stops capturing the trailing 4 and 0 in the first and second test cases respectively. Why?
Then when I add ([A-Z]{3}), it seems to work perfectly for all of the test cases, but obviously selects nothing in the first two.
So I changed it to ([A-Z]{0,3}), which seems to break everything.
What's happening? How can I change the expression so that it works?
This is where I'm at: ([0-9,.]+)((?:.+)([A-Z]{3}))?
This should work:
([0-9,.]+).*?([A-Z]{3})?$
A few changes I made:
I changed the .+ to .*? because there isn't always something after the number (like the first two cases). I used lazy matching here because otherwise it would match everything till the end.
I made group 2 optional with a ? because there isn't always a currency (first 2 cases)
I added an end of line anchor $ to make the lazy .*? match something instead of nothing.
If you don't know what "lazy" means in this context, see this post.
Demo
For the example data, you could use an optional non capturing group to match the space and the characters before the currency:
([0-9,.]+)(?:(?: [A-Za-z]+)? ([A-Z]{3}))?
Regex demo
That will match
( Capture group
[0-9,.]+ match 1+ times what is listed in the character class
) Close capture group
(?: Non capturing group
(?: [A-Za-z]+ )? Optional group to match a space, 1+ times a-zA-Z and space
([A-Z]{3}) Capture 3 uppercase chars
)? Close non capturing group and make it optional
Related
Sorry for the somewhat unhelpful title, I'm having a really hard time explaining this issue.
I have a list of unique identifiers that can appear in a number of different ways and I'm trying to use regex to normalize them so I can compare across several databases. Here are some examples of them:
AB1201
AB-1201
AB1201-T
AB-12-01L1
AB1201-TER
AB1201 Transit
I've written a line of code that pulls out all hypens and spaces, and the used this regex:
([a-zA-Z]{2}[\d]{4})(L\d|Transit|T$)?
This works exactly as expected, returning a list looking like this:
AB1201
AB1201
AB1201T
AB1201L1
AB1201
AB1201T
The issue is, I have one identifier that looks like this: AB1201-02. I need this to be raised as an exception, and not included as a match.
Any ideas? I'm happy to provide more clarification if necessary. Thanks!
From Regex101 online tester
You can exclude matching the following hyphen and a digit (?!-\d) using a negative lookahead.
If it should start at the beginning of the string, you could use an anchor ^
Note that you could write [\d] as \d
^([a-zA-Z]{2}\d{4})(?!-\d)(L\d|Transit|T$)?
The pattern will look like
^ Start of string
( Capture group 1
[a-zA-Z]{2}\d{4} Match 2 times a-zA-Z and 4 digits
) Close group
(?!-\d) Negative lookahead, assert what is directly to the right is not - and a digit
(L\d|Transit|T$)? Optional capture group 2
Regex demo
Try this regular expression
^([a-zA-Z]{2}[\d]{4})(?!-\d)(L\d|Transit|T|-[A-Z]{3})?$
I have added the (?!...) Negative Lookahead to avoid matching with the -02.
(?!...) Negative Lookahead: Starting at the current position in the expression, ensures that the given pattern will not match. Does not consume characters.
You can view a demo on this link.
I am trying to clean up text for use in a machine learning application. Basically these are specification documents that are "semi-structured" and I am trying to remove the section number that is messing with NLTK sent_tokenize() function.
Here is a sample of the text I am working with:
and a Contract for the work and/or material is entered into with some other person for a
greater amount, the undersigned hereby agrees to forfeit all right and title to the
aforementioned deposit, and the same is forfeited to the Crown.
2.3.3
...
(b)
until thirty-five days after the time fixed for receiving this tender,
whichever first occurs.
2.4
AGREEMENT
Should this tender be accepted, the undersigned agrees to enter into written agreement with
the Minister of Transportation of the Province of Alberta for the faithful performance of the
works covered by this tender, in accordance with the said plans and specifications and
complete the said work on or before October 15, 2019.
I am trying to remove all the section breaks (ex. 2.3.3, 2.4, (b)), but not the date numbers.
Here is the regex I have so far: [0-9]*\.[0-9]|[0-9]\.
Unfortunately it matches part of the date in the last paragraph (2019. turns into 201) and I really dont know how to fix this being a non-expert at regex.
Thanks for any help!
You may try replacing the following pattern with empty string
((?<=^)|(?<=\n))(?:\d+(?:\.\d+)*|\([a-z]+\))
output = re.sub(r'((?<=^)|(?<=\n))(?:\d+(?:\.\d+)*|\([a-z]+\))', '', input)
print(output)
This pattern works by matching a section number as \d+(?:\.\d+)*, but only if it appears as the start of a line. It also matches letter section headers as \([a-z]+\).
To your specific case, I think \n[\d+\.]+|\n\(\w\) should works. The \n helps to diferentiate the section.
The pattern you tried [0-9]*\.[0-9]|[0-9]\. is not anchored and will match 0+ digits, a dot and single digit or | a single digit and a dot
It does not take the match between parenthesis into account.
Assuming that the section breaks are at the start of the string and perhaps might be preceded with spaces or tabs, you could update your pattern with the alternation to:
^[\t ]*(?:\d+(?:\.\d+)+|\([a-z]+\))
^ Start of string
[\t ]* Match 0+ times a space or tab
(?: Non capturing group
\d+(?:\.\d+)+ Match 1+ digits and repeat 1+ times a dot and 1+ digits to match at least a single dot to match 2.3.3 or 2.4
|
\([a-z]+\) Match 1+ times a-z between parenthesis
) Close non capturing group
Regex demo | Python demo
For example using re.MULTILINE whers s is your string:
pattern = r"^(?:\d+(?:\.\d+)+|\([a-z]+\))"
result = re.sub(pattern, "", s, 0, re.MULTILINE)
I have the following code:
import pandas as pd
s = pd.Series(['toy story (1995)', 'the pirates (2014)'])
print(s.str.extract('.*\((.*)\).*',expand = True))
with output:
0
0 1995
1 2014
I understand that the extract function is pulling the values between the parentheses for both series objects. However I do not understand how. What exactly does '.*\((.*)\).*' mean? I think that the asterisks represent wild card characters but beyond that I am quite confused as to what is actually going on with this expression.
.*\( matches everything up until the first (
\).* matches everything from ) until the end
(.*) returns everything in between the first two matches
.* Match any number of characters
\( Match one opening parenthesis
(.*) Match any number of characters into the first capturing group
\) Match a closing parenthesis
.* Match any number of characters
This notation is called a regular expression, and I guess Pandas uses regexes in the extract function so you can get more precise data. Things inside capturing groups would be returned.
You can learn more about regexes at the Wikipedia page.
Here's a test example using your regex.
I have a code to scrape a million websites and detect contact info from their homepage.
For some reasons, when I run code, it gets stuck and does not proceed after crawling about 60k requests, I am marking the website URLs in my DB as status=done
I have run code several times but it gets stuck around 60k requests.
It doesnt get stuck on a certain website.
Here is Regex I am using
emails = re.findall('[\w\.-]+#[\w-]+\.[\w\.-]+', lc_body)
mobiles = re.findall(r"(\(?(?<!\d)\d{3}\)?-? *\d{3}-? *-?\d{4})(?!\d)|(?<!\d)(\+\d{11})(?!\d)", lc_body)
abns = re.findall('[a][-\.\s]??[b][-\.\s]??[n][-\:\.\s]?[\:\.\s]?(\d+[\s\-\.]?\d+[\s\-\.]?\d+[\s\-\.]?\d+)', lc_body)
licences = re.findall(r"(Licence|Lic|License|Licence)\s*(\w*)(\s*|\s*#\s*|\s*.\s*|\s*-\s*|\s*:\s+)(\d+)", lc_body, re.IGNORECASE)
My thought is licences's regex is causing issues, how can I simplify it? How can I remove Backtracking ?
I want to find all Licence numbers possible.
It can be License No: 2543 , License: 2543, License # 2543, License #2543, License# 2543 and many other combinations as well.
The issue is caused with the third group: (\s*|\s*#\s*|\s*.\s*|\s*-\s*|\s*:\s+) - all alternatives start with \s* here. This causes lots of redundant backtracking as these alternatives can match at the same location in a string. The best practice is to use alternatives in an alternation group that do not match at the same location.
Now, looking at the strings you need to match, I suggest using
Lic(?:en[cs]e)?(?:\W*No:)?\W*\d+
See the regex demo
Make the pattern more specific and linear, get rid of as many alternations as possible, use optional non-capturing groups and character classes.
Details:
Lic(?:en[cs]e)? - Lic followed with 1 or 0 occurrences (the (?:...)? is an optional non-capturing group since ? quantifier matches 1 or 0 occurrences of the quantified subpatterns) of ence or ense (the character class [sc] matches either s or c and is much more efficient than (s|c))
(?:\W*No:)? - a non-capturing group that matches 1 or 0 occurrences of 0+ non-word chars (with \W*) followed with No: substring
\W*
\d+ - 1 or more digits.
I'n trying to match a string with the following different combinations using python
(here x's are digits of lenght 4)
W|MON-FRI|xxxx-xxxx
W|mon-fri|xxxx-xxxx
W|MON-THU,SAT|xxxx-xxxx
W|mon-thu,sat|xxxx-xxxx
W|MON|xxxx-xxxx
Here the first part and the last is static, second part is can have any of the combinations as shown above, like sometime the days were separated by ',' or '-'.
I'm a newbie to Regular Expressions, I was googled on how regular expressions works, I can able to do the RE for bits & pieces of above expressions like matching the last part with re.compile('(\d{4})-(\d{4})$') and the first part with re.compile('[w|W]').
I tried to match the 2nd part but couldn't succeeded with
new_patt = re.compile('(([a-zA-Z]{3}))([,-]?)(([a-zA-Z]{3})?))
How can I achieve this?
Here is a regular expression that should work:
pat = re.compile('^W\|(mon|tue|wed|thu|fri|sat|sun)(-(mon|tue|wed|thu|fri|sat|sun))?(,(mon|tue|wed|thu|fri|sat|sun)(-(mon|tue|wed|thu|fri|sat|sun))?)?\⎪\d{4}-\d{4}$', re.IGNORECASE)
Note first how you can ignore the case to take care of lower and upper cases. In addition to the static text at the beginning and the numbers at the end, this regex matches a day of the week, followed by an optional dash+day of the week, followed by an optional sequence that contains a ,and the previous sequence.
"^W\|(mon|tue|wed|thu|fri|sat|sun)(-(mon|tue|wed|thu|fri|sat|sun))?(,(mon|tue|wed|thu|fri|sat|sun)(-(mon|tue|wed|thu|fri|sat|sun))?)?\|\d{4}-\d{4}$"i
^ assert position at start of the string
W matches the character W literally (case insensitive)
\| matches the character | literally
1st Capturing group (mon|tue|wed|thu|fri|sat|sun)
2nd Capturing group (-(mon|tue|wed|thu|fri|sat|sun))?
Quantifier: ? Between zero and one time, as many times as possible, giving back as needed [greedy]
Note: A repeated capturing group will only capture the last iteration. Put a capturing group around the repeated group to capture all iterations or use a non-capturing group instead if you're not interested in the data
- matches the character - literally
3rd Capturing group (mon|tue|wed|thu|fri|sat|sun)
4th Capturing group (,(mon|tue|wed|thu|fri|sat|sun)(-(mon|tue|wed|thu|fri|sat|sun))?)?
Quantifier: ? Between zero and one time, as many times as possible, giving back as needed [greedy]
Note: A repeated capturing group will only capture the last iteration. Put a capturing group around the repeated group to capture all iterations or use a non-capturing group instead if you're not interested in the data
, matches the character , literally
5th Capturing group (mon|tue|wed|thu|fri|sat|sun)
6th Capturing group (-(mon|tue|wed|thu|fri|sat|sun))?
Quantifier: ? Between zero and one time, as many times as possible, giving back as needed [greedy]
Note: A repeated capturing group will only capture the last iteration. Put a capturing group around the repeated group to capture all iterations or use a non-capturing group instead if you're not interested in the data
- matches the character - literally
7th Capturing group (mon|tue|wed|thu|fri|sat|sun)
\| matches the character | literally
\d{4} match a digit [0-9]
Quantifier: {4} Exactly 4 times
- matches the character - literally
\d{4} match a digit [0-9]
Quantifier: {4} Exactly 4 times
$ assert position at end of the string
i modifier: insensitive. Case insensitive match (ignores case of [a-zA-Z])
https://regex101.com/r/dW4dQ7/1
You can get everything in one go:
^W\|(?:\w{3}[-,]){0,2}\w{3}\|(?:\d{4}[-]?){2}$
With Live Demo
Thanks for your posts and comments,
At last I am able to satisfy my requirement with regular expressions
here it is
"^[w|W]\|(mon|sun|fri|thu|sat|wed|tue|[0-6])(-(mon|fri|sat|sun|wed|thu|tue|[0-6]))?(,(mon|fri|sat|sun|wed|thu|tue|[0-6]))*?\|(\d{4}-\d{4})$"img
I just tweaked the answer posted by Julien Spronck
Once again thanks all