I have been struggling with creating a regular expression that will differentiate between an object definition and calling that object in Python. The purpose is syntax highlighting.
This is the situation which needs to be resolved: (numbers denote line)
0 def otherfunc(vars...):
1 pass
2
3 otherfunc(vars...)
I am interested in matching the name of the object, but not if preceded anywhere by def in the same line. The result on the above code should be:
"otherfunc", line: 3
Is regular expressions capable of doing something like this?
EDIT: I am only concerned with scanning/searching a single line at a time.
You could use negative lookbehind. This matches an atom that is not preceded by an atom. So in your case your looking for otherfunc which is not preceded by "def"
I'm use PCRE regex here.
(?<!def\s)otherfunc
I like Richards answer, however I would also take into considerarion the valid function name characters of phyton and intendation. So this is what I came up with:
(?<!(def\s))(?<=^|\s)[a-zA-Z_][\w_]*(?=\()
See this working sample on Rexex101
Explanation
Matches valid python function names if
(?<!(def\s)) they are not following a def and a whitespace and
(?<=^|\s) are either at the beginning of a line, or following a whitespace (this is the closest you get, since lookbehinds dont support wildcard specifiers) and
are followed by a opening bracket (()
Note that I am not an phyton dev, so for the sake of simplicity [a-zA-Z_][\w_]* matches valid phyton 2.x function names, you can extend this part of the expression to phyton 3.x which I have no clue of ;)
Related
Suppose I have a string which consists of a part of latex file. How can I use python re module to remove any math expression in it?
e.g:
text="This is an example $$a \text{$a$}$$. How to remove it? Another random math expression $\mathbb{R}$..."
I would like my function to return ans="This is an example . How to remove it? Another random math expression ...".
Thank you!
Try this Regex:
(\$+)(?:(?!\1)[\s\S])*\1
Click for Demo
Code
Explanation:
(\$+) - matches 1+ occurrences of $ and captures it in Group 1
(?:(?!\1)[\s\S])* - matches 0+ occurrences of any character that does not start with what was captured in Group 1
\1 - matches the contents of Group 1 again
Replace each match with a blank string.
As suggested by #torek, we should not match 3 or more consecutive $, hence changing the expression to (\${1,2})(?:(?!\1)[\s\S])*\1
It's commonly said that regular expressions cannot count, which is kind of a loose way of describing a problem more formally discussed in Count parentheses with regular expression. See that for what this means.
Now, with that in mind, note that LaTeX math expressions can include nested sub-equations, which can include further nested sub-equations, and so on. This is analogous to the problem of detecting whether a closing parenthesis closes an inner parenthesized expression (as in (for instance) this example, where the first one does not) or an outer parenthesis. Therefore, regular expressions are not going to be powerful enough to handle the full general case.
If you're willing to do a less-than-complete job, you can construct a regular expression that finds $...$ and $$...$$. You will need to pay attention to the particular regular expression language available. Python's is essentially the same as Perl's here.
Importantly, these $-matchers will completely miss \begin{equation} ... \end{equation}, \begin{eqnarray} ... \end{eqnarray}, and so on. We've already noted that handling LaTeX expression parsing with a mere regular expression recognizer is inadequate, so if you want to do a good job—while ignoring the complexity of lower-level TeX manipulation of token types, where one can change any individual character's category code —you will want a more general parser. You can then tokenize \begin, {, }, and words, and match up the begin/end pairs. You can also tokenize $ and $$ and match those up. Since parsers can count, in exactly the way that regular expressions can't, you can do a much better job this way.
I'm trying to craft a regex able to match anything up to a specific pattern. The regex then will continue looking for other patterns until the end of the string, but in some cases the pattern will not be present and the match will fail. Right now I'm stuck at:
.*?PATTERN
The problem is that, in cases where the string is not present, this takes too much time due to backtraking. In order to shorten this, I tried mimicking atomic grouping using positive lookahead as explained in this thread (btw, I'm using re module in python-2.7):
Do Python regular expressions have an equivalent to Ruby's atomic grouping?
So I wrote:
(?=(?P<aux1>.*?))(?P=aux1)PATTERN
Of course, this is faster than the previous version when STRING is not present but trouble is, it doesn't match STRING anymore as the . matches everyhing to the end of the string and the previous states are discarded after the lookahead.
So the question is, is there a way to do a match like .*?STRING and alse be able to fail faster when the match is not present?
You could try using split
If the results are of length 1 you got no match. If you get two or more you know that the first one is the first match. If you limit the split to size one you'll short-circuit the later matching:
"HI THERE THEO".split("TH", 1) # ['HI ', 'ERE THEO']
The first element of the results is up to the match.
One-Regex Solution
^(?=(?P<aux1>(?:[^P]|P(?!ATTERN))*))(?P=aux1)PATTERN
Explanation
You wanted to use the atomic grouping like this: (?>.*?)PATTERN, right? This won't work. Problem is, you can't use lazy quantifiers at the end of an atomic grouping: the definition of the AG is that once you're outside of it, the regex won't backtrack inside.
So the regex engine will match the .*?, because of the laziness it will step outside of the group to check if the next character is a P, and if it's not it won't be able to backtrack inside the group to match that next character inside the .*.
What's usually used in Perl are structures like this: (?>(?:[^P]|P(?!ATTERN))*)PATTERN. That way, the equivalent of .* (here (?:[^P]|P(?!ATTERN))) won't "eat up" the wanted pattern.
This pattern is easier to read in my opinion with possessive quantifiers, which are made just for these occasions: (?:[^P]|P(?!ATTERN))*+PATTERN.
Translated with your workaround, this would lead to the above regex (added ^ since you should anchor the regex, either to the start of the string or to another regex).
The Python documentation includes a brief outline of the differences between the re.search() and re.match() functions http://docs.python.org/2/library/re.html#search-vs-match. In particular, the following quote is relevant:
Sometimes you’ll be tempted to keep using re.match(), and just add .* to the front of your RE. Resist this temptation and use re.search() instead. The regular expression compiler does some analysis of REs in order to speed up the process of looking for a match. One such analysis figures out what the first character of a match must be; for example, a pattern starting with Crow must match starting with a 'C'. The analysis lets the engine quickly scan through the string looking for the starting character, only trying the full match if a 'C' is found.
Adding .* defeats this optimization, requiring scanning to the end of the string and then backtracking to find a match for the rest of the RE. Use re.search() instead.
In your case, it would be preferable to define your pattern simply as:
pattern = re.compile("PATTERN")
And then call pattern.search(...), which will not backtrack when the pattern is not found.
I am using python 2.7 with scrapy .20
I have this test
0552121152, +97143321090
I want to get the value before the comma and the value after it.
My regular expression is
\s*(.*), for the one before
and ,\s*(.*) for the one after
in both ways I got the whole test.
why please?
Edit
For who need a full details, here it is:
This is the page I am scraping http://www.justproperty.com/apartments/old-town/1092713-amazing-2br-apartment-in-the-old-town-island-with-a-burj-khalifa-and-address-hotel-view.html
and this is my scrapy code in cmd in shell:
s =sel.xpath("normalize-space(.//span[#class='content_agent']/span[last()]/span[2]/text())")
s.re(r'\s*(.*),')
Note
I am not asking about the solution to get these values, I am asking about why my mentioned-regular expression is not working
Try doing it this way:
import re
string = "0552121152, +97143321090"
pattern = re.compile(r"\s*(.*),\s*(.*)")
print pattern.search(string).groups()
Result:
('0552121152', '+97143321090')
[Finished in 0.3s]
The only difference might be because I used the .groups() method, that's why I'm getting it. If you can reproduce this on your end, then it's not your regular expression that is wrong but your coding style.
Other than that, I fail to see how the regex fails at all.
That because . matches also , and it's a greedy selection. Use something like more specific like [^,] or \d instead of . symbol.
Something like this would store the part before the comma in \1 and the part after the comma in \2
/(\s?[\d]+)(?:[, ]+)([+\d]+)/
This looks for an optional number of whitespace characters followed by some digits. Then it looks for either a space or comma (or both) followed by another string made up of a character class of numbers and/or a plus sign.
I'm trying to match a C/C++ function definition using a fairly complex regular expression. I've found a case where it's not working and I'm trying to understand why. Here is the input string which does not match:
void Dump(const char * itemName, ofstream & os)
which clearly is a valid C++ method declaration. Here is the RE:
^[^=+-|#]*?([\w<>]+\s+(?!if|for|switch|while|catch|return)\w+)\s*\([^;=+-|]*$
This basically tries to distinguish between other C syntax which looks like a method declaration, i.e. which has words followed by paraentheses.
Using the very useful Python regular expression debugger (http://www.pythonregex.com/) I've narrowed it down to the trailing "$" - if I remove the trailing $ in the regular expression, it matches the method signature above; if I leave in the $, it doesn't. There must be some idiosyncracy of Python RE's that is eluding me here. Thanks.
The use of +-| in your character class [^;=+-|] is a range specification. This will result in the character class containing (actually excluding since you're using ^) much more than you intend. To specify a literal - in a character class, mention it first like [^-;=+|].
The output of PythonRegex is somewhat misleading. The results of r.groups() and r.findall() are both the same: u'void Dump', which is the content of the first capturing group. If it showed the whole match, you'd see that when remove the $ you're only matching
void Dump(
...not the whole function definition as you intended. The reason for that (as Greg explained) is a syntax error in your last character class. You need to escape the hyphen by listing it first ([^-;=+|]) or last ([^;=+|-]), or by adding a backslash ([^;=+\-|]).
The only way I can see to get PythonRegex to show the whole match is by removing all capturing groups (or converting them to non-capturing).
Goal: Given a number (it may be very long and it is greater than 0), I'd like to get the five least meaningful digits dropping any 0 at the end of that number.
I tried to solve this with regex, Helped by RegexBuddy I came to this one:
[\d]+([\d]{0,4}+[1-9])0*
But python can't compile that.
>>> import re
>>> re.compile(r"[\d]+([\d]{0,4}+[1-9])0*")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/re.py", line 188, in compile
return _compile(pattern, flags)
File "/usr/lib/python2.5/re.py", line 241, in _compile
raise error, v # invalid expression
sre_constants.error: multiple repeat
The problem is the "+" after "{0,4}", it seems it doesn't work in python (even in 2.6)
How can I write a working regex?
PS:
I know you can start dividing by 10 and then using the remainder n%100000... but this is a problem about regex.
That regular expression is very superfluous. Try this:
>>> import re
>>> re.compile(r"(\d{0,4}[1-9])0*$")
The above regular expression assumes that the number is valid (it will also match "abc0123450", for example.) If you really need the validation that there are no non-number characters, you may use this:
>>> import re
>>> re.compile(r"^\d*?(\d{0,4}[1-9])0*$")
Anyways, the \d does not need to be in a character class, and the quantifier {0,4} does not need to be forced to be greedy (as the additional + specifies, although apparently Python does not recognize that.)
Also, in the second regular expression, the \d is non-greedy, as I believe this will improve the performance and accuracy. I also made it "zero or more" as I assume that is what you want.
I also added anchors as this ensures that your regular expression won't match anything in the middle of a string. If this is what you desired though (maybe you're scanning a long text?), remove the anchors.
\d{0,4}+ is a possessive quantifier supported by certain regular expression flavors such as .NET and Java. Python does not support possessive quantifiers.
In RegexBuddy, select Python in the toolbar at the top, and RegexBuddy will tell you that Python doesn't support possessive quantifiers. The + will be highlighted in red in the regular expression, and the Create tab will indicate the error.
If you select Python on the Use tab in RegexBuddy, RegexBuddy will generate a Python source code snippet with a regular expression without the possessive quantifier, and a comment indicating that the removal of the possessive quantifier may yield different results. Here's the Python code that RegexBuddy generates using the regex from the question:
# Your regular expression could not be converted to the flavor required by this language:
# Python does not support possessive quantifiers
# Because of this, the code snippet below will not work as you intended, if at all.
reobj = re.compile(r"[\d]+([\d]{0,4}[1-9])0*")
What you probably did is select a flavor such as Java in the main toolbar, and then click Copy Regex as Python String. That will give you a Java regular expression formatted as a Pythong string. The items in the Copy menu do not convert your regular expression. They merely format it as a string. This allows you to do things like format a JavaScript regular expression as a Python string so your server-side Python script can feed a regex into client-side JavaScript code.
Small tip. I recommend you test with reTest instead of RegExBuddy. There are different regular expression engines for different programming languages. ReTest is valuable in that it allows you to quickly test regular expression strings within Python itself. That way you can insure that you tested your syntax with the Python's regular expression engine.
The error seems to be that you have two quantifiers in a row, {0,4} and +. Unless + is meant to be a literal here (which I doubt, since you're talking about numbers), then I don't think you need it at all. Unless it means something different in this situation (possibly the greediness of the {} quantifier)? I would try
[\d]+([\d]{0,4}[1-9])0*
If you actually intended to have both quantifiers to be applied, then this might work
[\d]+(([\d]{0,4})+[1-9])0*
But given your specification of the problem, I doubt that's what you want.
This is my solution.
re.search(r'[1-9]\d{0,3}[1-9](?=0*(?:\b|\s|[A-Za-z]))', '02324560001230045980a').group(1)
'4598'
[1-9] - the number must start with 1 - 9
\d{0,3} - 0 or 3 digits
[1-9] - the number must finish with 1 or 9
(?=0*(:?\b|\s\|[A-Za-z])) - the final part of string must be formed from 0 and or \b, \s, [A-Za-z]