I am reading through http://docs.python.org/2/library/re.html. According to this the "r" in pythons re.compile(r' pattern flags') refers the raw string notation :
The solution is to use Python’s raw string notation for regular
expression patterns; backslashes are not handled in any special way in
a string literal prefixed with 'r'. So r"\n" is a two-character string
containing '\' and 'n', while "\n" is a one-character string
containing a newline. Usually patterns will be expressed in Python
code using this raw string notation.
Would it be fair to say then that:
re.compile(r pattern) means that "pattern" is a regex while, re.compile(pattern) means that "pattern" is an exact match?
As #PauloBu stated, the r string prefix is not specifically related to regex's, but to strings generally in Python.
Normal strings use the backslash character as an escape character for special characters (like newlines):
>>> print('this is \n a test')
this is
a test
The r prefix tells the interpreter not to do this:
>>> print(r'this is \n a test')
this is \n a test
>>>
This is important in regular expressions, as you need the backslash to make it to the re module intact - in particular, \b matches empty string specifically at the start and end of a word. re expects the string \b, however normal string interpretation '\b' is converted to the ASCII backspace character, so you need to either explicitly escape the backslash ('\\b'), or tell python it is a raw string (r'\b').
>>> import re
>>> re.findall('\b', 'test') # the backslash gets consumed by the python string interpreter
[]
>>> re.findall('\\b', 'test') # backslash is explicitly escaped and is passed through to re module
['', '']
>>> re.findall(r'\b', 'test') # often this syntax is easier
['', '']
No, as the documentation pasted in explains the r prefix to a string indicates that the string is a raw string.
Because of the collisions between Python escaping of characters and regex escaping, both of which use the back-slash \ character, raw strings provide a way to indicate to python that you want an unescaped string.
Examine the following:
>>> "\n"
'\n'
>>> r"\n"
'\\n'
>>> print "\n"
>>> print r"\n"
\n
Prefixing with an r merely indicates to the string that backslashes \ should be treated literally and not as escape characters for python.
This is helpful, when for example you are searching on a word boundry. The regex for this is \b, however to capture this in a Python string, I'd need to use "\\b" as the pattern. Instead, I can use the raw string: r"\b" to pattern match on.
This becomes especially handy when trying to find a literal backslash in regex. To match a backslash in regex I need to use the pattern \\, to escape this in python means I need to escape each slash and the pattern becomes "\\\\", or the much simpler r"\\".
As you can guess in longer and more complex regexes, the extra slashes can get confusing, so raw strings are generally considered the way to go.
No. Not everything in regex syntax needs to be preceded by \, so ., *, +, etc still have special meaning in a pattern
The r'' is often used as a convenience for regex that do need a lot of \ as it prevents the clutter of doubling up the \
Related
I want to use, for example, this patterns
rules = {
'\s': '_',
'.(?P<word>\w)': '\1',
'text1': 'text2',
#etc
}
using re.sub()
There are some examples like this, but it doesn't work with regex special charecters.
I use raw strings when using regex in python. Saves you from having to escape special characters: https://docs.python.org/2/library/re.html
Try:
rules = {
r"\s": r"_",
r"text1": r"text2",
#etc
}
You should use raw strings like so:
rules = {
r'\s': r'_',
r'.(?P<word>\w)': r'\1',
r'text1': r'text2',
#etc
}
It means you don't need to escape special characters
Here is why it happens (direct quote from the docs):
Regular expressions use the backslash character ('\') to indicate
special forms or to allow special characters to be used without
invoking their special meaning. This collides with Python’s usage of
the same character for the same purpose in string literals; for
example, to match a literal backslash, one might have to write '\\'
as the pattern string, because the regular expression must be \, and
each backslash must be expressed as \ inside a regular Python string
literal.
And how to solve it (another quote from the docs):
The solution is to use Python’s raw string notation for regular
expression patterns; backslashes are not handled in any special way in
a string literal prefixed with 'r'. So r"\n" is a two-character string
containing '\' and 'n', while "\n" is a one-character string
containing a newline. Usually patterns will be expressed in Python
code using this raw string notation.
Surely, you need to use raw strings when declaring Python regexes, and there are some issues with your examples, but you are interested in how to run the regex replacements.
I suggest using an OrderedDict so that the replacements could be performed in a strict order, as they were defined in the dictionary. Then, the code will look like
import re
from collections import OrderedDict # adding the import
rules=OrderedDict() # defining the regex
rules[r'\s'] = '-' # replacement
rules[r'.(\w)'] = r'\1' # pairs
rules['text1'] = 'text2' # here
s = "nnoo mmoorree tteexxtt11" # a test string
for key in rules.keys(): # iterating through keys
s = re.sub(key, rules[key], s) # perform the S&R
print(s) # Demo printing
See the IDEONE demo
Use raw string notation to avoid having to escape your special characters:
rules = {
'\s': '_',
'.(?P<word>\w)': '\1',
'text1': 'text2',
#etc
}
Directly from the regular expression module (re) documentation:
Raw string notation (r"text") keeps regular expressions sane. Without it, every backslash ('\') in a regular expression would have to be prefixed with another one to escape it. For example, the two following lines of code are functionally identical:
>>> re.match(r"\W(.)\1\W", " ff ")
<_sre.SRE_Match object at ...>
>>> re.match("\\W(.)\\1\\W", " ff ")
<_sre.SRE_Match object at ...>
When one wants to match a literal backslash, it must be escaped in the regular expression. With raw string notation, this means r"\". Without raw string notation, one must use "\\", making the following lines of code functionally identical:
>>> re.match(r"\\", r"\\")
<_sre.SRE_Match object at ...>
>>> re.match("\\\\", r"\\")
<_sre.SRE_Match object at ...>
Could you tell me why '?\\\?'=='?\\\\?' gives True? That drives me crazy and I can't find a reasonable answer...
>>> list('?\\\?')
['?', '\\', '\\', '?']
>>> list('?\\\\?')
['?', '\\', '\\', '?']
Basically, because python is slightly lenient in backslash processing. Quoting from https://docs.python.org/2.0/ref/strings.html :
Unlike Standard C, all unrecognized escape sequences are left in the string unchanged, i.e., the backslash is left in the string.
(Emphasis in the original)
Therefore, in python, it isn't that three backslashes are equal to four, it's that when you follow backslash with a character like ?, the two together come through as two characters, because \? is not a recognized escape sequence.
This is because backslash acts as an escape character for the character(s) immediately following it, if the combination represents a valid escape sequence. The dozen or so escape sequences are listed here. They include the obvious ones such as newline \n, horizontal tab \t, carriage return \r and more obscure ones such as named unicode characters using \N{...}, e.g. \N{WAVY DASH} which represents unicode character \u3030. The key point though is that if the escape sequence is not known, the character sequence is left in the string as is.
Part of the problem might also be that the Python interpreter output is misleading you. This is because the backslashes are escaped when displayed. However, if you print those strings, you will see the extra backslashes disappear.
>>> '?\\\?'
'?\\\\?'
>>> print('?\\\?')
?\\?
>>> '?\\\?' == '?\\?' # I don't know why you think this is True???
False
>>> '?\\\?' == r'?\\?' # but if you use a raw string for '?\\?'
True
>>> '?\\\\?' == '?\\\?' # this is the same string... see below
True
For your specific examples, in the first case '?\\\?', the first \ escapes the second backslash leaving a single backslash, but the third backslash remains as a backslash because \? is not a valid escape sequence. Hence the resulting string is ?\\?.
For the second case '?\\\\?', the first backslash escapes the second, and the third backslash escapes the fourth which results in the string ?\\?.
So that's why three backslashes is the same as four:
>>> '?\\\?' == '?\\\\?'
True
If you want to create a string with 3 backslashes you can escape each backslash:
>>> '?\\\\\\?'
'?\\\\\\?'
>>> print('?\\\\\\?')
?\\\?
or you might find "raw" strings more understandable:
>>> r'?\\\?'
'?\\\\\\?'
>>> print(r'?\\\?')
?\\\?
This turns of escape sequence processing for the string literal. See String Literals for more details.
Because \x in a character string, when x is not one of the special backslashable characters like n, r, t, 0, etc, evaluates to a string with a backslash and then an x.
>>> '\?'
'\\?'
From the python lexical analysis page under string literals at:
https://docs.python.org/2/reference/lexical_analysis.html
There is a table that lists all the recognized escape sequences.
\\ is an escape sequence that is === \
\? is not an escape sequence and is === \?
so '\\\\' is '\\' followed by '\\' which is '\\' (two escaped \)
and '\\\' is '\\' followed by '\' which is also '\\' (one escaped \ and one raw \)
also, it should be noted that python does not distinguish between single and double quotes surrounding a string literal, unlike some other languages.
So 'String' and "String" are the exact same thing in python, they do not affect the interpretation of escape sequences.
mhawke's answer pretty much covers it, I just want to restate it in a more concise form and with minimal examples that illustrate this behaviour.
I guess one thing to add is that escape processing moves from left to right, so that \n first finds the backslash and then looks for a character to escape, then finds n and escapes it; \\n finds first backslash, finds second and escapes it, then finds n and sees it as a literal n; \? finds backslash and looks for a char to escape, finds ? which cannot be escaped, and so treats \ as a literal backslash.
As mhawke noted, the key here is that interactive interpreter escapes the backslash when displaying a string. I'm guessing the reason for that is to ensure that text strings copied from interpreter into code editor are valid python strings. However, in this case this allowance for convenience causes confusion.
>>> print('\?') # \? is not a valid escape code so backslash is left as-is
\?
>>> print('\\?') # \\ is a valid escape code, resulting in a single backslash
'\?'
>>> '\?' # same as first example except that interactive interpreter escapes the backslash
\\?
>>> '\\?' # same as second example, backslash is again escaped
\\?
I would like to know the reason I get the same result when using string prefix "r" or not when looking for a period (full stop) using python regex.
After reading a number sources (Links below) a multiple times and experimenting with in code to find the same result (again see below), I am still unsure of:
What is the difference when using string prefix "r" and not using string prefix "r", when looking for a period using regex?
Which way is considered the correct way of finding a period in a string using python regex with string prefix "r" or without string prefix "r"?
re.compile("\.").sub("!", "blah.")
'blah!'
re.compile(r"\.").sub("!", "blah.")
'blah!'
re.compile(r"\.").search("blah.").group()
'.'
re.compile("\.").search("blah.").group()
'.'
Sources I have looked at:
Python docs: string literals
http://docs.python.org/2/reference/lexical_analysis.html#string-literals
Regular expression to replace "escaped" characters with their originals
Python regex - r prefix
r prefix is for raw strings
http://forums.udacity.com/questions/7000217/r-prefix-is-for-raw-strings
The raw string notation is just that, a notation to specify a string value. The notation results in different string values when it comes to backslash escapes recognized by the normal string notation. Because regular expressions also attach meaning to the backslash character, raw string notation is quite handy as it avoids having to use excessive escaping.
Quoting from the Python Regular Expression HOWTO:
The solution is to use Python’s raw string notation for regular expressions; backslashes are not handled in any special way in a string literal prefixed with 'r', so r"\n" is a two-character string containing '\' and 'n', while "\n" is a one-character string containing a newline. Regular expressions will often be written in Python code using this raw string notation.
The \. combination has no special meaning in regular python strings, so there is no difference, at all between the result of '\.' and r'\.'; you can use either:
>>> len('\.')
2
>>> len(r'\.')
2
Raw strings only make a difference when the backslash + other characters do have special meaning in regular string notation:
>>> '\b'
'\x08'
>>> r'\b'
'\\b'
>>> len('\b')
1
>>> len(r'\b')
2
The \b combination has special meaning; in a regular string it is interpreted as the backspace character. But regular expressions see \b as a word boundary anchor, so you'd have to use \\b in your Python string every time you wanted to use this in a regular expression. Using r'\b' instead makes it much easier to read and write your expressions.
The regular expression functions are passed string values; the result of Python interpreting your string literal. The functions do not know if you used raw or normal string literal syntax.
Python has this way of specifying regular expression pattern, where all special character should not be treated as special. From the docs:
So r"\n" is a two-character string containing '\' and 'n', while "\n" is a one-character string containing a newline.
Why then does this works?
import re
print re.split(r"\n", "1\n2\n3")
The first argument should be "\" and "n" and the second one should contain two newlines. But it prints:
['1', '2', '3']
The first one does contain backslash-and-n, but in regular-expression-language, backslash-and-n means newline (just like it does in Python string syntax). That is, the string r"\n" does not contain an actual newline, but it contains something that tells the regular expression engine to look for actual newlines.
If you want to search for a backslash followed by n, you need to use r"\\n".
The point of the raw strings is that they block Python's basic intepretation of string escapes, allowing you to use the backslash for its regular-expression meaning. If you don't want the regular-expression meaning, you still have to use two backslashes, as in my example above. But without raw strings it would be even worse: if you wanted to search for literal backslash-n without a raw string, you'd have to use "\\\\n". If the raw string blocked interpretation of the regular expression special characters (so that plain "\n" really meant backslash-n), you wouldn't have any way of using the regular expression syntax at all.
i am very new to regular expression and trying get "\" character using python
normally i can escape "\" like this
print ("\\");
print ("i am \\nit");
output
\
i am \nit
but when i use the same in regX it didn't work as i thought
print (re.findall(r'\\',"i am \\nit"));
and return me output
['\\']
can someone please explain why
EDIT: The problem is actually how print works with lists & strings. It prints the representation of the string, not the string itself, the representation of a string containing just a backslash is '\\'. So findall is actually finding the single backslash correctly, but print isn't printing it as you'd expect. Try:
>>> print(re.findall(r'\\',"i am \\nit")[0])
\
(The following is my original answer, it can be ignored (it's entirely irrelevant), I'd misinterpreted the question initially. But it seems to have been upvoted a bit, so I'll leave it here.)
The r prefix on a string means the string is in "raw" mode, that is, \ are not treated as special characters (it doesn't have anything to do with "regex").
However, r'\' doesn't work, as you can't end a raw string with a backslash, it's stated in the docs:
Even in a raw string, string quotes can be escaped with a backslash, but the backslash remains in the string; for example, r"\"" is a valid string literal consisting of two characters: a backslash and a double quote; r"\" is not a valid string literal (even a raw string cannot end in an odd number of backslashes). Specifically, a raw string cannot end in a single backslash (since the backslash would escape the following quote character).
But you actually can use a non-raw string to get a single backslash: "\\".
can someone please explain why
Because re.findall found one match, and the match text consisted of a backslash. It gave you a list with one element, which is a string, which has one character, which is a backslash.
That is written ['\\'] because '\\' is how you write "a string with one backslash" - just like you had to do when you wrote the example code print "\\".
Note that you're using two different kinds of string literal here -- there's the regular string "a string" and the raw string r"a raw string". Regular string literals observe backslash escaping, so to actually put a backslash in the string, you need to escape it too. Raw string literals treat backslashes like any other character, so you're more limited in which characters you can actually put in the string (no specials that need an escape code) but it's easier to enter things like regular expressions, because you don't need to double up backslashes if you need to add a backslash to have meaning inside the string, not just when creating the string.
It is unnecessary to escape backslashes in raw strings, unless the backslash immediately precedes the closing quote.