We Keep Coding

How to replace '..' and '?.' with single periods and question marks in pandas? df['column'].str.replace not working

This is a follow up to this SO post which gives a solution to replace text in a string column
How to replace text in a column of a Pandas dataframe?
df['range'] = df['range'].str.replace(',','-')
However, this doesn't seem to work with double periods or a question mark followed by a period
testList = ['this is a.. test stence', 'for which is ?. was a time']
testDf = pd.DataFrame(testList, columns=['strings'])
testDf['strings'].str.replace('..', '.').head()
results in
0 ...........e
1 .............
Name: strings, dtype: object
and
testDf['strings'].str.replace('?.', '?').head()
results in
error: nothing to repeat at position 0

Add regex=False parameter, because as you can see in the docs, regex it's by default True:
-regex bool, default True
Determines if assumes the passed-in pattern is a regular expression:
If True, assumes the passed-in pattern is a regular expression.
And ? . are special characters in regular expressions.
So, one way to do it without regex will be this double replacing:
testDf['strings'].str.replace('..', '.',regex=False).str.replace('?.', '?',regex=False)
Output:
strings
0 this is a. test stence
1 for which is ? was a time

Replace using regular expression. In this case, replace any sepcial character '.' followed immediately by white space. This is abit curly, I advice you go with #Mark Reed answer.
testDf.replace(regex=r'([.](?=\s))', value=r'')
strings
0 this is a. test stence
1 for which is ? was a time

str.replace() works with a Regex where . is a special character which denotes "any" character. If you want a literal dot, you need to escape it: "\.". Same for other special Regex characters like ?.

First, be aware that the Pandas replace method is different from the standard Python one, which operates only on fixed strings. The Pandas one can behave as either the regular string.replace or re.sub (the regular-expression substitute method), depending on the value of a flag, and the default is to act like re.sub. So you need to treat your first argument as a regular expression. That means you do have to change the string, but it also has the benefit of allowing you to do both substitutions in a single call.
A regular expression isn't a string to be searched for literally, but a pattern that acts as instructions telling Python what to look for. Most characters just ask Python to match themselves, but some are special, and both . and ? happen to be in the special category.
The easiest thing to do is to use a character class to match either . or ? followed by a period, and remember which one it was so that it can be included in the replacement, just without the following period. That looks like this:
testDF.replace(regex=r'([.?])\.', value=r'\1')
The [.?] means "match either a period or a question mark"; since they're inside the [...], those normally-special characters don't need to be escaped. The parentheses around the square brackets tell Python to remember which of those two characters is the one it actually found. The next thing that has to be there in order to match is the period you're trying to get rid of, which has to be escaped with a backslash because this one's not inside [...].
In the replacement, the special sequence \1 means "whatever you found that matched the pattern between the first set of parentheses", so that's either the period or question mark. Since that's the entire replacement, the following period is removed.
Now, you'll notice I used raw strings (r'...') for both; that keeps Python from doing its own interpretation of the backslashes before replace can. If the replacement were just '\1' without the r it would replace them with character code 1 (control-A) instead of the first matched group.

To replace both the ? and . at the same time you can separate by | (the regex OR operator).
testDf['strings'].str.replace('\?.|\..', '.')
Prefix the .. with a \, because you need to escape as . is a regex character:
testDf['strings'].str.replace('\..', '.')
You can do the same with the ?, which is another regex character.
testDf['strings'].str.replace('\?.', '.')

Why '\A' in python regex doesn't work inside [ ]?

I was trying to get a regex which would match a word in the beginning of the line or after certain word. I tried:
r"[\A|my_word](smth)"
But it failed because it doesn't match the \A in that case. What's wrong with that?
It turns out that \A doesn't work inside []:
In [163]: type(re.search(r"\A123", "123"))
Out[163]: <type '_sre.SRE_Match'>
In [164]: type(re.search(r"[\A]123", "123"))
Out[164]: <type 'NoneType'>
But I don't understand why.
I'm using Python 2.6.6
EDIT:
After some comments I realized that the example I used with [\A|my_word] is bad. The actual expression is [\AV] to match either beginning of the string or V. The main problem I had is that I was curious why [\A] doesn't work.

My understanding of backslashes in bracket character classes was off, it seems, but even so, it is the case that [\A|my_word] is equivalent to [A|my_word] and will try to match a single one of A, |, m, y, _, w, o, r, or d before smth.
Here's a regular expression that should do what you want; unfortunately, a lookbehind can't be used in Python due to \A and my_word having different lengths, but a non-capturing group can be used instead: (?:\A|abc)(smth).
(You can also use ^ instead of \A if you want, though the usage may differ in multiline mode as ^ will also match at the start of each new line [or rather, immediately after every newline] in that mode.)

Anchors vs Character Classes
\A is an anchor that matches a position in the string - in this case the position before the first char in the string. Other anchors are \b: word boundary, ^: start of string/line, $: end of string/line, (?=...): Positive lookahead, (?!...): negative lookahead, etc. Anchors consume no characters and only match a position within the string.
[abc] is a character class that always matches exactly one character - in this case either a, b or c
Thus, placing an anchor inside a character class makes no sense.

[\A] matches a single character that is either a \ or an A. This is probably not what you wanted.

The \ character in the brackets clauses loses its special meaning as escaping character.
I.e. in [ ] it will treat as two characters: \ and A.
[REF]
Regex referencies:
The Single UNIX Specification
Python 2.6 - re module
UPDATE
Bracket expression is special case iteself, thus that special sequences like \A (almost control commands for regex) will work there is very unlikely. It's somehow unnatural...
ONE MORE THING
As stated from Python reference:
(brackets) Used to indicate a set of characters.
\A is special sequence which:
Matches only at the start of the string.
It is obviously not a character of any set, I know \n NEWLINE, but I've never heard about STARTLINE (maybe pretty one).
Also, for escapists:
You could even put ] into bracket without escaping it, if it comes right after the starting [ left bracket:
The pattern []] will match ']', for example.

Regular expression sub

I have a question about regular expression sub in python. So, I have some lines of code and what I want is to replace all floating point values eg: 2.0f,-1.0f...etc..to doubles 2.0,-1.0. I came up with this regular expression '[-+]?[0-9]*\.?[0-9]+f' and it finds what I need but I am not sure how to replace it?
so here's what I have:
# check if floating point value exists
if re.findall('[-+]?[0-9]*\.?[0-9]+f', line):
line = re.sub('[-+]?[0-9]*\.?[0-9]+f', ????? ,line)
I am not sure what to put under ????? such that it will replace what I found in '[-+]?[0-9]*\.?[0-9]+f' without the char f in the end of the string.
Also there might be more than one floating point values, which is why I used re.findall
Any help would be great. Thanks

Capture the part of the text you want to save in a capturing group and use the \1 substitution operator:
line = re.sub(r'([-+]?[0-9]*\.?[0-9]+)f', r'\1' ,line)
Note that findall (or any kind of searching) is unnecessary since re.sub will look for the pattern itself and return the string unchanged if there are no matches.
Now, for several regular expression writing tips:
Always use raw strings (r'...') for regular expressions and substitution strings, otherwise you will need to double your backslashes to escape them from Python's string parser. It is only by accident that you didn't need to do this for \., since . is not part of an escape sequence in Python strings.
Use \d instead of [0-9] to match a digit. They are equivalent, but \d is easier to recognize for "digit", while [0-9] needs to be visually verified.
Your regular expression will not recognize 10.f, which is likely a valid decimal number in your input. Matching floating-point numbers in various formats is trickier than it seems at first, but simple googling will reveal many reasonably complete solutions for this.
The re.X flag will allow you to add arbitrary whitespace and even comments to your regexp. With small regexps that can seem downright silly, but for large expressions the added clarity is a life-saver. (Your regular expression is close to the threshold.)
Here is an example of an extended regular expression that implements the above style tips:
line = re.sub(r'''
( [-+]?
(?: \d+ (?: \.\d* )? # 12 or 12. or 12.34
|
\.\d+ # .12
)
) f''',
r'\1', line, flags=re.X)
((?:...) is a non-capturing group, only used for precedence.)

This is my goto reference for all things regex.
http://www.regular-expressions.info/named.html
The result should be something like:
line = re.sub('(<first>[-+]?[0-9]*\).?[0-9]+f', '\g<first>', line)

Surround the part of the regex you want to "keep" in a "capture group", e.g.
'([-+]?[0-9]*\.?[0-9]+)f'
^ ^
And then you can refer to these capture groups using \1 in your substitution:
r'\1'
For future reference, you can have many capture groups, i.e. \2, \3, etc. by order of the opening parentheses.

Some Regex stuff

I'm just starting to figure out regex and would love some help trying to understand it. I've been using this to help me get started, but am still having some trouble figuring it out.
What I am trying to do is take this text:
<td>8.54/10 over 190 reviews</td>
And pull out the "8.54", so basically anything in between the first ">" and the "/"
Using my noob skills, I came up with this: [0-9].[0-9][0-9], which WILL match that 8.54, and will work for everything BUT 10.00, which I do need to account for.
Can anyone help me refine my expression to apply to that last case as well?

Use quantifiers.
You want one or more digits, followed by a dot, followed by one or more digits. A digit can also be written \d, and the "one or more" quantifier is +.
The dot needs to be escaped as it is a regex metacharacter which means "any character". Your regex therefore should be:
\d+\.\d+
Now, beware that a quantifier applies to atoms only. Character classes ([...]), complemented character classes ([^...]) and special character classes (\d, \w...) are atoms, however if you want to apply a quantifier to more than a simple atom, you'll need to group these atoms using the grouping operator, (). Ie, (ab)+ will look for one or more of ab.

Maybe answered my own question. Found this:
[0-9]+(?:.[0-9]*)
It seems to work, does anyone have any changes to this?

\d is often used instead of [0-9] (mnemonically, “digit”) and it's necessary to remember that sometimes fractional numbers are written without any digits before the decimal point. Thus:
(?<=>)(?:\d+(?:\.\d*)?|\.\d+)(?=/)
OK, that's a bit of a complex RE. Here's how it breaks down (in extended form).
(?<= > ) # With a “>” before (but not matched)…
(?: # … match either this
\d+ # at least one digit, followed by…
(?: # …match
\. \d* # a dot followed by any number of digits
) ? # optionally
| # … or this
\. \d+ # a dot followed by at least one digit
) #
(?= / ) # … and with a “/” afterwards (but not matched)

This might work:
\>(.*?)/
# (.*?) is a "non-greedy" group which maches as few characters as possible
Then access the actual value using
m.group(1)
where m is the match object returned by re.search or re.finditer
If you want to access the value directly (re.findall), use
(?>=\>)(.*?)(?=/)

python regular expresssion for a string

consider this string
prison break: proof of innocence (2006) {abduction (#1.10)}
i just want to know whether there is (# floating point value )} in the string or not
i tried few regular expressions like
re.search('\(\#+\f+\)\}',xyz)
and
re.search('\(\#+(\d\.\d)+\)\}',xyz)
nothing worked though...can someone suggest me something here

Try r'\(#\d+\.\d+\)\}'
The (, ), ., and } are all special metacharacters, that's why they're preceded by \, so they're matched literally instead.
You also need to apply the + repetition at the right element. Here it's attached to the \d -- the shorthand for digit character class -- to mean that only the digits can appear one-or-more times.
The use of r'raw string literals' makes it easier to work with regex patterns because you don't have to escape backslashes excessively.
See also
What exactly do u and r string flags in Python do, and what are raw string literals?
Variations
For instructional purposes, let's consider a few variations. This will show a few basic features of regex. Let's first consider one of the attempted patterns:
\(\#+(\d\.\d)+\)\}
Let's space out the parts for readability:
\( \#+ ( \d \. \d )+ \) \}
\__________/
this is one group, repeated with +
So this pattern matches:
A literal (, followed by one-or-more #
Followed by one-or-more of:
A digit, a literal dot, and a digit
Followed by a literal )}
Thus, the pattern will match e.g. (###1.23.45.6)} (as seen on rubular.com). Obviously this is not the pattern we want.
Now let's try to modify the solution pattern and say that perhaps we also want to allow just a sequence of digits, without the subsequent period and following digits. We can do this by grouping that part (…), and making it optional with ?.
BEFORE
\(#\d+\.\d+\)\}
\___/
let's make this optional! (…)?
AFTER
\(#\d+(\.\d+)?\)\}
Now the pattern matches e.g. (#1.23)} as well as e.g. (#666)} (as seen on rubular.com).
References
regular-expressions.info - Optional, Brackets for Grouping

"Escape everything" and use raw-literal syntax for safety:
>>> s='prison break: proof of innocence (2006) {abduction (#1.10)}'
>>> re.search(r'\(\#\d+\.\d+\)\}', s)
<_sre.SRE_Match object at 0xec950>
>>> _.group()
'(#1.10)}'
>>>
This assumes that by "floating point value" you mean "one or more digits, a dot, one or more digits", and is not tolerant of other floating point syntax variations, multiple hashes (which you appear from your RE patterns to want to support but don't mention in your Q's text), arbitrary whitespace among the relevant parts (again, unclear from your Q whether you need it), ... -- some issues can be adjusted pretty easily, others "not so much" (it's particularly hard to guess what gamut of FP syntax variations you want to support, for example).