We Keep Coding

Removing different string patterns from Pandas column

I have the following column which consists of email subject headers:
Subject
EXT || Transport enquiry
EXT || RE: EXTERNAL: RE: 0001 || Copy of enquiry
EXT || FW: Model - Jan
SV: [EXTERNAL] Calculations
What I want to achieve is:
Subject
Transport enquiry
0001 || Copy of enquiry
Model - Jan
Calculations
and for this I am using the below code which only takes into account the first regular expression that I am passing and ignoring the rest
def clean_subject_prelim(text):
text = re.sub(r'^EXT \|\| $' , '' , text)
text = re.sub(r'EXT \|\| RE: EXTERNAL: RE:', '' , text)
text = re.sub(r'EXT \|\| FW:', '' , text)
text = re.sub(r'^SV: \[EXTERNAL]$' , '' , text)
return text
df['subject_clean'] = df['Subject'].apply(lambda x: clean_subject_prelim(x))
Why this is not working, what am I missing here?

You can use
pattern = r"""(?mx) # MULTILINE mode on
^ # start of string
(?: # non-capturing group start
EXT\s*\|\|\s*(?:RE:\s*EXTERNAL:\s*RE:|FW:)? # EXT || or EXT || RE: EXTERNAL: RE: or EXT || FW:
| # or
SV:\s*\[EXTERNAL]# SV: [EXTERNAL]
) # non-capturing group end
\s* # zero or more whitespaces
"""
df['subject_clean'] = df['Subject'].str.replace(pattern', '', regex=True)
See the regex demo.
Since the re.X ((?x)) is used, you should escape literal spaces and # chars, or just use \s* or \s+ to match zero/one or more whitespaces.

Get rid of the $ sign in the first expression and switch some of regex expressions from place. Like this:
import pandas as pd
import re
def clean_subject_prelim(text):
text = re.sub(r'EXT \|\| RE: EXTERNAL: RE:', '' , text)
text = re.sub(r'EXT \|\| FW:', '' , text)
text = re.sub(r'^EXT \|\|' , '' , text)
text = re.sub(r'^SV: \[EXTERNAL]' , '' , text)
return text
data = {"Subject": [
"EXT || Transport enquiry",
"EXT || RE: EXTERNAL: RE: 0001 || Copy of enquiry",
"EXT || FW: Model - Jan",
"SV: [EXTERNAL] Calculations"]}
df = pd.DataFrame(data)
df['subject_clean'] = df['Subject'].apply(lambda x: clean_subject_prelim(x))

validation-remove currency symbol from price

I have this one string, which is actually price, this price value comes with any currency symbol (currency_list), I am trying to remove these currency symbols from price and return only price.\
Till now I am able to do it for prefix and suffix currency symbol using below code , everything works till here.
I just want to add one validation where if the symbol is not prefix or suffix like "200$434" in btw, then it should return not valid format. which I am not able to understand how should be implemented.
currency_list = ['USD', 'UNITED STATES DOLLAR', '$', 'EUR', 'EURO', '€', 'GBP','BRITISH POUND', '£']
Normally input string can be
"$1212212"
"1212212EURO"
"1212212"
"1212212 BRITISH POUND"
need help to validate values like "1212$343" or "1212212EURO323.23"
Code:
for symb in currency_list:
if symb in amount:
data = amount.replace(symb, '')

After going through multiple blog post, I found this answer which gets the job done.
def validateCurrency(amount):
new_amount=None
for cur in currency_list:
if amount.startswith(cur) or amount.endswith(cur):
new_amount = amount.replace(cur, "", 1)
if new_amount == None:
return "Currency is not valid a string."
return f"Price after removeing symbol is {new_amount}"
// print(validateCurrency('$1212212'))

You can use regex to achieve your purpose.
import re
currency_list = ['USD', 'UNITED STATES DOLLAR', '$', 'EUR', 'EURO', '€', 'GBP', 'BRITISH POUND', '£']
p = re.compile(r'([\D]*)([\d]+\.?[\d]+)(.*)')
def verify_or_get_amount(amount):
first, mid, last = [i.strip() for i in p.search(amount).groups()]
if (first and first not in currency_list) or (last and last not in currency_list):
print('invalid:', amount)
else:
amount = mid
print('amount:', amount)
return mid
for i in ['EURO123', 'EURO 123', 'EURO 123.', 'EURO .12', 'EURO 12.12', '$1212212', '1212212EURO', '1212212', '1212212 BRITISH POUND', '1212$343']:
verify_or_get_amount(i)

using regex:
import re
currency_list = ['USD', 'UNITED STATES DOLLAR', '\$', 'EUR', 'EURO', '€', 'GBP', 'BRITISH POUND', '£']
currencies = '|'.join(currency_list)
c = re.compile(rf'^({currencies})? *(\d+(\.\d+)?) *({currencies})?$')
for i in ['$1212212', '1212212EURO', '1212212', '1212212 BRITISH POUND', '1212$343']:
match_obj = c.match(i)
if match_obj:
print(match_obj.group(2))
else:
print('not found')
output :
1212212
1212212
1212212
1212212
not found
Explanation :
to see actual pattern : print(c.pattern) which gives :
^(USD|UNITED STATES DOLLAR|\$|EUR|EURO|€|GBP|BRITISH POUND|£)?(\d+(\.\d+)?) *(USD|UNITED STATES DOLLAR|\$|EUR|EURO|€|GBP|BRITISH POUND|£)?$
I've escaped $ in the currency_list.
currencies = '|'.join(currency_list) for building possible prefixes or suffixes.
(\d+(\.\d+)?) is for matching price which accept float as well. (you can omit the (\.\d+) part)
the * that you see in regex, is for for example BRITISH POUND which have a space after the number.

I am assuming you want a currency validation function
def validateCurrency(input):
input_length = len(input)
if input.isdigit():return False
split = [re.findall(r'(\D+?)(\d+)|(\d+?)(\D+)', input)[0] ]
total_length = 0
for i in split[0]:
if i in currency_list:
total_length+=len(i)
if str(i).isdigit():
total_length+=len(i)
if total_length == input_length:
return True
else:
return False

replace trademark symbol (™) when alone

I'm trying to remove trademark symbol (™) but only in the case it's not followed by any other symbol for instance I might have ’ which is a bad encoding of quotation mark (') so I don't want to remove trademark symbol (™) and hence broking the pattern that i'm using to replace xx™ with quotation mark.
dict = {};
chars = {
'\xe2\x84\xa2': '', # ™
'\xe2\x80\x99': "'", # ’
}
def stats_change(char, number):
if dict.has_key(char):
dict[char] = dict[char]+number
else:
dict[char] = number # Add new entry
def replace_chars(match):
char = match.group(0)
stats_change(char,1)
return chars[char]
i, nmatches = re.subn("(\\" + '|\\'.join(chars.keys()) + ")", replace_chars, i)
count_matches += nmatches
Input: foo™ oof
Output: foo oof
Input: o’f oof
Output: o'f oof
Any suggestions ?

How to find and replace case sensitive whole words in python

Consider the below mcve:
import re
import textwrap
import traceback
import unittest
def replace_words(content, replacements):
rc = re.compile(r"[A-Za-z_]\w*")
def translate(match):
word = match.group(0)
return replacements.get(word, word)
return rc.sub(translate, content, re.IGNORECASE | re.MULTILINE)
class class_name(unittest.TestCase):
def setUp(self):
self.replacements = [
{
'PLUS': '"+"',
'DASH': '"-"',
'BANG': '"!"',
'TILDE': '"~"',
'STAR': '"*"',
'SLASH': '"/"',
'PERCENT': '"%"',
'LEFT_PAREN': '"("',
'RIGHT_PAREN': '")"'
}, {
"IF": "fi",
"FOO": "oof",
"BAR": "rab",
"OP_FOO": "oof_op"
}
]
self.texts = [
textwrap.dedent("""\
variable_identifier :
IDENTIFIER
primary_expression :
foo1
foo2
foo3
LEFT_PAREN expression RIGHT_PAREN
unary_operator :
PLUS
DASH
BANG
TILDE
multiplicative_expression :
unary_expression
multiplicative_expression STAR unary_expression
multiplicative_expression SLASH unary_expression
multiplicative_expression PERCENT unary_expression\
"""),
textwrap.dedent("""\
IF identifier IDENTIFIER FOO BAR BARycentric
OP_FOO
""")
]
self.expected_results = [
textwrap.dedent("""\
variable_identifier :
IDENTIFIER
primary_expression :
foo1
foo2
foo3
"(" expression ")"
unary_operator :
"+"
"-"
"!"
"~"
multiplicative_expression :
unary_expression
multiplicative_expression "*" unary_expression
multiplicative_expression "/" unary_expression
multiplicative_expression "%" unary_expression\
"""),
textwrap.dedent("""\
fi identifier IDENTIFIER oof rab BARycentric
oof_op
""")
]
def _tester(self, f):
replacements = self.replacements
expected_results = self.expected_results
texts = self.texts
self.assertEqual(f(texts[0], replacements[0]), expected_results[0])
self.assertEqual(f(texts[1], replacements[1]), expected_results[1])
def test_replace_words(self):
self._tester(replace_words)
if __name__ == "__main__":
unittest.main()
replace_words function is attempting to search and replace case sensitive whole words in a given text using a dictionary of replacements above code but it will fail in the line self.assertEqual(f(texts[0], replacements[0]), expected_results[0]) though and I don't know why.
So the question would be, how do you find and replace case sensitive whole words using a replacements dictionary in python?

You can use re.sub and re.findall:
import re
def regex_string(d, to_lower = False):
if not to_lower:
return '|'.join(r'\b{}\b'.format(i) for i in d.keys())
return '|'.join([c for b in [[r'\b{}\b'.format(i.lower()), r'\b{}\b'.format(i)] for i in d.keys()] for c in b])
replacements = {
'PLUS': '"+"',
'DASH': '"-"',
'BANG': '"!"',
'TILDE': '"~"',
'STAR': '"*"',
'SLASH': '"/"',
'PERCENT': '"%"',
'LEFT_PAREN': '"("',
'RIGHT_PAREN': '")"'
}
replaced = re.sub(regex_string(replacements, True), '{}', content)
final_result = replaced.format(*[replacements.get(i, i) for i in re.findall(regex_string(replacements, True), content)])
Output (case 1):
variable_identifier :
IDENTIFIER
primary_expression :
foo1
foo2
foo3
"(" expression ")"
unary_operator :
"+"
"-"
"!"
"~"
multiplicative_expression :
unary_expression
multiplicative_expression "*" unary_expression
multiplicative_expression "/" unary_expression
multiplicative_expression "%" unary_expression
Output (case 2):
fi identifier IDENTIFIER oof rab BARycentric
oof_op
Or, even shorter:
replaced = re.sub(regex_string(replacements, True), lambda x:replacements.get(x.group(), x.group()), content)

How to replace some special characters from user input for different Python platforms

I need to replace some special characters from user input for different platform (i.e. Linux and Windows) using Python. Here is my code:
if request.method == 'POST':
rname1 = request.POST.get('react')
Here I am getting the user input by post method. I need to the following characters to remove from the user input (if there is any).
1- Escape or filter special characters for windows, ( ) < > * ‘ = ? ; [ ] ^ ~ ! . ” % # / \ : + , `
2- Escape or filter special characters for Linux, { } ( ) < > * ‘ = ? ; [ ] $ – # ~ ! . ” % / \ : + , `
The special characters are given above. Here I need to remove for both Linux and Windows.

Python strings have a built in method translate for substitution/deletion of characters. You need to build a translation table and then call the function.
import sys
if "win" in sys.platform:
special = """( ) < > * ‘ = ? ; [ ] ^ ~ ! . ” % # / \ : + , `""".split()
else:
special = """{ } ( ) < > * ‘ = ? ; [ ] $ – # ~ ! . ” % / \ : + , `""".split()
trans_dict = {character: None for character in special}
trans_table = str.maketrans(trans_dict)
print("Lo+=r?e~~m ipsum dol;or sit!! amet, consectet..ur ad%".translate(trans_table))
Will print Lorem ipsum dolor sit amet consectetur ad.
If you want to use a replacement character instead of deleting, then replace None above with the character. You can build a translation table with specific substitutions, `{"a": "m", "b": "n", ...}
Edit: The above snippet is indeed in Python3. In Python2 (TiO) it's easier to delete characters:
>>> import sys
>>> import string
>>> if "win" in sys.platform:
... special = """()<>*'=?;[]^~!%#/\:=,`"""
... else:
... special = """{}()<>*'=?;[]$-#~!."%/\:+"""
...
>>> s = "Lo+r?e~~/\#<>m ips()u;m"
>>> string.translate(s, None, special)
'Lorem ipsum'
Note that I've substituted ‘ with ' and similarly replaced ” with " because I think you're only dealing with ascii strings.