This question already has answers here:
Best way to replace multiple characters in a string?
(16 answers)
Closed 3 years ago.
We are working on the Vigenere cipher in my computer science class and one of the first steps our teacher wants us to take is to delete all whitespace, punctuation, and capitalization from a string.
#pre-process - removing spaces, punctuation, and capitalization
def pre_process(s):
str = s.lower()
s = (str.replace(" ", "") + str.replace("!?'.", ""))
return s
print(pre_process("We're having a surprise birthday party for Eve!"))
What I want the output to be is "werehavingasurpisebirthdaypartyforeve" but what I'm actually getting is "we'rehavingasurprisebirthdaypartyforeve!we're having a surprise birthday party for eve!"
You should use regex instead of string replace. Try this code.
import re
mystr="We're having a surprise birthday party for Eve!"
#here you can pass as many punctuations you want
result=re.sub("[.'!#$%&\'()*+,-./:;<=>?#[\\]^ `{|}~]","",mystr)
print(result.lower())
str.replace("!?'.", "")) replaces only the string !?'., not any of the four characters on their own.
You need to use a separate replace call for each character, or otherwise use regular expressions.
The reason your solution does not work, is because it is attempting to remove the literal string "!?'.", and not each character individually.
One way to accomplish this would be the following:
import re
regex = re.compile('[^a-zA-Z]')
s = "We're having a surprise birthday party for Eve!"
s = regex.sub('', s).lower()
import re
def preprocess(s):
return re.sub(r'[\W_]', '', s).lower()
re.sub removes all non-alphanumeric characters (everything except A-Z and 0-9).
lower() removes capitalization.
An approach without using RegEx.
>>> import string
>>> s
"We're having a surprise birthday party for Eve!"
>>> s.lower().translate(None, string.punctuation).replace(" ", "")
'werehavingasurprisebirthdaypartyforeve'
Change your code as below:-
def pre_process(s):
str = s.lower()
s = (str.replace(" ", ""))
s= s.replace("!", "")
s= s.replace("'", "")
return s
print(pre_process("We're having a surprise birthday party for Eve!"))
str.translate is also an option. you can create a translation table using str.maketrans where the first arguments (ascii_uppercase) will be translated to the second ones (ascii_lowercase). the third argument (punctuation + whitespace) is a list of the characters you want deleted:
from string import ascii_lowercase, ascii_uppercase, punctuation, whitespace
table = str.maketrans(ascii_uppercase, ascii_lowercase, punctuation + whitespace)
s = "We're having a surprise birthday party for Eve!"
print(s.translate(table))
# werehavingasurprisebirthdaypartyforeve
once you have the table initialized every subsequent string can just be converted by applying
s.translate(table)
You could use re ?,
>>> import re
>>> x
"We're having a surprise birthday party for Eve!"
>>> re.sub(r'[^a-zA-Z0-9]', '', x).lower() # negate the search. Fastest among the two :)
'werehavingasurprisebirthdaypartyforeve'
or list comprehension ?
>>> import string
>>> ''.join(y for y in x if y in string.ascii_letters).lower()
'werehavingasurprisebirthdaypartyforeve'
Just a benchmark,
>>> timeit.timeit("''.join(y for y in x if y in string.ascii_letters).lower()", setup='import string;x = "We\'re having a surprise birthday party for Eve!"')
7.747261047363281
>>> timeit.timeit("re.sub(r'[^a-zA-Z0-9]', '', x).lower()", setup='import re;x = "We\'re having a surprise birthday party for Eve!"')
2.912994146347046
Related
I want to find out if a substring is contained in the string and remove it from it without touching the rest of the string. The thing is that the substring pattern that I have to perform the search on is not exactly what will be contained in the string. In particular the problem is due to spanish accent vocals and, at the same time, uppercase substring, so for example:
myString = 'I'm júst a tésting stríng'
substring = 'TESTING'
Perform something to obtain:
resultingString = 'I'm júst a stríng'
Right now I've read that difflib library can compare two strings and weight it similarity somehow, but I'm not sure how to implement this for my case (without mentioning that I failed to install this lib).
Thanks!
This normalize() method might be a little overkill and maybe using the code from #Harpe at https://stackoverflow.com/a/71591988/218663 works fine.
Here I am going to break the original string into "words" and then join all the non-matching words back into a string:
import unicodedata
def normalize(text):
return unicodedata.normalize("NFD", text).encode('ascii', 'ignore').decode('utf-8').lower()
myString = "I'm júst a tésting stríng"
substring = "TESTING"
newString = " ".join(word for word in myString.split(" ") if normalize(word) != normalize(substring))
print(newString)
giving you:
I'm júst a stríng
If your "substring" could be multi-word I might think about switching strategies to a regex:
import re
import unicodedata
def normalize(text):
return unicodedata.normalize("NFD", text).encode('ascii', 'ignore').decode('utf-8').lower()
myString = "I'm júst á tésting stríng"
substring = "A TESTING"
match = re.search(f"\\s{ normalize(substring) }\\s", normalize(myString))
if match:
found_at = match.span()
first_part = myString[:found_at[0]]
second_part = myString[found_at[1]:]
print(f"{first_part} {second_part}".strip())
I think that will give you:
I'm júst stríng
You can use the package unicodedata to normalize accented letters to ascii code letters like so:
import unicodedata
output = unicodedata.normalize('NFD', "I'm júst a tésting stríng").encode('ascii', 'ignore')
print(str(output))
which will give
b"I'm just a testing string"
You can then compare this with your input
"TESTING".lower() in str(output).lower()
which should return True.
This capitalizes even the letter after the apostrophe which wasn't the intended result,
>>> test = "there's a need to capitalize every word"
>>> test.title()
"There'S A Need To Capitalize Every Word"
some people suggest using capwords, but capwords seems to be crippled(only
capitalizing words preceded by whitespace). In this case I also need to be able to capitalize words separated by periods (eg: one.two.three should result on One.Two.Three).
Is there a method that doesn't fail where capwords and title do?
There's a solution to your exact problem in python's docs, here:
>>>
>>> import re
>>> def titlecase(s):
... return re.sub(r"[A-Za-z]+('[A-Za-z]+)?",
... lambda mo: mo.group(0)[0].upper() +
... mo.group(0)[1:].lower(),
... s)
...
Use string.capwords
import string
string.capwords("there's a need to capitalize every word")
You may use anonymous function in the replacement part of re.sub
>>> import re
>>> test = "there's a need to capitalize every word"
>>> re.sub(r'\b[a-z]', lambda m: m.group().upper(), test)
"There'S A Need To Capitalize Every Word"
I am trying to write code that will take a string and remove specific data from it. I know that the data will look like the line below, and I only need the data within the " " marks, not the marks themselves.
inputString = 'type="NN" span="123..145" confidence="1.0" '
Is there a way to take a Substring of a string within two characters to know the start and stop points?
You can extract all the text between pairs of " characters using regular expressions:
import re
inputString='type="NN" span="123..145" confidence="1.0" '
pat=re.compile('"([^"]*)"')
while True:
mat=pat.search(inputString)
if mat is None:
break
strings.append(mat.group(1))
inputString=inputString[mat.end():]
print strings
or, easier:
import re
inputString='type="NN" span="123..145" confidence="1.0" '
strings=re.findall('"([^"]*)"', inputString)
print strings
Output for both versions:
['NN', '123..145', '1.0']
fields = inputString.split('"')
print fields[1], fields[3], fields[5]
You could split the string at each space to get a list of 'key="value"' substrings and then use regular expressions to parse the substrings.
Using your input string:
>>> input_string = 'type="NN" span="123..145" confidence="1.0" '
>>> input_string_split = input_string.split()
>>> print input_string_split
[ 'type="NN"', 'span="123..145"', 'confidence="1.0"' ]
Then use regular expressions:
>>> import re
>>> pattern = r'"([^"]+)"'
>>> for substring in input_string_split:
match_obj = search(pattern, substring)
print match_obj.group(1)
NN
123..145
1.0
The regular expression '"([^"]+)"' matches anything within quotation marks (provided there is at least one character). The round brackets indicate the bit of the regular expression that you are interested in.
How to define a function that takes a string (sentence) and inserts an extra space after a period if the period is directly followed by a letter.
sent = "This is a test.Start testing!"
def normal(sent):
list_of_words = sent.split()
...
This should print out
"This is a test. Start testing!"
I suppose I should use split() to brake a string into a list, but what next?
P.S. The solution has to be as simple as possible.
Use re.sub. Your regular expression will match a period (\.) followed by a letter ([a-zA-Z]). Your replacement string will contain a reference to the second group (\2), which was the letter matched in the regular expression.
>>> import re
>>> re.sub(r'\.([a-zA-Z])', r'. \1', 'This is a test.This is a test. 4.5 balloons.')
'This is a test. This is a test. 4.5 balloons'
Note the choice of [a-zA-Z] for the regular expression. This matches just letters. We do not use \w because it would insert spaces into a decimal number.
One-liner non-regex answer:
def normal(sent):
return ".".join(" " + s if i > 0 and s[0].isalpha() else s for i, s in enumerate(sent.split(".")))
Here is a multi-line version using a similar approach. You may find it more readable.
def normal(sent):
sent = sent.split(".")
result = sent[:1]
for item in sent[1:]:
if item[0].isalpha():
item = " " + item
result.append(item)
return ".".join(result)
Using a regex is probably the better way, though.
Brute force without any checks:
>>> sent = "This is a test.Start testing!"
>>> k = sent.split('.')
>>> ". ".join(l)
'This is a test. Start testing!'
>>>
For removing spaces:
>>> sent = "This is a test. Start testing!"
>>> k = sent.split('.')
>>> l = [x.lstrip(' ') for x in k]
>>> ". ".join(l)
'This is a test. Start testing!'
>>>
Another regex-based solution, might be a tiny bit faster than Steven's (only one pattern match, and a blacklist instead of a whitelist):
import re
re.sub(r'\.([^\s])', r'. \1', some_string)
Improving pyfunc's answer:
sent="This is a test.Start testing!"
k=sent.split('.')
k='. '.join(k)
k.replace('. ','. ')
'This is a test. Start testing!'
I am close but I am not sure what to do with the restuling match object. If I do
p = re.search('[/#.* /]', str)
I'll get any words that start with # and end up with a space. This is what I want. However this returns a Match object that I dont' know what to do with. What's the most computationally efficient way of finding and returning a string which is prefixed with a #?
For example,
"Hi there #guy"
After doing the proper calculations, I would be returned
guy
The following regular expression do what you need:
import re
s = "Hi there #guy"
p = re.search(r'#(\w+)', s)
print p.group(1)
It will also work for the following string formats:
s = "Hi there #guy " # notice the trailing space
s = "Hi there #guy," # notice the trailing comma
s = "Hi there #guy and" # notice the next word
s = "Hi there #guy22" # notice the trailing numbers
s = "Hi there #22guy" # notice the leading numbers
That regex does not do what you think it does.
s = "Hi there #guy"
p = re.search(r'#([^ ]+)', s) # this is the regex you described
print p.group(1) # first thing matched inside of ( .. )
But as usually with regex, there are tons of examples that break this, for example if the text is s = "Hi there #guy, what's with the comma?" the result would be guy,.
So you really need to think about every possible thing you want and don't want to match. r'#([a-zA-Z]+)' might be a good starting point, it literally only matches letters (a .. z, no unicode etc).
p.group(0) should return guy. If you want to find out what function an object has, you can use the dir(p) method to find out. This will return a list of attributes and methods that are available for that object instance.
As it's evident from the answers so far regex is the most efficient solution for your problem. Answers differ slightly regarding what you allow to be followed by the #:
[^ ] anything but space
\w in python-2.x is equivalent to [A-Za-z0-9_], in py3k is locale dependent
If you have better idea what characters might be included in the user name you might adjust your regex to reflect that, e.g., only lower case ascii letters, would be:
[a-z]
NB: I skipped quantifiers for simplicity.
(?<=#)\w+
will match a word if it's preceded by a # (without adding it to the match, a so-called positive lookbehind). This will match "words" that are composed of letters, numbers, and/or underscore; if you don't want those, use (?<=#)[^\W\d_]+
In Python:
>>> strg = "Hi there #guy!"
>>> p = re.search(r'(?<=#)\w+', strg)
>>> p.group()
'guy'
You say: """If I do p = re.search('[/#.* /]', str) I'll get any words that start with # and end up with a space."" But this is incorrect -- that pattern is a character class which will match ONE character in the set #/.* and space. Note: there's a redundant second / in the pattern.
For example:
>>> re.findall('[/#.* /]', 'xxx#foo x/x.x*x xxxx')
['#', ' ', '/', '.', '*', ' ']
>>>
You say that you want "guy" returned from "Hi there #guy" but that conflicts with "and end up with a space".
Please edit your question to include what you really want/need to match.