We Keep Coding

Python - Split string on first occurrence of non-allowable characters

I have some python code where i want to scan and split string on first occurrence of non-allowable characters.
import re,string
mystring="my_id=abc-something_123&anything#;?lcdkahck;my_id%3Dkckdkkj_bcjc"
if "my_id=" in mystring:
mystring = mystring[mystring.index("my_id=") + 6 : len(mystring)][0:100]
mystring = re.split('[;&#]', mystring)[0]
print(mystring)
What happens in this, I get string correctly where ;&# is coming, but my data can have any unpredictable character put of ;&#.
What i tried drive out these characters
allowable_character = '-' + '_' + string.ascii_letters + string.digits
mystring = re.sub('[^%s]' % allowable_character, '', mystring)
print(mystring)
But this just filters the string with characters that are not in 'allowable_character'.
What i am trying to achieve is to split string once the character which is not in 'allowable_character' and return that string.
So I want expected output as 'abc-something_123'
Any help is appreciated here

You could just use re.findall here:
mystring = "my_id=abc-something_123&anything#;?lcdkahck;my_id%3Dkckdkkj_bcjc"
match = re.findall(r'^my_id=([\w-]*).*$', mystring)[0]
print(match)
This prints:
'abc-something_123'

Python Removing non-alphabetical characters with exceptions

I am having a hard time doing Data Analysis on a large text that has lots of non-alphabetical chars. I tried using
string = filter(str.isalnum, string)
but I also have "#" in my text that I want to keep. How do I make an exception for a character like "#" ?

It is easier to use regular expressions:
string = re.sub("[^A-Za-z0-9#]", "", string)

You can use re.sub
re.sub(r'[^\w\s\d#]', '', string)
Example:
>>> re.sub(r'[^\w\s\d#]', '', 'This is # string 123 *$^%')
This is # string 123

One way to do this would be to create a function that returns True or False if an input character is valid.
import string
valid_characters = string.ascii_letters + string.digits + '#'
def is_valid_character(character):
return character in valid_characters
# Instead of using `filter`, we `join` all characters in the input string
# if `is_valid_character` is `True`.
def get_valid_characters(string):
return "".join(char for char in string if is_valid_character(char))
Some example output:
>>> print(valid_characters)
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789#
>>> get_valid_characters("!Hello_#world?")
'Helloworld'
>>> get_valid_characters("user#example")
'user#example'
A simpler way to write it would be using regex. This will accomplish the same thing:
import re
def get_valid_characters(string):
return re.sub(r"[^\w\d#]", "", string)

You could use a lambda function to specify your allowed characters. But also note that filter returns a <filter object> which is an iterator over the returned values. So you will have to stich it back to a string:
string = "?filter_#->me3!"
extra_chars = "#!"
filtered_object = filter(lambda c: c.isalnum() or c in extra_chars, string)
string = "".join(filtered_object)
print(string)
Gives:
filter#me3!

Non-ASCII characters appearing in string

So I need to compare 2 strings :
str1 = 'this is my string/ndone'
str2 = 'this is my string done'
So I replace the new line from str1 with ' ':
new_str = str1.replace('\n', ' ')
And when print the 2 strings there are identical:
'this is my string done'
But when compared using == operator the not so I convert this 2 strings into array to see why they are not equal:
arr1 = bytearray(str1 , 'utf-8')
print(arr1)
arr2 = bytearray(str2 , 'utf-8')
print(arr2)
And this is the output:
str1 = bytearray(b'this is\xc2\xa0my string done')
str2 = bytearray(b'this is my string done')
So what is this \xc2\xa0 ?

'\xc2\xa0' is the UTF-8 encoding of the Unicode character 'NO-BREAK SPACE' (U+00A0).

use python unidecode library
from unidecode import unidecode
str = "this is\xc2\xa0my string done"
print(unidecode(str))
o/p
this isA my string done

== is working in comparing two string
str1 = 'this is my string\ndone'
str2 = 'this is my string done'
str1 = str1.replace("\n"," ")
print(str1)
if (str1 == str2):
print("y")
else:
print("n")
and output is
this is my string done
y

As stated elsewhere your string had a "/n" not "\n" in it.
Assuming though that what you wanted to do was normalise all whitespace characters, this is a very handy trick I use all the time:
string = ' '.join(string.split())
Update: Okay this is why:
If you don't specify what string.split() should use a separater the, per docs:
If sep is not specified or is None, a different splitting algorithm is
applied: runs of consecutive whitespace are regarded as a single
separator, and the result will contain no empty strings at the start
or end if the string has leading or trailing whitespace.
So it splits on whitspaces, and treats multiple whitespaces as a single seperator. I don't know what characters are all defined as "whitespaces", but is certainly includes all the usual suspects. Then when you rejoin the list into a string with ' '.join(), you know for sure that all whitespaces are now the same.

Python: How to remove [' and ']?

I want to remove [' from start and '] characters from the end of a string.
This is my text:
"['45453656565']"
I need to have this text:
"45453656565"
I've tried to use str.replace
text = text.replace("['","");
but it does not work.

You need to strip your text by passing the unwanted characters to str.strip() method:
>>> s = "['45453656565']"
>>>
>>> s.strip("[']")
'45453656565'
Or if you want to convert it to integer you can simply pass the striped result to int function:
>>> try:
... val = int(s.strip("[']"))
... except ValueError:
... print("Invalid string")
...
>>> val
45453656565

Using re.sub:
>>> my_str = "['45453656565']"
>>> import re
>>> re.sub("['\]\[]","",my_str)
'45453656565'

You could loop over the character filtering if the element is a digit:
>>> number_array = "['34325235235']"
>>> int(''.join(c for c in number_array if c.isdigit()))
34325235235
This solution works even for both "['34325235235']" and '["34325235235"]' and whatever other combination of number and characters.
You also can import a package and use a regular expresion to get it:
>>> import re
>>> theString = "['34325235235']"
>>> int(re.sub(r'\D', '', theString)) # Optionally parse to int

Instead of hacking your data by stripping brackets, you should edit the script that created it to print out just the numbers. E.g., instead of lazily doing
output.write(str(mylist))
you can write
for elt in mylist:
output.write(elt + "\n")
Then when you read your data back in, it'll contain the numbers (as strings) without any quotes, commas or brackets.

Breaking up substrings in Python based on characters

I am trying to write code that will take a string and remove specific data from it. I know that the data will look like the line below, and I only need the data within the " " marks, not the marks themselves.
inputString = 'type="NN" span="123..145" confidence="1.0" '
Is there a way to take a Substring of a string within two characters to know the start and stop points?

You can extract all the text between pairs of " characters using regular expressions:
import re
inputString='type="NN" span="123..145" confidence="1.0" '
pat=re.compile('"([^"]*)"')
while True:
mat=pat.search(inputString)
if mat is None:
break
strings.append(mat.group(1))
inputString=inputString[mat.end():]
print strings
or, easier:
import re
inputString='type="NN" span="123..145" confidence="1.0" '
strings=re.findall('"([^"]*)"', inputString)
print strings
Output for both versions:
['NN', '123..145', '1.0']

fields = inputString.split('"')
print fields[1], fields[3], fields[5]

You could split the string at each space to get a list of 'key="value"' substrings and then use regular expressions to parse the substrings.
Using your input string:
>>> input_string = 'type="NN" span="123..145" confidence="1.0" '
>>> input_string_split = input_string.split()
>>> print input_string_split
[ 'type="NN"', 'span="123..145"', 'confidence="1.0"' ]
Then use regular expressions:
>>> import re
>>> pattern = r'"([^"]+)"'
>>> for substring in input_string_split:
match_obj = search(pattern, substring)
print match_obj.group(1)
NN
123..145
1.0
The regular expression '"([^"]+)"' matches anything within quotation marks (provided there is at least one character). The round brackets indicate the bit of the regular expression that you are interested in.