We Keep Coding

Remove "characters with encodings larger than 3 bytes" using Python 3

I want to remove characters with encodings larger than 3 bytes.
Because when I upload my CSV data to Amazon Mechanical Turk system, it asks me to do it.
Your CSV file needs to be UTF-8 encoded and cannot contain characters
with encodings larger than 3 bytes. For example, some non-English
characters are not allowed (learn more).
To overcome this problem,
I want to make a filter_max3bytes funciton to remove those characters in Python3.
x = 'below ð\x9f~\x83,'
y = remove_max3byes(x) # y=="below ~,"
Then I will apply the function before saving it to a CSV file, which is UTF-8 encoded.
This post is related my problem, but they uses python 2 and the solution did not worked for me.
Thank you!

None of the characters in your string seems to take 3 bytes in UTF-8:
x = 'below ð\x9f~\x83,'
Anyway, the way to remove them, if there were any would be:
filtered_x = ''.join(char for char in x if len(char.encode('utf-8')) < 3)
For example (with such characters):
>>> x = 'abcd漢字efg'
>>> ''.join(char for char in x if len(char.encode('utf-8')) < 3)
'abcdefg'
BTW, you can verify that your original string does not have 3-byte encodings by doing the following:
>>> for char in 'below ð\x9f~\x83,':
... print(char, [hex(b) for b in char.encode('utf-8')])
...
b ['0x62']
e ['0x65']
l ['0x6c']
o ['0x6f']
w ['0x77']
['0x20']
ð ['0xc3', '0xb0']
['0xc2', '0x9f']
~ ['0x7e']
['0xc2', '0x83']
, ['0x2c']
EDIT: A wild guess
I believe the OP asks the wrong question and the question is in fact whether the character is printable. I'll assume anything Python displays as \x<number> is not printable, so this solution should work:
x = 'below ð\x9f~\x83,'
filtered_x = ''.join(char for char in x if not repr(char).startswith("'\\x"))
Result:
'below ð~,'

While indirectly stated, the website only allows characters from the Basic Multilingual Plane (BMP). That includes Unicode code points U+0000 to U+FFFF. In UTF-8, it takes four bytes to encode anything above U+FFFF:
>>> '\uffff'.encode('utf8')
b'\xef\xbf\xbf'
>>> '\U00010000'.encode('utf8')
b'\xf0\x90\x80\x80'
This filters out Unicode code points above U+FFFF:
>>> test_string = 'abc马克😀' # emoticon is U+1F600
>>> ''.join(c for c in test_string if ord(c) < 0x10000)
'abc马克'
When encoded (note three bytes for each Chinese character):
>>> ''.join(c for c in test_string if ord(c) < 0x10000).encode('utf8')
b'abc\xe9\xa9\xac\xe5\x85\x8b'

According to the UTF-8 standard, characters with Unicode code points below U+0800 will use at most two bytes in the encoding. So just remove any character at or above U+0800. This code copies all characters that take at most two bytes and just leave out the other characters.
def remove_max3byes(x):
return ''.join(c for c in x if ord(c) < 0x800)
As a comment pointed out, your example string has no characters that take more than two bytes. But this command at the REPL
remove_max3byes(chr(0x07ff))
gives
'\u07ff'
and this command
remove_max3byes(chr(0x0800))
gives
''
Both are as wanted.

python write file dealing with encode

I'm confused. I need HELP!!!
I'm dealing with a file contains Chinese characters,for instance, let's call it a.TEST, and here is what's inside.
你好 中国 Hello China 1 2 3
You don't need to understand what the chinese means.(Actually it's 'hello China')
>>> f=open('wr.TRAIN')
>>> print f.read()
你好 中国 Hello China 1 2 3
>>> f.seek(0)
>>> content = f.readline()
>>> content
'\xe4\xbd\xa0\xe5\xa5\xbd \xe4\xb8\xad\xe5\x9b\xbd Hello China 1 2 3\n'
>>> print content
你好 中国 Hello China 1 2 3
>>> type(content)
<type 'str'>
>>> isinstance(content,unicode)
False
Here comes the first Question: Why python shell give me the utf-8of content when i just type content,meanwhile print content cmd can output the form that I want to see?
The Second Question: what's the difference between unicode and str?
Someone told me that encode is convert unicode to str, but what i learned from Unicode HowTo tells me encode is convert unicode to utf-8
Not over yet! :)
here is test.py
#!/usr/bin/python
#-*- coding: utf-8 -*-
fr = open('a.TEST')
fw = open('out.TEST','w')
content = fr.readline()
content_list = content.split()
print content
fw.write('{0}'.format(content_list))
fr.close()
fw.close()
Third Question:Why the chinese character turn into utf-8 code when I do .split()?
and I thought fw.write('{0}'.format(content_list).decode('utf-8')) will work, but it doesn't.
I don't want what's written into out.TEST is character encoding form, I want it to be exactly the character that look like originally（你好）. How to do it?

What is Encoding
A file consists of bytes. You can represent each byte with a number between 0 and 255 (or 0x00 and 0xFF in hexadecimal).
Text is also written as bytes. There is an agreement on the way text is written. That is an encoding. The most basic encoding is ASCII and other encodings are usually based on it. For example, ASCII defines that number 65 (0x41) represents 'A', 66 (0x42) represents 'B' etc.
How are Strings Represented
In python, you can define a string using numeric values:
>>> '\x41\x42\x43'
'ABC'
'\x41\x42\x43' is exactly the same thing as 'ABC'. Python will always represent the string using the more readable textual representation ('ABC').
However, some numeric values are not printable characters, so they will be represented in numeric form:
>>> '\x00\x01\x02\x03\x04'
'\x00\x01\x02\x03\x04'
Others characters have aliases to make your job easier:
>>> '\x0a\x0d\x09'
'\n\r\t'
Different Encodings
ASCII table defines meaning of numbers 0-127 and includes only the english alphabet. Numbers 128-255 are undefined. So, other encodings define a meaning for 128-255. Yet others change the meaning of the whole range 0-255.
There are many encodings and they define 128-255 differently.
For example, character 185 (0xB9) is ą in windows-1250 encoding, but it is š in iso-8859-2 encoding.
So, what happens if you print \xb9? It depends on the encoding used in the console. In my case (my console uses cp852 encoding) it is:
>>> print '\xb9'
╣
Because of that ambiguity, string '\xb9' will never be represented as '╣' (nor 'ą'...). That would hide the true value. It will be represented as the numeric value:
>>> '\xb9'
'\xb9'
Also:
>>> '╣'
'\xb9'
See also the string from the question in my console:
>>> content = '\xe4\xbd\xa0\xe5\xa5\xbd \xe4\xb8\xad\xe5\x9b\xbd Hello China 1 2 3\n'
>>>
>>> content
'\xe4\xbd\xa0\xe5\xa5\xbd \xe4\xb8\xad\xe5\x9b\xbd Hello China 1 2 3\n'
>>>
>>> print content
ńŻáňąŻ ńŞşňŤŻ Hello China 1 2 3
But what happens if variable is just entered in the console?
When a variable is enteren in cosole without print, its representation is printed. It is the same as the following:
>>> print repr(content)
'\xe4\xbd\xa0\xe5\xa5\xbd \xe4\xb8\xad\xe5\x9b\xbd Hello China 1 2 3\n'
What is Unicode?
Unicode table aims to define a numeric representation of all characters in the world and more. It can actually do that, because it is not limited to 256 values (or to any other limit actually). This is not an encoding, but a universal mapping of numbers to characters.
For example, unicode defines that number 353 (0x0161) is character š. That is allways true regardless of your locale and encodings you use. That character can be stored in files (or memory) in any encoding which supports š.
What is UTF-8?
When encoding a unicode character, one can use any encoding, but not all of them will support all characters.
For example, š (unicode 0x0161) can be encoded in iso-8869-2 as 0xB9, but it cannot be encoded in iso-8869-1 at all.
So, to be able to encode anything, you need an encoding which supports every unicode character. UTF-8 is one of those encodings, but there are others:
>>> u'\u0161'.encode('utf-7')
'+AWE-'
>>> u'\u0161'.encode('utf-8')
'\xc5\xa1'
>>> u'\u0161'.encode('utf-16le')
'a\x01'
>>> u'\u0161'.encode('utf-16be')
'\x01a'
>>> u'\u0161'.encode('utf-32le')
'a\x01\x00\x00'
>>> u'\u0161'.encode('utf-32be')
'\x00\x00\x01a'
The good thing about utf-8 is that the whole ASCII range is unchanged and as long as only ASCII is used, only one byte is used per character:
>>> u'abcdefg'.encode('utf-8')
'abcdefg'
Unicode in Python 2
Important: This is really specific to Python 2. Python 3 is different.
Unlike str objects, which are strings of bytes, unicode objects are strings of unicode characters.
They can be encoded into a str in chosen encoding, or decoded from str in chosen encoding.
A unicode string is specified using u before the opening quote. The characters inside are interpreted using current encoding, or they can be specified in numeric format \uHEX:
>>> u'ABCD'
u'ABCD'
>>>
>>> u'\u0041\u0042\u0043'
u'ABC'
>>> u'šâů'
u'\u0161\xe2\u016f'
And Now the Answers
First Question
contents prints repr(contents)
print contents prints contents
Second Question
UTF-8 strings are byte strings (str). You get them by encoding the unicode:
>>> u'\u0161'.encode('utf-8')
'\xc5\xa1'
>>> '\xc5\xa1'.decode('utf-8')
u'\u0161'
So yes, encode converts unicode to str. The str can be utf-8, but it does not have to be.
Third Question
A) "Why the chinese character turn into utf-8 code when I do .split()?"
They were utf-8 all the time.
B) "I thought fw.write('{0}'.format(content_list).decode('utf-8')) will work"
content_list is not a string. It is a list. When a list is converted to a string, it is done using its repr, which also does repr of all of the contents.
For example:
>>> 'a \n a \n a'
'a \n a \n a'
>>> print 'a \n a \n a'
a
a
a
>>> print ['a \n a \n a']
['a \n a \n a']
The last print printed repr(list) which contains repr(str).

In the beginning, there was just english characters, and people was not satisfied.
Then they want to display every character in the world.But there is problem. One byte can only represent 255 characters. There just simply not enough place to hold them.
Then people decide to use two byte to represent one character.And call it 'utf8'.
No matter what characters you write in, it's all store in byte form.
In Python, there is no such datatype called 'unicode', just 'str'. And 'unicode' is an encoding system of 'str'.
'\xe4\xbd\xa0\xe5\xa5\xbd \xe4\xb8\xad\xe5\x9b\xbd' is byte form of "你好 中国".
It can not display without an encoding system specified.
I suppose you could blame linux/unix. Python has no problem to display 'utf-8' characters, while 'cat' cannot.

Weird behaviour when trying to print characters of a byte string

Why this short code behaves differently from a run to other ?
# -*- coding: utf-8 -*-
for c in 'aɣyul':
print c
The outputs that I have in each run are:
# nothing
---
a
---
l
---
u
l
---
a
y
u
l
...etc
EDIT:
I know how to solve the problem, the question is just why Python prints a different part of the string, instead of the same part, at each run ?

You need to add an u at leading of your string which make that python treads with your string as an unicode, and decode your character while printing:
>>> for c in u'aɣyul':
... print c
...
a
ɣ
y
u
l
Note that without encoding python will break the unicode character in two separate hex value and in each print you will get the string representation of this hex values:
>>> 'aɣyul'
'a\xc9\xa3yul'
^ ^
If you want to know that why python break the unicode to 2 hex value that's because of that instances of str contain raw 8-bit values while a unicode character used more than 8 bit memory.
You can also decode the hex values manually:
>>> print '\xc9\xa3'.decode('utf8')
ɣ

Decode unicode string in python

I'd like to decode the following string:
t\u028c\u02c8m\u0251\u0279o\u028a\u032f
It should be the IPA of 'tomorrow' as given in a JSON string from http://rhymebrain.com/talk?function=getWordInfo&word=tomorrow
My understanding is that it should be something like:
x = 't\u028c\u02c8m\u0251\u0279o\u028a\u032f'
print x.decode()
I have tried the solutions from here , here , here, and here (and several other that more or less apply), and several permutations of its parts, but I can't get it to work.
Thank you

You need a u before your string (in Python 2.x, which you appear to be using) to indicate that this is a unicode string:
>>> x = u't\u028c\u02c8m\u0251\u0279o\u028a\u032f' # note the u
>>> print x
tʌˈmɑɹoʊ̯
If you have already stored the string in a variable, you can use the following constructor to convert the string into unicode:
>>> s = 't\u028c\u02c8m\u0251\u0279o\u028a\u032f' # your string has a unicode-escape encoding but is not unicode
>>> x = unicode(s, encoding='unicode-escape')
>>> print x
tʌˈmɑɹoʊ̯
>>> x
u't\u028c\u02c8m\u0251\u0279o\u028a\u032f' # a unicode string

Bytes in a unicode Python string

In Python 2, Unicode strings may contain both unicode and bytes:
a = u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \xd0\xb5\xd0\xba'
I understand that this is absolutely not something one should write in his own code, but this is a string that I have to deal with.
The bytes in the string above are UTF-8 for ек (Unicode \u0435\u043a).
My objective is to get a unicode string containing everything in Unicode, which is to say Русский ек (\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \u0435\u043a).
Encoding it to UTF-8 yields
>>> a.encode('utf-8')
'\xd0\xa0\xd1\x83\xd1\x81\xd1\x81\xd0\xba\xd0\xb8\xd0\xb9 \xc3\x90\xc2\xb5\xc3\x90\xc2\xba'
Which then decoded from UTF-8 gives the initial string with bytes in them, which is not good:
>>> a.encode('utf-8').decode('utf-8')
u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \xd0\xb5\xd0\xba'
I found a hacky way to solve the problem, however:
>>> repr(a)
"u'\\u0420\\u0443\\u0441\\u0441\\u043a\\u0438\\u0439 \\xd0\\xb5\\xd0\\xba'"
>>> eval(repr(a)[1:])
'\\u0420\\u0443\\u0441\\u0441\\u043a\\u0438\\u0439 \xd0\xb5\xd0\xba'
>>> s = eval(repr(a)[1:]).decode('utf8')
>>> s
u'\\u0420\\u0443\\u0441\\u0441\\u043a\\u0438\\u0439 \u0435\u043a'
# Almost there, the bytes are proper now but the former real-unicode characters
# are now escaped with \u's; need to un-escape them.
>>> import re
>>> re.sub(u'\\\\u([a-f\\d]+)', lambda x : unichr(int(x.group(1), 16)), s)
u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \u0435\u043a' # Success!
This works fine but looks very hacky due to its use of eval, repr, and then additional regex'ing of the unicode string representation. Is there a cleaner way?

In Python 2, Unicode strings may contain both unicode and bytes:
No, they may not. They contain Unicode characters.
Within the original string, \xd0 is not a byte that's part of a UTF-8 encoding. It is the Unicode character with code point 208. u'\xd0' == u'\u00d0'. It just happens that the repr for Unicode strings in Python 2 prefers to represent characters with \x escapes where possible (i.e. code points < 256).
There is no way to look at the string and tell that the \xd0 byte is supposed to be part of some UTF-8 encoded character, or if it actually stands for that Unicode character by itself.
However, if you assume that you can always interpret those values as encoded ones, you could try writing something that analyzes each character in turn (use ord to convert to a code-point integer), decodes characters < 256 as UTF-8, and passes characters >= 256 as they were.

(In response to the comments above): this code converts everything that looks like utf8 and leaves other codepoints as is:
a = u'\u0420\u0443\u0441 utf:\xd0\xb5\xd0\xba bytes:bl\xe4\xe4'
def convert(s):
try:
return s.group(0).encode('latin1').decode('utf8')
except:
return s.group(0)
import re
a = re.sub(r'[\x80-\xFF]+', convert, a)
print a.encode('utf8')
Result:
Рус utf:ек bytes:blää

The problem is that your string is not actually encoded in a specific encoding. Your example string:
a = u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \xd0\xb5\xd0\xba'
Is mixing python's internal representation of unicode strings with utf-8 encoded text. If we just consider the 'special' characters:
>>> orig = u'\u0435\u043a'
>>> bytes = u'\xd0\xb5\xd0\xba'
>>> print orig
ек
>>> print bytes
ек
But you say, bytes is utf-8 encoded:
>>> print bytes.encode('utf-8')
ек
>>> print bytes.encode('utf-8').decode('utf-8')
ек
Wrong! But what about:
>>> bytes = '\xd0\xb5\xd0\xba'
>>> print bytes
ек
>>> print bytes.decode('utf-8')
ек
Hurrah.
So. What does this mean for me? It means you're (probably) solving the wrong problem. What you should be asking us/trying to figure out is why your strings are in this form to begin with and how to avoid it/fix it before you have them all mixed up.

You should convert unichrs to chrs, then decode them.
u'\xd0' == u'\u00d0' is True
$ python
>>> import re
>>> a = u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \xd0\xb5\xd0\xba'
>>> re.sub(r'[\000-\377]*', lambda m:''.join([chr(ord(i)) for i in m.group(0)]).decode('utf8'), a)
u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \u0435\u043a'
r'[\000-\377]*' will match unichrs u'[\u0000-\u00ff]*'
u'\xd0\xb5\xd0\xba' == u'\u00d0\u00b5\u00d0\u00ba'
You use utf8 encoded bytes as unicode code points (this is the PROBLEM)
I solve the problem by pretending those mistaken unichars as the corresponding bytes
I search all these mistaken unichars, and convert them to chars, then decode them.
If I'm wrong, please tell me.

You've already got an answer, but here's a way to unscramble UTF-8-like Unicode sequences that is less likely to decode latin-1 Unicode sequences in error. The re.sub function:
Matches Unicode characters < U+0100 that resemble valid UTF-8 sequences (ref: RFC 3629).
Encodes the Unicode sequence into its equivalent latin-1 byte sequence.
Decodes the sequence using UTF-8 back into Unicode.
Replaces the original UTF-8-like sequence with the matching Unicode character.
Note this could still match a Unicode sequence if just the right characters appear next to each other, but it is much less likely.
import re
# your example
a = u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \xd0\xb5\xd0\xba'
# printable Unicode characters < 256.
a += ''.join(chr(n) for n in range(32,256)).decode('latin1')
# a few UTF-8 characters decoded as latin1.
a += ''.join(unichr(n) for n in [2**7-1,2**7,2**11-1,2**11]).encode('utf8').decode('latin1')
# Some non-BMP characters
a += u'\U00010000\U0010FFFF'.encode('utf8').decode('latin1')
print repr(a)
# Unicode codepoint sequences that resemble UTF-8 sequences.
p = re.compile(ur'''(?x)
\xF0[\x90-\xBF][\x80-\xBF]{2} | # Valid 4-byte sequences
[\xF1-\xF3][\x80-\xBF]{3} |
\xF4[\x80-\x8F][\x80-\xBF]{2} |
\xE0[\xA0-\xBF][\x80-\xBF] | # Valid 3-byte sequences
[\xE1-\xEC][\x80-\xBF]{2} |
\xED[\x80-\x9F][\x80-\xBF] |
[\xEE-\xEF][\x80-\xBF]{2} |
[\xC2-\xDF][\x80-\xBF] # Valid 2-byte sequences
''')
def replace(m):
return m.group(0).encode('latin1').decode('utf8')
print
print repr(p.sub(replace,a))
###Output
u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \xd0\xb5\xd0\xba
!"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff\x7f\xc2\x80\xdf\xbf\xe0\xa0\x80\xf0\x90\x80\x80\xf4\x8f\xbf\xbf'
u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \u0435\u043a
!"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff\x7f\x80\u07ff\u0800\U00010000\U0010ffff'

I solved it by
unicodeText.encode("utf-8").decode("unicode-escape").encode("latin1")