We Keep Coding

What character set is "é" from? (Python: Filename with "é", how to use os.path.exists , filecmp.cmp, shutil.move?)

What Character set is é from? In Windows notepad having this character in an ANSI text file will save fine. Insert something like 😍 and you'll get an error. é seems to work fine in ASCII terminal in Putty (Are CP437 and IBM437 the same?) where as 😍 does not.
I can see that 😍 is Unicode, not ASCII. But what is é? It doesn't give errors I get with Unicode in Notepad, but Python was throwing SyntaxError: Non-ASCII character '\xc3' in file on line , but no encoding declared; before I added a "magic comment" as suggested by Python NLTK: SyntaxError: Non-ASCII character '\xc3' in file (Sentiment Analysis -NLP).
I added the "magic comment" and don't get that error, but os.path.isfile() is saying a filename with é doesn't exist. Ironic that the character é is in Marc-André Lemburg, the author of the PEP the error links to.
EDIT: If I print the path of the file, the accented e shows up as ├⌐ but I can copy and paste é into the command prompt.
EDIT2: See below
Private > cat scratch.py ### LOL cat scratch :3
# coding=utf-8
file_name = r"Filéname"
file_name = unicode(file_name)
Private > python scratch.py
Traceback (most recent call last):
File "scratch.py", line 3, in <module>
file_name = unicode(file_name)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
Private >
EDIT3:
Private > PS1="Private > " ; echo code below ; cat scratch.py ; echo ======= ; echo output below ; python scratch.py
code below
# -*- coding: utf-8 -*-
file_name = r"Filéname"
file_name = unicode(file_name, encoding="utf-8")
# I have code here to determine a path depending on the hostname of the
# machine, the folder paths contain no Unicode characters, for my debug
# version of the script, I will hardcode the redacted hostname.
hostname = "One"
if hostname == "One":
folder = "C:/path/folder_one"
elif hostname == "Two":
folder = "C:/path/folder_two"
else:
folder = "C:/path/folder_three"
path = "%s/%s" % (folder, file_name)
path = unicode(path, encoding="utf-8")
print path
=======
output below
Traceback (most recent call last):
File "scratch.py", line 18, in <module>
path = unicode(path, encoding="utf-8")
TypeError: decoding Unicode is not supported
Private >

You need to tell unicode what encoding the string is in, in this case it's utf-8 not ascii, and the file header should be # -*- coding: utf-8 -*-, Encoding Declarations
# -*- coding: utf-8 -*-
file_name = r"Filéname"
file_name = unicode(file_name, encoding="utf-8")
1 Help on class unicode in module __builtin__:
2
3 class unicode(basestring)
4 | unicode(object='') -> unicode object
5 | unicode(string[, encoding[, errors]]) -> unicode object
6 |
7 | Create a new Unicode object from the given encoded string.
8 | encoding defaults to the current default string encoding.
9 | errors can be 'strict', 'replace' or 'ignore' and defaults to 'strict'.
And as I mentioned in my previous comment you will save yourself a lot of headaches by switching to Python 3. Python 2 on a Windows filesystem with unicode characters can be a nightmare.

Why not monkey patch sys.getfilesystemencoding()?

In Python can read the filesystem encoding with sys.getfilesystemencoding().
But there seems to be no official way to set the filesystem encoding.
See: How to change file system encoding via python?
I found this dirty hack:
import sys
sys.getfilesystemencoding = lambda: 'UTF-8'
Is there a better solution, if changing environment variable LANG before starting the interpreter is not an option?
Background, why I want this:
This works:
user#host:~$ python src/setfilesystemencoding.py
LANG: de_DE.UTF-8
sys.getdefaultencoding(): ascii
sys.getfilesystemencoding(): UTF-8
This does not work:
user#host:~$ LANG=C python src/setfilesystemencoding.py
LANG: C
sys.getdefaultencoding(): ascii
sys.getfilesystemencoding(): ANSI_X3.4-1968
Traceback (most recent call last):
File "src/setfilesystemencoding.py", line 10, in <module>
with open('/tmp/german-umlauts-üöä', 'wb') as fd:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 20-22: ordinal not in range(128)
Here is the simple script:
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, unicode_literals, print_function
import os, sys
print('LANG: {}'.format(os.environ['LANG']))
print('sys.getdefaultencoding(): {}'.format(sys.getdefaultencoding()))
print('sys.getfilesystemencoding(): {}'.format(sys.getfilesystemencoding()))
with open('/tmp/german-umlauts-üöä', 'wb') as fd:
fd.write('foo')
I hopped that above monkey patching would solve this ... but it doesn't. Sorry, this question does not make sense any more. I close it.
My solution: use LANG=C.UTF-8

Decoding with 'utf-8' but error show an Unicode Encode Error?

My Python 2.x script trys to download a web page including Chinese words. It's encoded in UTF-8. By urllib.openurl(url), I get content in type str, so I decode content with UTF-8. It throws UnicodeEncodeError. I googled a lot of posts like this and this, but they don't work for me. Am I misunderstand something?
My code is:
import urllib
import httplib
def get_html_content(url):
response = urllib.urlopen(url)
html = response.read()
print type(html)
return html
if __name__ == '__main__':
url = 'http://weekly.manong.io/issues/58'
html = get_html_content(url)
print html.decode('utf-8')
Error message:
<type 'str'>
Traceback (most recent call last):
File "E:\src\infra.py", line 32, in <module>
print html.decode('utf-8')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 44: ordinal not in range(128)
[Finished in 1.6s]

print statement converts arguments to str objects. Encoding it manually will prevent to encode it with ascii:
import sys
...
if __name__ == '__main__':
url = 'http://weekly.manong.io/issues/58'
html = get_html_content(url)
print html.decode('utf-8').encode(sys.stdout.encoding, 'ignore')
Replace sys.stdout.encoding with encoding of your terminal unless it print correctly.
UPDATE
Alternatively you can use PYTHONIOENCODING environmental variable without encoding in the source code:
PYTHONIOENCODING=utf-8:ignore python program.py

If the standard output is redirected to a pipe then Python 2 fails to use your locale encoding:
⟫ python -c'print u"\u201c"' # no redirection -- works
“
⟫ python -c'print u"\u201c"' | cat
Traceback (most recent call last):
File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 0: ordinal not in range(128)
To fix it; you could specify PYTHONEIOENCODING environment variable e.g., in bash:
⟫ PYTHONIOENCODING=utf-8 python -c'print u"\u201c"' | cat
“
On Windows, you need to set the envvar using a different syntax.
If your Windows console doesn't support utf-8 (it matters only for the first command where there is no redirection) then you could try to print Unicode directly using Win32 API calls like win-unicode-console does. See windows console doesn't print or input Unicode.

Searching and replacing cent symbol in Python

OS: CentOS 6.5
Python version: 2.7.5
I have a file with the following sample of information.
I would like to search and replace the cent symbol and replace with $0. infront.
Alpha $1.00
Beta ¢55 <<<< note
Charlie $2.00
Delta ¢23 <<<< note
I want it to look like this:
Alpha $1.00
Beta $0.55 <<<< note
Charlie $2.00
Delta $0.23 <<<< note
So this code in command line (which works) is:
sed 's/¢/$0./g' *file name*
However using python to code it does not work:
import subprocess
hello = subprocess.call('cat datafile ' + '| sed "s/¢/$0./g"',shell=True)
print hello
There seems to be an error whenever I try to paste the ¢ symbol.
Slightly closer, when I print the unicode for the cent sign in Python, it comes out below:
print(u"\u00A2")
¢
When I cat my datafile, it actually shows up as the ¢ sign, missing the Â. << not sure if this is any help
I think when I'm trying to sed with the Unicode, the added symbol before the ¢ is not allowing me to search and replace.
Error code when trying unicode:
hello = subprocess.call(u"cat datafile | sed 's/\uxA2/$0./g'",shell=True)
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 25-26: truncated \uXXXX escape
Fixing the uxA2 to u00A2, i get this:
sed: -e expression #1, char 7: unknown option to `s'
1
Any ideas/thoughts?
Both examples I get the error below:
[root#centOS user]# python test2.py
Traceback (most recent call last):
File "test2.py", line 3, in <module>
data = data.decode('utf-8') # decode immediately to Unicode
File "/usr/local/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa2 in position 6: invalid start byte
[root#centOS user]# python test1.py
Traceback (most recent call last):
File "test1.py", line 11, in <module>
hello_unicode = hello_utf8.decode('utf-8')
File "/usr/local/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa2 in position 6: invalid start byte
This is the cat of the file:
[root#centOS user]# cat datafile
alpha ¢79
this is the Nano of the datafile:
alpha �79
This is the Vim of the datafile:
[root#centOS user]# vim fbasdf
alpha ¢79
~
Thanks again for all your help guys
ANSWER!!
The SED output from Rob and Thomas works.
File format was saved as charset=iso-8859-1. I was unable to search the document for utf-8 format character.
Identified file charset:
file -bi datafile
text/plain; charset=iso-8859-1
Used following code to change file:
iconv -f iso-8859-1 -t utf8 datafile > datafile1

Stealing Thomas's answer and expanding on it:
import subprocess
# Keep all strings in unicode as long as you can.
cmd_unicode = u"sed 's/\u00A2/$0./g' < datafile"
# only convert them to encoded byte strings when you send them out
# also note the use of .check_output(), NOT .call()
cmd_utf8 = cmd_unicode.encode('utf-8')
hello_utf8 = subprocess.check_output(cmd_utf8, shell=True)
# Decode any incoming byte string to unicode immediately on receipt
hello_unicode = hello_utf8.decode('utf-8')
# And you have your answer
print hello_unicode
The code above demonstrates the use of a "Unicode sandwich": bytes on the outside, Unicode on the inside. See http://nedbatchelder.com/text/unipain.html
For this simple example, you could have just as easily done everything in Python:
with open('datafile') as datafile:
data = datafile.read() # Read in bytes
data = data.decode('utf-8') # decode immediately to Unicode
data = data.replace(u'\xa2', u'$0.') # Do all operations in Unicode
print data # Implicit encode during output

Also, change your string to a unicode string, and replace the cent sign with \u00A2.
Here's the fixed code:
import subprocess
hello = subprocess.call(u"cat datafile | sed \"s#\u00A2#$0.#g\"",shell=True)
print hello

"cannot execute binary file" error in python

I keep getting the following error:
$ ./test.py
-bash: ./test.py: cannot execute binary file
when trying to run the following file in python via cygwin:
#!usr/bin/python
with open("input.txt") as inf:
try:
while True:
latin = inf.next().strip()
gloss = inf.next().strip()
trans = inf.next().strip()
process(latin, gloss, trans)
inf.next() # skip blank line
except StopIteration:
# reached end of file
pass
from itertools import chain
def chunk(s):
"""Split a string on whitespace or hyphens"""
return chain(*(c.split("-") for c in s.split()))
def process(latin, gloss, trans):
chunks = zip(chunk(latin), chunk(gloss))
How do I fix this??
After taking on the below suggestions, still getting the same error.
If this helps, I tried
$ python ./test.py
and got
$ python ./test.py
File "./test.py", line 1
SyntaxError: Non-ASCII character '\xff' in file ./test.py on line 1, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

There is a problem. You are missing the '/' in front of usr in #!usr/bin/python. Your line should look like this.
#!/usr/bin/python

In addition to protecting the file executable, #!/usr/bin/python may not work. At least it has never worked for me on Red Hat or Ubuntu Linux. Instead, I have put this in my Python files:
#!/usr/bin/env python
I don't know how this works on Windows platforms.