We Keep Coding

Using unicode character u201c

I'm a new to python and am having problems understand unicode. I'm using
Python 3.4.
I've spent an entire day trying to figure this out by reading about unicode including http://www.fileformat.info/info/unicode/char/201C/index.htm and
http://python-notes.curiousefficiency.org/en/latest/python3/text_file_processing.html
I need to refer to special quotes because they are used in the text I'm analyzing. I did test that the W7 command window can read and write the 2 special quote characters.
To make things simple, I wrote a one line script:
print ('“') # that's the special quote mark in between normal single quotes
and get this output:
Traceback (most recent call last):
File "C:\Users\David\Documents\Python34\Scripts\wordCount3.py", line 1, in <module>
print ('\u201c')
File "C:\Python34\lib\encodings\cp437.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u201c' in position 0: character maps to <undefined>
So how do I write something to refer to these two characters u201C and u201D?
Is this the correct encoding choice in the file open statement?
with open(fileIn, mode='r', encoding='utf-8', errors='replace') as f:

The reason is that in 3.x Python You can't just mix unicode strings with byte strings. Probably, You've read the manuals dealing with Python 2.x where such things are possible as long as bytestring contains convertable chars.
print('\u201c', '\u201d')
works fine for me, so the only reason is that you're using wrong encoding for source file or terminal.
Also You may explicitly point python to codepage you're using, by throwing the next line ontop of your source:
# -*- coding: utf-8 -*-
Added: it seems that You're working on Windows machine, if so you could change Your console codepage to utf-8 by running
chcp 65001
before You fire up your python interpreter. That changes would be temporary, and if You want permanent, run the next .reg file:
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Console]
"CodePage"=dword:fde9

Trying to print check mark U+2713 in Python on Windows produces UnicodeEncodeError

I have read through Print the "approval" sign/check mark (✓) U+2713 in Python, but none of the answers work for me. I am running Python 2.7 on Windows.
print u'\u2713'
produces this error:
exceptions.UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-1: character maps to
This:
#!/usr/bin/python
# -*- coding: UTF-8 -*-
print '✓'
does not work because I am using Windows.
print u'\u2713'.encode('utf8')
Prints out ✓, which is not the right character.
print('\N{check mark}')
Is just silly. This prints \N{check mark} literally.

Read http://www.joelonsoftware.com/articles/Unicode.html and you will understand what is going on.
The bad news is: you won't be able to print that caracter because it simply is not available in the default text encoding of your Windows terminal. Modify your terminal configuration to use "utf-8" instead of the default "cp-852" or whatever is the default for Window's cmd these days, and you should be good, but do so only after reading the above article, seriously.

UnicodeEncodeError in Python on Windows Console

I'm having the following error while recursing the files in a directory and printing file names in the console:
Traceback (most recent call last):
File "C:\Program Files\Python33\lib\encodings\cp437.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position
53: character maps to <undefined>
According to the error, one of the characters in the file name string is \u2013 which is an EN DASH – character different from the commonly seen - minus character.
I have checked my Windows encoding which is set to 437. Now, I see that I have two options to workaround this by either changing the encoding of Windows console or convert the characters in get from the file names to suit the console encoding. How would I go do that in Python 3.3?

Windows console is using cp437 encoding and there is a character \u2013 that isn't supported by that encoding. Try adding this to your code:
sys.stdout = io.TextIOWrapper(sys.stdout.buffer,'cp437','backslashreplace')

or convert the characters in get from the file names to suit the console encoding
Probably the console encoding is already correct (can't tell from the error message though). Code page 437 simply doesn't include that character so you won't be able to print it.
You can reopen stdout with a text encoder that has a fallback encoding, as demonstrated in iamsudip's answer which uses backslashreplace, to at least get readable (if not reliably recoverable) output instead of an error.
changing the encoding of Windows console
You can do this by executing the console command chcp 1252 before running Python, but that will still only give you a different limited repertoire of printable characters - including U+2013, but not many other Unicode characters.
In theory you can chcp to 65001 to get UTF-8 which would allow you to print any character. Unfortunately there are serious bugs in the C runtime's standard IO implementation, which usually make this unusable in practice.
This sorry state of affairs affects all applications that use the MS C runtime's stdio library calls, including Python and most other languages, with the result that Unicode on the Windows console just doesn't work in most cases.
If you really have to get Unicode out to the Windows console you can use the Win32 WriteConsoleW API directly using ctypes, but it's not much fun.

Python 2.7 Unicode Error within a function (using __future__ print_function and unicode_literals)

I've read some threads about unicode now.
I am using Python 2.7.2 but with the future print_function (because the raw print statement is quite confusing for me..)
So here is some code:
# -*- coding: L9 -*-
from __future__ import print_function, unicode_literals
now if I print things like
print("öäüߧ€")
it works perfectly.
However, and yes I am totally new to python, if I declare a function which shall print unicode strings it blows my script
def foo():
print("öäü߀")
foo()
Traceback (most recent call last):
File "C:\Python27\test1.py", line 7, in <module>
foo()
File "C:\Python27\test1.py", line 5, in foo
print("÷õ³▀Ç")
File "C:\Python27\lib\encodings\cp850.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\x80' in position 4: character maps to <undefined>
What's the best way to handle this error and unicode in general?
And should I stick with the 2.7 print statement instead?

I suspect that print("öäü߀".encode('L9')) will solve your problems.

This may help:
print(type(s1))
s1.encode('ascii',errors='ignore') #this works
s1.decode('ascii',errors='ignore') #this does not work
The reason is that s1.decode can't decode unicode directly so an explicit call to encode is first made, but without the errors='ignore' flag thus an error is raised
Depending on whether you were issuing your commands from a file or from a python prompt with unicode support may explain why you get an error in the latter but not the former.

Console code pages use legacy "OEM" code pages for compatibility with by old DOS console programs, while the rest of Windows uses updated code pages that support modern characters, but still differ by region. In your case the console uses cp850 and GUI programs use cp1252. cp850 doesn't support the Euro character, so Python raises an exception when trying to print the character on the console. You can run chcp 1252 before running your script if you need the Euro to work. Make sure the console font supports the character, though.
BTW, L9 != cp1252 either.
Are you sure printing from the console worked with a Euro? When I cut-and-paste your print, I get the following if the code page is 850, but it works after chcp 1252.
>>> print("öäüߧ€")
öäüߧ? # Note the ?
Encoding charts:
cp850
cp1252
L9 (aka ISO-8859-15)

Can't print character '\u2019' in Python from JSON object

As a project to help me learn Python, I'm making a CMD viewer of Reddit using the json data (for example www.reddit.com/all/.json). When certain posts show up and I attempt to print them (that's what I assume is causing the error), I get this error:
Traceback (most recent call last):
File "C:\Users\nsaba\Desktop\reddit_viewer.py", line 33, in
print ( "%d. (%d) %s\n" % (i+1, obj['data']['score'], obj['data']['title']))
File "C:\Python33\lib\encodings\cp437.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2019' in position
32: character maps to
Here is where I handle the data:
request = urllib.request.urlopen(url)
content = request.read().decode('utf-8')
jstuff = json.loads(content)
The line I use to print the data as listed in the error above:
print ( "%d. (%d) %s\n" % (i+1, obj['data']['score'], obj['data']['title']))
Can anyone suggest where I might be going wrong?

It's almost certain that you problem has nothing to do with the code you've shown, and can be reproduced in one line:
print(u'\2019')
If your terminal's character set can't handle U+2019 (or if Python is confused about what character set your terminal uses), there's no way to print it out. It doesn't matter whether it comes from JSON or anywhere else.
The Windows terminal (aka "DOS prompt" or "cmd window") is usually configured for a character set like cp1252 that only knows about 256 of the 110000 characters, and there's nothing Python can do about this without a major change to the language implementation.*
See PrintFails on the Python Wiki for details, workarounds, and links to more information. There are also a few hundred dups of this problem on SO (although many of them will be specific to Python 2.x, without mentioning it).
* Windows has a whole separate set of APIs for printing UTF-16 to the terminal, so Python could detect that stdout is a Windows terminal, and if so encode to UTF-16 and use the special APIs instead of encoding to the terminal's charset and using the standard ones. But this raises a bunch of different problems (e.g., different ways of printing to stdout getting out of sync). There's been discussion about making these changes, but even if everyone were to agree and the patch were written tomorrow, it still wouldn't help you until you upgrade to whatever future version of Python it's added to…

#N-Saba, what is the string that causes the error to be thrown?
In my test case, this looks to be a version-specific bug in python 2.7.3.
In the feed I was parsing, the "title" field had the following value:
u'title': u'Intel\u2019s Sharp-Eyed Social Scientist'
I get the expected right single quote char when I call either of these, in python 2.7.6.
python -c "print {u'title': u'Intel\u2019s Sharp-Eyed Social Scientist'}['title']"
Intel’s Sharp-Eyed Social Scientist
In 2.7.3, I get the error, unless I encode the value that I pulled by KeyName.
print {u'title': u'Intel\u2019s Sharp-Eyed Social Scientist'}['title']
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 5: ordinal not in range(128)
print {u'title': u'Intel\u2019s Sharp-Eyed Social Scientist'}['title'].encode('utf-8', 'replace')
Intel’s Sharp-Eyed Social Scientist
fwiw, the #abamert command print('\u2019') prints "9". I think the intended code was print(u'\u2019').

I came across a similar error when attempting to write an API JSON output to a .cav file via pd.DataFrame.to_csv() on a Win install of Python 2.7.14.
Specifying the encoding as utf-8 fixed my process:
pd.DataFrame.to_csv(filename, encoding='utf-8')

For anyone encountering this in macOS, #abarnert's answer is correct and I was able to fix it by putting this at the top of the offending source file:-
# magic to make everything work in Unicode
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
To clarify, this is making sure the terminal output accepts Unicode correctly.

I set IDLE (Python Shell) and Window's CMD default font to Lucida Console (a utf-8 supported font) and these types of errors went away; and you no longer see boxes [][][][][][][][]
:)