We Keep Coding

UnicodeEncodeError / python3

Can't solve typical issue with encodings. Cyrrlic text is received via post and error is raised
'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)
The text key itself has a look and it must cyrillic text in russian: u'\u043f\u0440\u043e'
After that error tried this way and some others:
key = key.decode('ascii').encode('utf8')
or :
key = key.decode('ascii')
Localy it works, error is raised in production only. Python system encoding in production is utf8
EDIT: in order to clear things up. Error is raised on form handler function(again, works localy, doesn't in production)
def search(request):
if request.method == 'POST':
key = request.POST.get("key")
if key is not None:
..
So it's str received from input form, and error is first raised at this point, so I supposed it must be decoded but it didn't help.
more traceback:
UnicodeEncodeError at /search/
'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

UnicodeEncodeError with nginx and django

I've tried this SO answer, this doc is inapplicable as I'm running nginx, I've added charset utf-8; to my nginx config and I'm still getting this error.
Summarised traceback is here:
UnicodeEncodeError at /
'ascii' codec can't encode character u'\xe1' in position 69: ordinal not in range(128)
Request Method: GET
Request URL: http://django/
Django Version: 1.4.20
Exception Type: UnicodeEncodeError
Exception Value:
'ascii' codec can't encode character u'\xe1' in position 69: ordinal not in range(128)
Exception Location: /opt/envs/venv/lib/python2.7/genericpath.py in getmtime, line 54
Unicode error hint
The string that could not be encoded/decoded was: choacán.jpg

I think this error is not about nginx. It's on the file creation step.
Python uses system locale when saving files.
Check your system locale:
$ python manage.py shell
> import os
> print os.popen("locale").read()
If it's incorrect you should set system locale.
But filenames like this can cause any kind of troubles for users. Please think about defining custom file storage for models.FileField and generating random file name for every file - it's good practice.

Django 1.4 - django.db.models.FileField.save(filename, file, save=True) produces error with non-ascii filename

I'm making a fileupload feature using django.db.models.FileField of Django 1.4
When I try to upload a file whose name includes non-ascii characters, it produces error below.
'ascii' codec can't encode characters in position 109-115: ordinal not
in range(128)
The actual code is like below
file = models.FileField(_("file"),
max_length=512,
upload_to=os.path.join('uploaded', 'files', '%Y', '%m', '%d'))
file.save(filename, file, save=True) #<- This line produces the error
above, if 'filename' includes non-ascii character
If I try to use unicode(filename, 'utf-8') insteadof filename, it produces error below
TypeError: decoding Unicode is not supported
How can I upload a file whose name has non-ascii characters?
Info of my environment:
sys.getdefaultencoding() : 'ascii'
sys.getfilesystemencoding() : 'UTF-8'
using Django-1.4.10-py2.7.egg

You need to use .encode() to encode the string:
file.save(filename.encode('utf-8', 'ignore'), file, save=True)

In your FileField definition the 'upload_to' argument might be like os.path.join(u'uploaded', 'files', '%Y', '%m', '%d')
(see the first u'uploaded' started with u') so all string will be of type unicode and this may help you.

How can I solve UnicodeDecodeError in Django?

I am getting this error in Django:
UnicodeDecodeError at /category/list/
'utf8' codec can't decode byte 0xf5 in position 7: invalid start byte
Request Method: GET
Request URL: ...
Django Version: 1.3.1
Exception Type: UnicodeDecodeError
Exception Value:
'utf8' codec can't decode byte 0xf5 in position 7: invalid start byte
Exception Location: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py in iterencode, line 264
...
I should save Turkish characters in the database. How can I fix this error?

A start-byte of 0xf5 would indicate the start of a 4-character UTF-8 encoding. One strong possibility is that the input isn't UTF-8 at all but ISO-8859-9, the Turkish ISO encoding. On that codepage 0xf5 is a lowercase o with tilde or õ.

Below code solved my problem. Thank you.
if isinstance(encObject, unicode):
myStr = encObject.encode('utf-8')

http://www.fileformat.info/info/unicode/char/f5/index.htm
it is an o with a tilde
try
some_string.decode('latin1','replace')

Insert record of utf-8 character (Chinese, Arabic, Japanese.. etc) into GAE datastore programatically with python

I just want to build simple UI translation built in GAE (using python SDK).
def add_translation(self, pid=None):
trans = Translation()
trans.tlang = db.Key("agtwaW1kZXNpZ25lcnITCxILQXBwTGFuZ3VhZ2UY8aIEDA")
trans.ttype = "UI"
trans.transid = "ui-about"
trans.content = "关于我们"
trans.put()
this is resulting encoding error:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 0: ordinal not in range(128)
How to encode the correct insert content with unicode(utf-8) character?

using the u notation:
>>> s=u"关于我们"
>>> print s
关于我们
Or explicitly, stating the encoding:
>>> s=unicode('אדם מתן', 'utf8')
>>> print s
אדם מתן
Read more at the Unicode HOWTO page in the python documentation site.