I'm using python3 to read through a string and extracting certain elements into a list, using the following on the top of my script:
# -*- coding: utf-8 -*-
import ast
import re
Works with all except for one character: which in unicode is: U+C0545 and on the command line looks like:
I would just like to skip this character, but the script can't recognize it. Is there any way skip this character?
I don't know whether this should be taken as an authoritative source, but http://www.fileformat.info/info/unicode/char/c0545/index.htm indicates that this is not a valid unicode character. Some systems may choose to represent it using some placeholder glyph, others may raise an error or behave in other strange ways.
In your python code, your best bet may be to handle the exception and do what is appropriate in the context.
Without seeing the actual source where the exception happens and the actual exception text, it's hard to guess what's really wrong.
Related
I have Python behaving very strange.
Updated question to get rid of all unrelated code:
My new file has only this content:
renamingDict = {'ID':'lead_id','Статус':'status','Название лида':'title','Имя':'name','Дата создания':'created_at','Источник':'source','Рабочий телефон':'phones','Ответственный':'manager','Дата изменения':'changed_at','Комментарий':'lead_comment','Дата экскурсии, ожидаемая':'exc_date','Ссылка':'link','ID брони/заявки':'res_id','ID экскурсии':'lead_exc_id','ID организатора':'lead_org_id'}
it gives error:
SyntaxError: Non-UTF-8 code starting with '\xd0' in file .\lead_analysis.py on line 10, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
but it WON'T give error if I add new element to dictionary's source. After 'lead_comment',
I will add 'UTM Source':'',
and I would have:
renamingDict = {'ID':'lead_id','Статус':'status','Название лида':'title','Имя':'name','Дата создания':'created_at','Источник':'source','Рабочий телефон':'phones','Ответственный':'manager','Дата изменения':'changed_at','Комментарий':'lead_comment','UTM Source':'','Дата экскурсии, ожидаемая':'exc_date','Ссылка':'link','ID брони/заявки':'res_id','ID экскурсии':'lead_exc_id','ID организатора':'lead_org_id'}
Which works fine!
I can fix all this by adding
#!/usr/bin/python
# -*- coding: UTF-8 -*-
at the start of my code, but the most weird part is that I encounter the error after REMOVING independent part of working code.
How can python think that dictionary of 17 elements is fine, but give 'encoding' error when I remove one element from the dictionary, without even proceeding further. It understood my code with extra symbols, how can it not understand it now?
The issue seems so absurd that I really want someone to help me understand how this works, even though I have workaround with
#!/usr/bin/python
# -*- coding: UTF-8 -*-
More so, my "buggy" line with dictionary works without errors if I just make a new line in my dictionary listing
So you found the solution yourself: Specifying the encoding
My theory for why would be:
UTF-8 uses 1 byte for some characters and multiple for others. So adding a few non-ascii characters that don't produce an error by themselves can bring the parser out of sync with the character count and thus cause an error later.
If you want to test it, put your code back in and randomly add a few russian characters to see if it starts working again all of the sudden.
Is there any way to define a string with accented letters in python?
An extreme example is this one:
message = "ÂÃÄÀÁÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ"
Error:
SyntaxError: Non-UTF-8 code starting with '\xc2'
When souce code contains something else than ASCII, you have to add a line to tell the python interpreter:
#!/usr/bin/env python
# encoding: utf-8
Read more in PEP-0263 for the exact rules how to include the encoding hint in a magic comment.
If you use Python 3.x you can use accented (Unicode) strings without doing anything special. If you are using Python 2.x, use u prefix to denote Unicode:
message = u"ÂÃÄÀÁÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ"
Also remember to include the following line at the top of your script:
# coding=utf-8
PEP-0263 explains this in detail:
To define a source code encoding, a magic comment must
be placed into the source files either as first or second
line in the file, such as:
# coding=<encoding name>
PEP 263 defines how to declare Python source code encoding.
Normally, the first 2 lines of a Python file should start with:
#!/usr/bin/python
# -*- coding: <encoding name> -*-
But I have seen a lot of files starting with:
#!/usr/bin/python
# -*- encoding: <encoding name> -*-
=> encoding instead of coding.
So what is the correct way of declaring the file encoding?
Is encoding permitted because the regex used is lazy? Or is it just another form of declaring the file encoding?
I'm asking this question because the PEP does not talk about encoding, it just talks about coding.
Check the docs here:
"If a comment in the first or second line of the Python script matches the regular expression coding[=:]\s*([-\w.]+), this comment is processed as an encoding declaration"
"The recommended forms of this expression are
# -*- coding: <encoding-name> -*-
which is recognized also by GNU Emacs, and
# vim:fileencoding=<encoding-name>
which is recognized by Bram Moolenaar’s VIM."
So, you can put pretty much anything before the "coding" part, but stick to "coding" (with no prefix) if you want to be 100% python-docs-recommendation-compatible.
More specifically, you need to use whatever is recognized by Python and the specific editing software you use (if it needs/accepts anything at all). E.g. the coding form is recognized (out of the box) by GNU Emacs but not Vim (yes, without a universal agreement, it's essentially a turf war).
Just copy paste below statement on the top of your program.It will solve character encoding problems
#!/usr/bin/env python
# -*- coding: utf-8 -*-
PEP 263:
the first or second line must match
the regular
expression "coding[:=]\s*([-\w.]+)"
So, "encoding: UTF-8" matches.
PEP provides some examples:
#!/usr/bin/python
# vim: set fileencoding=<encoding name> :
# This Python file uses the following encoding: utf-8
import os, sys
As of today — June 2018
PEP 263 itself mentions the regex it follows:
To define a source code encoding, a magic comment must be placed into
the source files either as first or second line in the file, such as:
# coding=<encoding name>
or (using formats recognized by popular editors):
#!/usr/bin/python
# -*- coding: <encoding name> -*-
or:
#!/usr/bin/python
# vim: set fileencoding=<encoding name> :
More precisely, the first or second line must match the following regular expression:
^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)
So, as already summed up by other answers, it'll match coding with any prefix, but if you'd like to be as PEP-compliant as it gets (even though, as far as I can tell, using encoding instead of coding does not violate PEP 263 in any way) — stick with 'plain' coding, with no prefixes.
If I'm not mistaken, the original proposal for source file encodings was to use a regular expression for the first couple of lines, which would allow both.
I think the regex was something along the lines of coding: followed by something.
I found this: http://www.python.org/dev/peps/pep-0263/
Which is the original proposal, but I can't seem to find the final spec stating exactly what they did.
I've certainly used encoding: to great effect, so obviously that works.
Try changing to something completely different, like duhcoding: ... to see if that works just as well.
I suspect it is similar to Ruby - either method is okay.
This is largely because different text editors use different methods (ie, these two) of marking encoding.
With Ruby, as long as the first, or second if there is a shebang line contains a string that matches:
coding: encoding-name
and ignoring any whitespace and other fluff on those lines. (It can often be a = instead of :, too).
Say I have a function:
def NewFunction():
return '£'
I want to print some stuff with a pound sign in front of it and it prints an error when I try to run this program, this error message is displayed:
SyntaxError: Non-ASCII character '\xa3' in file 'blah' but no encoding declared;
see http://www.python.org/peps/pep-0263.html for details
Can anyone inform me how I can include a pound sign in my return function? I'm basically using it in a class and it's within the '__str__' part that the pound sign is included.
I'd recommend reading that PEP the error gives you. The problem is that your code is trying to use the ASCII encoding, but the pound symbol is not an ASCII character. Try using UTF-8 encoding. You can start by putting # -*- coding: utf-8 -*- at the top of your .py file. To get more advanced, you can also define encodings on a string by string basis in your code. However, if you are trying to put the pound sign literal in to your code, you'll need an encoding that supports it for the entire file.
Adding the following two lines at the top of my .py script worked for me (first line was necessary):
#!/usr/bin/env python
# -*- coding: utf-8 -*-
First add the # -*- coding: utf-8 -*- line to the beginning of the file and then use u'foo' for all your non-ASCII unicode data:
def NewFunction():
return u'£'
or use the magic available since Python 2.6 to make it automatic:
from __future__ import unicode_literals
The error message tells you exactly what's wrong. The Python interpreter needs to know the encoding of the non-ASCII character.
If you want to return U+00A3 then you can say
return u'\u00a3'
which represents this character in pure ASCII by way of a Unicode escape sequence. If you want to return a byte string containing the literal byte 0xA3, that's
return b'\xa3'
(where in Python 2 the b is implicit; but explicit is better than implicit).
The linked PEP in the error message instructs you exactly how to tell Python "this file is not pure ASCII; here's the encoding I'm using". If the encoding is UTF-8, that would be
# coding=utf-8
or the Emacs-compatible
# -*- encoding: utf-8 -*-
If you don't know which encoding your editor uses to save this file, examine it with something like a hex editor and some googling. The Stack Overflow character-encoding tag has a tag info page with more information and some troubleshooting tips.
In so many words, outside of the 7-bit ASCII range (0x00-0x7F), Python can't and mustn't guess what string a sequence of bytes represents. https://tripleee.github.io/8bit#a3 shows 21 possible interpretations for the byte 0xA3 and that's only from the legacy 8-bit encodings; but it could also very well be the first byte of a multi-byte encoding. But in fact, I would guess you are actually using Latin-1, so you should have
# coding: latin-1
as the first or second line of your source file. Anyway, without knowledge of which character the byte is supposed to represent, a human would not be able to guess this, either.
A caveat: coding: latin-1 will definitely remove the error message (because there are no byte sequences which are not technically permitted in this encoding), but might produce completely the wrong result when the code is interpreted if the actual encoding is something else. You really have to know the encoding of the file with complete certainty when you declare the encoding.
Adding the following two lines in the script solved the issue for me.
# !/usr/bin/python
# coding=utf-8
Hope it helps !
You're probably trying to run Python 3 file with Python 2 interpreter. Currently (as of 2019), python command defaults to Python 2 when both versions are installed, on Windows and most Linux distributions.
But in case you're indeed working on a Python 2 script, a not yet mentioned on this page solution is to resave the file in UTF-8+BOM encoding, that will add three special bytes to the start of the file, they will explicitly inform the Python interpreter (and your text editor) about the file encoding.
I have a python file that contains a long string of HTML. When I compile & run this file/script I get this error:
_SyntaxError: Non-ASCII character '\x92' in file C:\Users...\GlobalVars.py on line 2509, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details_
I have followed the instructions and gone to the url suggested. But putting something like this at the top of my script still doesn't work:
#!/usr/bin/python
# -*- coding: latin-1 -*-
What do you think I can do to stop this compiler error from occuring?
First, in order to prevent problems like the one specified in the question you should not ever use other encoding than utf-8 for python source code.
This is the correct header to use
#! /usr/bin/env python
# -*- coding: utf-8 -*-
Now you have to convert the file from whatever encoding you may have to utf-8, probably your current text editor is able to do that.
If you wonder why I say this remember that it is impossible for a text editor to safely guess your non-unicode encoding because there is no BOM for non-unicode. For this reason most decent editors are using UTF-8 as default even when encoding is not specified. And BTW, the encoding specified in the python file header is for Python only, most editors ignore what you wrote there.
Also, as you can see Python is trying to decode a character above 128 using ASCII (not latin-1), this is supposed to fail. I am not sure why this happens but I don't even care too much because there is a much better way to solve the problem.
It must be at the top of the script that has the non-ASCII text, and it must match the actual encoding of the file. \x92 is CP1252, not Latin-1.
If you are just concerned about getting rid of this error without getting into the details of it(which you can get from the other answers on this page), you can do the following -
1) Copy your code and paste it in Notepad++
2) Select Encoding -> Encode in UTF-8
3) Select View -> Show Symbol -> Show All Characters
Now it would be visible to you that which symbol is causing the issue(x92 would be visible). Replace/Remove it to solve the problem.
Found this and hope it's helpful to the next person:
http://www.sitepoint.com/forums/showthread.php?567734-Anyone-know-what-this-error-means
Code point 0x92 (146 decimal) is the right single quotation mark, or
apostrophe (’) in Windows-1252. It's an invalid character in ISO 8859
and in UTF-8, since the 0x80-0x9F range is reserved for C1 control
characters.
Not sure if I'm busting copyright. If so please remove the blockquote.
The encoding declaration indicates that you think the file is in latin-1 encoding, but the python interpreter is finding that a char at or very near line 2509 in GlobalVars.py that is not what you think it is.
You should first confirm the encoding of GlobalVars.py. Is it really latin-1?
Next, you should check the characters near line 2509. Are they also latin-1, or were they cut and pasted from a web page or somewhere else (maybe there are UTF-8 chars mixed up in there)?
If you have chars in your source file that aren't what you think they are, then you may need to clean up the file before going any further.
add these lines on top of your code
#! /usr/bin/env python
# -*- coding: utf-8 -*-
An easy workaround solution if your file is really in latin-1 is to change the html string with its representation.
Afaik:
\x92 => 146 in decimal => Æ => Æ
If your character is not Æ, then your file is not encoded into latin-1 ;-) (and you might wanna check if utf-8/cp1292 works better as a quick win)
EDIT:
Of course, you want to check your ACTUAL file encoding before trying. I might be wrong, not 100% sure \x92 is Æ in Iso8859-1 : according to this page, it doesn't seem defined.