We Keep Coding

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 1: invalid continuation byte in Django

This is very possibly a duplicate, since I saw a certain amount of similar questions but I can't seem to find a solution.
My problem is as stated in the description. I am working on a Django project on python 3.8.5. My professor wanted me to program a website and use PostgreSQL as db. I did use it but I always got the following error when I used python manage.py runserver.
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 1: invalid continuation byte
I uninstalled PostgreSQL and tryed to start a older project of mine that uses sqlite3, but it did not work and threw the same error. Following the stack trace.
Exception in thread django-main-thread:
Traceback (most recent call last):
File "C:\Users\Startklar\AppData\Local\Programs\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Users\Startklar\AppData\Local\Programs\Python\Python38\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\Startklar\PycharmProject\Modul133_Movie\venv\lib\site-packages\django\utils\autoreload.py", line 53, in wrapper
fn(*args, **kwargs)
File "C:\Users\Startklar\PycharmProject\Modul133_Movie\venv\lib\site-packages\django\core\management\commands\runserver.py", line 139, in inner_run
run(self.addr, int(self.port), handler,
File "C:\Users\Startklar\PycharmProject\Modul133_Movie\venv\lib\site-packages\django\core\servers\basehttp.py", line 206, in run
httpd = httpd_cls(server_address, WSGIRequestHandler, ipv6=ipv6)
File "C:\Users\Startklar\PycharmProject\Modul133_Movie\venv\lib\site-packages\django\core\servers\basehttp.py", line 67, in __init__
super().__init__(*args, **kwargs)
File "C:\Users\Startklar\AppData\Local\Programs\Python\Python38\lib\socketserver.py", line 452, in __init__
self.server_bind()
File "C:\Users\Startklar\AppData\Local\Programs\Python\Python38\lib\wsgiref\simple_server.py", line 50, in server_bind
HTTPServer.server_bind(self)
File "C:\Users\Startklar\AppData\Local\Programs\Python\Python38\lib\http\server.py", line 140, in server_bind
self.server_name = socket.getfqdn(host)
File "C:\Users\Startklar\AppData\Local\Programs\Python\Python38\lib\socket.py", line 756, in getfqdn
hostname, aliases, ipaddrs = gethostbyaddr(name)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 1: invalid continuation byte
I am running this entire thing locally on a Windows 10.
I have no clue whatsoever on how to solve this issue. I am really new to django and even my professor can't seem to help me. I have been looking through a variety of questions here in stack overflow and tried a numbers of things out but I can't seem to find a solution at all.
Every bit of help or advice is welcomed.

Your database table or columns may be using a different encoding. Try running the following query in SQL:
ALTER TABLE your_table_name CONVERT TO CHARACTER SET utf8;
This will change the character set of your table to UTF-8.
Also found an issue similar to yours:
Unicodedecodeerror with runserver
It may have to do with non-ASCII characters in the hostname or computer name.

indian rupee symbol UnicodeEncodeError while uploading file to s3 using pandas

I have scraped some data from a website for my assignment. It consists of Indian rupee character - "₹". The data when I'm trying to save into CSV file in utf-8 characters on local machine using pandas, it is saving effortlessly. The same file, I have changed the delimiters and tried to save the file to s3 using pandas, but it gave "UnicodeEncodeError" error. I'm scraping the web page using scrapy framework.
Earlier I was trying to save the file in Latin-1 i.e. "ISO-8859-1" formatting and hence changed to "utf-8" but the same error is occurring. I'm using pythn 3.7 for the development.
Below code used for saving on the local machine which is working:
result_df.to_csv(filename+str2+'.csv',index=False)
Below code is used to save the file to S3:
search_df.to_csv('s3://my-bucket/folder_path/filename_str2.csv',encoding = 'utf-8',line_terminator='^',sep='~',index=False)
Below is the error while saving the file to S3:
2019-10-29 19:24:27 [scrapy.utils.signal] ERROR: Error caught on signal handler: <function Spider.close at 0x0000019CD3B1AA60>
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\site-packages\twisted\internet\defer.py", line 151, in maybeDeferred
result = f(*args, **kw)
File "c:\programdata\anaconda3\lib\site-packages\pydispatch\robustapply.py", line 55, in robustApply
return receiver(*arguments, **named)
File "c:\programdata\anaconda3\lib\site-packages\scrapy\spiders\__init__.py", line 94, in close
return closed(reason)
File "C:\local_path\spiders\Pduct_Scrape.py", line 430, in closed
search_df.to_csv('s3://my-bucket/folder_path/filename_str2.csv',encoding = 'utf-8',line_terminator='^',sep='~',index=False)
File "c:\programdata\anaconda3\lib\site-packages\pandas\core\generic.py", line 3020, in to_csv
formatter.save()
File "c:\programdata\anaconda3\lib\site-packages\pandas\io\formats\csvs.py", line 172, in save
self._save()
File "c:\programdata\anaconda3\lib\site-packages\pandas\io\formats\csvs.py", line 288, in _save
self._save_chunk(start_i, end_i)
File "c:\programdata\anaconda3\lib\site-packages\pandas\io\formats\csvs.py", line 315, in _save_chunk
self.cols, self.writer)
File "pandas/_libs/writers.pyx", line 75, in pandas._libs.writers.write_csv_rows
File "c:\programdata\anaconda3\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u20b9' in position 2661: character maps to <undefined>
I am very new to this StackOverflow platform and please let me know if more information is to be presented.

The error gives an evidence that the code tries to encode the filename_str2.csv file in cp1252. From your stack trace:
...File "C:\local_path\spiders\Pduct_Scrape.py", line 430, in closed
search_df.to_csv('s3://my-bucket/folder_path/ filename_str2.csv ',......
File "c:\programdata\anaconda3\lib\encodings\ cp1252.py ", line 19, in encode
The reason I do not know, because you explicitely ask for an utf-8 encoding. But as the codecs page in the Python Standard Library reference says that the canonical name for utf8 is utf_8 (notice the underline instead of minus sign) and does not list utf-8 in allowed aliases, I would first try to use utf_8. If it still uses cp1252, then you will have to give the exact versions of Python and pandas that you are using.

Spyder crashes at start: UnicodeDecodeError

During a Spyder session my Linux froze. After startup, I could not start Spyder; I got the following error instead:
(trusty)dreamer#localhost:~$ spyder
Traceback (most recent call last):
File "/home/dreamer/anaconda2/bin/spyder", line 2, in <module>
from spyderlib import start_app
File "/home/dreamer/anaconda2/lib/python2.7/site-packages/spyderlib/start_app.py", line 13, in <module>
from spyderlib.config import CONF
File "/home/dreamer/anaconda2/lib/python2.7/site-packages/spyderlib/config.py", line 736, in <module>
subfolder=SUBFOLDER, backup=True, raw_mode=True)
File "/home/dreamer/anaconda2/lib/python2.7/site-packages/spyderlib/userconfig.py", line 215, in __init__
self.load_from_ini()
File "/home/dreamer/anaconda2/lib/python2.7/site-packages/spyderlib/userconfig.py", line 260, in load_from_ini
self.readfp(configfile)
File "/home/dreamer/anaconda2/lib/python2.7/ConfigParser.py", line 324, in readfp
self._read(fp, filename)
File "/home/dreamer/anaconda2/lib/python2.7/ConfigParser.py", line 479, in _read
line = fp.readline()
File "/home/dreamer/anaconda2/lib/python2.7/codecs.py", line 690, in readline
return self.reader.readline(size)
File "/home/dreamer/anaconda2/lib/python2.7/codecs.py", line 545, in readline
data = self.read(readsize, firstline=True)
File "/home/dreamer/anaconda2/lib/python2.7/codecs.py", line 492, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position 2: invalid start byte
(trusty)dreamer#localhost:~$
I have found this solution, which sounds very much like my problem, but am curious if there are others, and whether anyone knows why this occurred.

My guess is that your spyder configuration file somehow got corrupted. This is the file spyder.ini, which resides in a directory like ~/.spyder2 (the exact name of the directory depends on the version you have installed). Maybe the encoding of the configuration file changed or a Unicode byte order mark was somehow introduced.
Possible solutions: use an editor to convert the file back to UTF-8; delete the configuration file; delete the whole directory containing the configuration file. The last two obviously delete any changes you made to the configuration.

django UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 87: invalid continuation byte

I'm trying to use aptana studio 3 for a django project, but got a unicodedecodeerror no matter what i did.
After a lot of unsuccessful google searches, tried a barebones django project. Not a line of code from me, just the start code from aptana.
I expected the usual blank page from django (project working, nothing done yet), but STILL got the encode error. I didn't write a single line of code, so there's no template with latin characters or any other encoding/decoding.
Tried:
Windows->Preferences->General->Workspace->Text file encoding to UTF-8
Right click on project ->Resource->Text file encoding to UTF-8
# -- coding: utf-8 -- on first line of every file
Nothing Worked.
I'm kind of stuck here. Can anyone help?
Win7 64bit
Error message from console:
Validating models...
0 errors found
April 02, 2014 - 17:55:34
Django version 1.6, using settings 'hello.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
Traceback (most recent call last):
File "C:\Users\bruna\Documents\Aptana Studio 3 Workspace\hello\src\manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 399, in execute_from_command_line
utility.execute()
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 285, in execute
output = self.handle(*args, **options)
File "C:\Python27\lib\site-packages\django\core\management\commands\runserver.py", line 76, in handle
self.run(*args, **options)
File "C:\Python27\lib\site-packages\django\core\management\commands\runserver.py", line 87, in run
self.inner_run(*args, **options)
File "C:\Python27\lib\site-packages\django\core\management\commands\runserver.py", line 132, in inner_run
self.stderr.write("Error: %s" % error_text)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 65, in write
if ending and not msg.endswith(ending):
File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 87: invalid continuation byte

Judging by the stacktrace (the actual place where you should look for the root cause), there's something about msg (most likely, since ending is hardly 87 chars long) in if ending and not msg.endswith(ending): that isn't valid utf-8. This is probably an encoding issue in a settings file or input parameter.
I suggest to:
run the whole thing under PDB to see the value of msg or add a debug printing of repr(msg) before the offending line
track the origin of the incorrect data. It can very well be a developers' oversight (failure to check input data).

You try:
manage.py dbshell
ALTER TABLE database.table MODIFY COLUMN col VARCHAR(255) CHARACTER SET utf8 COLLATE
utf8_general_ci NOT NULL;
Change database.table and col.
Enjoy !

One reason is ,you used sed with grep in the past time like this:
sed -i "s#views_gaoji#fenye#g" grep -rl views_gaoji ./`
The above command will modify .pyc file without notification for you.
Then when you start your Django,
Django will start from .pyc,NOT from your engineering file.
Then you will get error without knowing where the error is
OR
you will get error in your manage.py file but you never modify it,
so why did it say,your manage.py has error?
---------------------------------------------------------------------
That's because you *.pyc file are changed by sed command
Solution:
find . -name "*.pyc" | xargs rm -f
and then type:
python manage.py runserver port

UnicodeDecodeError in SUDS but only in the .exe generated by py2exe

I have a Python 2.7 script SendPreord.py which communicates with web services using SUDS. In the script I call a web service method passing some parameters as strings (runJob(par1, par2, par3)). It works well with western european characters in the strings. I run it in Eclipse with PyDev.
Then I generate the .exe using py2exe. Now it gives me the error
Traceback (most recent call last):
File "SendPreord.py", line 80, in <module>
File "suds\client.pyc", line 542, in __call__
File "suds\client.pyc", line 602, in invoke
File "suds\client.pyc", line 637, in send
File "suds\transport\https.pyc", line 64, in send
File "suds\transport\http.pyc", line 77, in send
File "suds\transport\http.pyc", line 118, in u2open
File "urllib2.pyc", line 391, in open
File "urllib2.pyc", line 409, in _open
File "urllib2.pyc", line 369, in _call_chain
File "urllib2.pyc", line 1173, in http_open
File "urllib2.pyc", line 1142, in do_open
File "httplib.pyc", line 946, in request
File "httplib.pyc", line 987, in _send_request
File "httplib.pyc", line 940, in endheaders
File "httplib.pyc", line 801, in _send_output
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 484: ordinal not in range(128)
The code causing the error is:
result = ws_client.service.runJob(par1, par2, par3)
Investigating, I realized that removing characters like °èòà solves the problem... but I can't! I have to preserve the strings I'm passing.
So I tried decoding strings before passing them:
result = ws_client.service.runJob(par1.decode('latin9'), par2.decode('latin9'), par3.decode('latin9'))
Again all works in .py, but not in .exe. Maybe PyDev corrects the issue in some way?
ATTACHMENTS
Setup.py:
from distutils.core import setup
import py2exe
setup(console=['src/SendPreord.py'])
Interesting extracts from py2exe output log:
*** copy dlls ***
copying C:\Python27\lib\site-packages\py2exe\run.exe -> C:\Users\xxxxxxx\workspace\eclipse\SendPreord\dist\SendPreord.exe
The following modules appear to be missing
['ElementC14N', '_scproxy', 'ntlm']
*** binary dependencies ***
Your executable(s) also depend on these dlls which are not included,
you may or may not need to distribute them.
Make sure you have the license if you distribute any of them, and
make sure you don't distribute files belonging to the operating system.
USER32.dll - C:\Windows\system32\USER32.dll
SHELL32.dll - C:\Windows\system32\SHELL32.dll
WSOCK32.dll - C:\Windows\system32\WSOCK32.dll
ADVAPI32.dll - C:\Windows\system32\ADVAPI32.dll
WS2_32.dll - C:\Windows\system32\WS2_32.dll
KERNEL32.dll - C:\Windows\system32\KERNEL32.dll

You've been bitten by Python's guessing on encoding conversions. The first part you tried is right: decode first with the (hopefully correct) encoding. Before you send it out, you have to
encode it again, preferrably using something like UTF-8, or else Python tries the "default" encoding (which is in most installations ASCII). I've written this here before