UnicodeEncodeError in a Debian terminal - python

guys! I'm in a Debian 7 terminal trying to run a python script that sends a message via telegram-cli. The following line gives me an error:
check_call(["/usr/local/tg/bin/telegram-cli", "-W", "-k", "/usr/local/tg/tg-server.pub", "-e", msg])
At the end of this line, "msg" is a variable...
The error is this:
Traceback (most recent call last):
File "LogServicos.py", line 60, in <module>
msg_telegram()
File "LogServicos.py", line 17, in msg_telegram
check_call(["/usr/local/tg/bin/telegram-cli", "-W", "-k", "/usr/local/tg/tg-server.pub", "-e", msg])
File "/usr/local/lib/python3.5/subprocess.py", line 579, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/local/lib/python3.5/subprocess.py", line 560, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/local/lib/python3.5/subprocess.py", line 950, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.5/subprocess.py", line 1483, in _execute_child
restore_signals, start_new_session, preexec_fn)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe7' in position 25: ordinal not in range(128)
The script is fully fuctional on my debian 15.10 using python 3.5.1, the same as in Debian.
Any help? Thanks!

The command line arguments passed to telegram-cli contain non-ascii characters, in this case msg contains the character ç (\xe7). On *nix, file names and command line arguments are just bytes in the end, so unicode strings need to be converted to bytes to be used as such.
python uses sys.getfilesystemencoding() for such conversions, which usually depends on the LANG and LC_* environment variables.
In your case on the computer where this program fails, the default locale only supports ascii (possibly the C locale), you can use the locale command to see which one is exactly used.
To fix this error, you can either:
encode msg when passing it to the command line, but that will only work correctly if telepathy-client doesn't also rely on the current locale to interpret it, otherwise you may end up with garbage characters
make sure the correct locale is set in the environment where you're running your script. You can use locale -a to check which locales are supported on your system, if no suitable locale shows up in the list, you may need to install a suitable new locale (like described here or here)

Related

Why is Rocket League the only application that doesn't open with the python subprocess library?

this is the code that I wrote:
import subprocess
rocket_league_path = 'D:\rocketleague\Binaries\Win64\RocketLeague.exe'
subprocess.Popen(rocket_league_path)
this is the error it shows to me:
Traceback (most recent call last):
File "c:\Users\Marco\Desktop\Untitled-1.py", line 7, in <module>
subprocess.Popen(rocket_league_path)
File "C:\Users\Marco\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 966, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\Marco\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1435, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] Cannot find the specified file
I have tried with other applications and it works with all except Rocket League
It's a quirk of the way string literals work in Python. The \ character is special, it indicates that the following characters may be interpreted as a special character. Or it may not, depending on what follows. In your case the two characters \r are being converted to a single character Carriage Return. The path is now incomplete and the file can't be found. If it had been \c instead it would have worked, because there's no special character corresponding to \c.
You can easily fix it by using a raw character string instead, where the \ loses its special meaning.
rocket_league_path = r'D:\rocketleague\Binaries\Win64\RocketLeague.exe'
# ^

How do I pass string parameter that has encoding problem from terminal?

I'm trying to pass a string parameter that has Korean characters. This causes an error, because Korean characters are apparently not properly encoded/decoded before it is passed to open() built-in function.
I wrote a command then executed it with os.system() which is equivalent to running it on the command prompt.
command = 'hwp5txt "C:\\Users\\username\\VSCodeProjects\\myproject\\data_files\\some_folder\\hwp\\2020-01-17_-_한국어가포함된 파일명(2020년도 제1차).hwp" > testdoc.txt'
os.system(command)
This throws an error because Korean characters are not properly decoded.
Traceback (most recent call last): File
"C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\asiae-bok-nlp-xpMr0EW7-py3.7\Scripts\hwp5txt-script.py",
line 11, in
load_entry_point('pyhwp==0.1b12', 'console_scripts', 'hwp5txt')() File
"c:\users\username\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\hwp5\hwp5txt.py",
line 102, in main
with closing(Hwp5File(hwp5path)) as hwp5file: File "c:\users\username\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\hwp5\filestructure.py",
line 537, in init
stg = Hwp5FileBase(stg) File "c:\users\username\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\hwp5\filestructure.py",
line 188, in init
stg = OleStorage(stg) File "c:\users\username\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\hwp5\storage\ole.py",
line 35, in init
self.impl = impl_class(*args, **kwargs) File "c:\users\uesrname\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\hwp5\plat\olefileio.py",
line 112, in init
if not isOleFile(olefile): File "c:\users\username\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\olefile\olefile.py",
line 309, in isOleFile
with open(filename, 'rb') as fp: OSError: [Errno 22] Invalid argument:
'C:\Users\username\VSCodeProjects\asiae-BOK-nlp\data_files\BOK_minutes\hwp\2020-01-17_-_???????
???(2020?? ?1?).hwp'
As you can see, OS Error was raised because the command I sent to the prompt somehow didn't manage to pass the right Korean characters, which is now ????? instead of its proper name.
I tried it on the terminal manually but it also fails.
How do I pass string characters that is not properly passed to the module?
I'm using the latest version of VSCode with Git Bash terminal.
Also, I can check this information. If you need further information, please comment.
sys.stdout.encoding
>> 'UTF-8'
sys.stdin.encoding
>> 'cp1252'
sys.getfilesystemencoding
>> 'UTF-8'
Turned out, this wasn't Python's problem nor VSCode. It was just hwp5txt's issue where hwp5txt.exe won't digest Korean sys.argv. It works by trying:
$ hwp5txt-script.py 'C:\\...\\한국어가포함된파일.hwp'
However, one thing that bugs me is that this script would work on the terminal but not on Jupyter Lab or .py script.
i.e,
os.system(command) or subprocess.run(command, shell=True) won't run.
Instead, they will raise an error popup that says:
"This file does not have an app associated with it for performing this
action. Please install an app or, if one is already installed, create
an association in the Default Apps Settings page."

Wing IDE's PEP8 checking error

I installed PEP8 plugin for Wing IDE and now I get error when validate any .PY:
Error::Error executing script .user.internal_script_C0x3a0x5cUsers0x5chome0x5cAppData0x5cRoaming0x5cWing0x20IDE0x2050x5cscripts_pep8panel.pep8_execute
Runtime failure details:
Exception: "<type 'exceptions.UnicodeDecodeError'>"
Value = "'utf8' codec can't decode byte 0xc4 in position 21: invalid continuation byte"
Traceback:
File "C:\src\ide\bin\2.7\src\command\commandmgr.pyo", line 823, in Execute
File "C:\Users\home\AppData\Roaming\Wing IDE 5\scripts\pep8panel.py", line 114, in pep8_execute
_pep8_execute(filenames)
File "C:\Users\home\AppData\Roaming\Wing IDE 5\scripts\pep8panel.py", line 282, in _pep8_execute
handler = app.AsyncExecuteCommandLineE(cmd, rundir, env, *args)
File "C:\Program Files (x86)\Wing IDE 5.0\bin\wingapi.py", line 832, in AsyncExecuteCommandLineE
handler = spawn.CAsyncExecute(cmd, env, dirname, 10000.0, 100000, *args)
File "C:\src\svn-mirror-ide\bin\2.7\src\wingutils\spawn.pyo", line 1233, in __init__
File "C:\src\svn-mirror-ide\bin\2.7\src\wingutils\spawn.pyo", line 1142, in CreatePopenE
File "C:\src\svn-mirror-ide\bin\2.7\src\wingutils\spawn.pyo", line 986, in __init__
File "C:\src\svn-mirror-ide\bin\2.7\src\wingutils\spawn.pyo", line 200, in _win32_cmd_line_from_argv
What's wrong?
UPD: I use Russian Windows 7 and russian letters can appear only in path to Python files.
Looks like you have written your script in some legacy encoding. Try removing any non-ASCII characters (strings in some language other than English, Hawai'ian, or Latin are often the culprits. 0xC4 in Latin-1 is Ä).
If that solves your problem, either encode any non-ASCII strings as hex sequences ('g\xf6\xf6dbye w\xf6rld') or re-save as UTF-8. (Then maybe add # encoding: utf-8 near the top of the file to keep Python happy.)

subprocess.call() fails on Mac and Linux

I'm running into a weird issue with subprocess.call() function. I am trying to execute Java's 'jar' command using subprocess.call(). Here's the code:
import os
import subprocess
def read_war():
war_file_path = "jackrabbit-webapp-2.6.5.war"
java_home = os.environ['JAVA_HOME']
jar_path = os.path.join(java_home, 'bin', 'jar')
jar_cmd = jar_path + ' tvf ' + war_file_path
print "command to be executed is : " + jar_cmd
subprocess.call(jar_cmd)
read_war()
I'm using Python v2.7.3 on both Windows and Linux (Oracle Enterprise Linux).
On Windows 7, I see the contents of the war file being displayed. On Linux, however, I see a 'no such file or directory' error.:
$ python example.py
command to be executed is : /usr/local/tools/jdk1.7.0_15/bin/jar tvf jackrabbit-webapp-2.6.5.war
Traceback (most recent call last):
File "example.py", line 24, in <module>
read_war()
File "example.py", line 23, in read_war
subprocess.call(jar_cmd)
File "/usr/local/tools/Python-2.7.3/Lib/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/local/tools/Python-2.7.3/Lib/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/local/tools/Python-2.7.3/Lib/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
$
I've tried the command '/usr/local/tools/jdk1.7.0_15/bin/jar tvf jackrabbit-webapp-2.6.5.war' from command prompt and it works fine. So, nothing's wrong with the command.
I've tried various combinations of subprocess.call() - passing a string, passing a list etc. None of them worked. Any help at all would be appreciated.
Add shell=True to the call. On windows, the CreateProcess command does string parsing to separate the command and its various arguments. On linux, you only get string processing if you specifically tell subprocess to call the shell. Otherwise, it treats that entire string you handed in as the command and you don't get very far.
subprocess.call(jar_cmd, shell=True)
Use a list (sequence) argument instead of a string as the docs say:
args is required for all calls and should be a string, or a sequence
of program arguments. Providing a sequence of arguments is generally
preferred, as it allows the module to take care of any required
escaping and quoting of arguments (e.g. to permit spaces in file
names). If passing a single string, either shell must be True (see
below) or else the string must simply name the program to be executed
without specifying any arguments.
Example:
import os
import subprocess
def read_war():
war_file_path = "jackrabbit-webapp-2.6.5.war"
jar_path = os.path.join(os.environ['JAVA_HOME'], 'bin', 'jar')
jar_cmd = [jar_path, 'tvf', war_file_path]
print("command to be executed is: %s" % jar_cmd)
subprocess.check_call(jar_cmd)
read_war()
I've used check_call to raise an exception if the command returns non-zero exit status.

django UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 87: invalid continuation byte

I'm trying to use aptana studio 3 for a django project, but got a unicodedecodeerror no matter what i did.
After a lot of unsuccessful google searches, tried a barebones django project. Not a line of code from me, just the start code from aptana.
I expected the usual blank page from django (project working, nothing done yet), but STILL got the encode error. I didn't write a single line of code, so there's no template with latin characters or any other encoding/decoding.
Tried:
Windows->Preferences->General->Workspace->Text file encoding to UTF-8
Right click on project ->Resource->Text file encoding to UTF-8
# -- coding: utf-8 -- on first line of every file
Nothing Worked.
I'm kind of stuck here. Can anyone help?
Win7 64bit
Error message from console:
Validating models...
0 errors found
April 02, 2014 - 17:55:34
Django version 1.6, using settings 'hello.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
Traceback (most recent call last):
File "C:\Users\bruna\Documents\Aptana Studio 3 Workspace\hello\src\manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 399, in execute_from_command_line
utility.execute()
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 285, in execute
output = self.handle(*args, **options)
File "C:\Python27\lib\site-packages\django\core\management\commands\runserver.py", line 76, in handle
self.run(*args, **options)
File "C:\Python27\lib\site-packages\django\core\management\commands\runserver.py", line 87, in run
self.inner_run(*args, **options)
File "C:\Python27\lib\site-packages\django\core\management\commands\runserver.py", line 132, in inner_run
self.stderr.write("Error: %s" % error_text)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 65, in write
if ending and not msg.endswith(ending):
File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 87: invalid continuation byte
Judging by the stacktrace (the actual place where you should look for the root cause), there's something about msg (most likely, since ending is hardly 87 chars long) in if ending and not msg.endswith(ending): that isn't valid utf-8. This is probably an encoding issue in a settings file or input parameter.
I suggest to:
run the whole thing under PDB to see the value of msg or add a debug printing of repr(msg) before the offending line
track the origin of the incorrect data. It can very well be a developers' oversight (failure to check input data).
You try:
manage.py dbshell
ALTER TABLE database.table MODIFY COLUMN col VARCHAR(255) CHARACTER SET utf8 COLLATE
utf8_general_ci NOT NULL;
Change database.table and col.
Enjoy !
One reason is ,you used sed with grep in the past time like this:
sed -i "s#views_gaoji#fenye#g" grep -rl views_gaoji ./`
The above command will modify .pyc file without notification for you.
Then when you start your Django,
Django will start from .pyc,NOT from your engineering file.
Then you will get error without knowing where the error is
OR
you will get error in your manage.py file but you never modify it,
so why did it say,your manage.py has error?
---------------------------------------------------------------------
That's because you *.pyc file are changed by sed command
Solution:
find . -name "*.pyc" | xargs rm -f
and then type:
python manage.py runserver port

Categories