We Keep Coding

How do I pass string parameter that has encoding problem from terminal?

I'm trying to pass a string parameter that has Korean characters. This causes an error, because Korean characters are apparently not properly encoded/decoded before it is passed to open() built-in function.
I wrote a command then executed it with os.system() which is equivalent to running it on the command prompt.
command = 'hwp5txt "C:\\Users\\username\\VSCodeProjects\\myproject\\data_files\\some_folder\\hwp\\2020-01-17_-_한국어가포함된 파일명(2020년도 제1차).hwp" > testdoc.txt'
os.system(command)
This throws an error because Korean characters are not properly decoded.
Traceback (most recent call last): File
"C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\asiae-bok-nlp-xpMr0EW7-py3.7\Scripts\hwp5txt-script.py",
line 11, in
load_entry_point('pyhwp==0.1b12', 'console_scripts', 'hwp5txt')() File
"c:\users\username\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\hwp5\hwp5txt.py",
line 102, in main
with closing(Hwp5File(hwp5path)) as hwp5file: File "c:\users\username\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\hwp5\filestructure.py",
line 537, in init
stg = Hwp5FileBase(stg) File "c:\users\username\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\hwp5\filestructure.py",
line 188, in init
stg = OleStorage(stg) File "c:\users\username\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\hwp5\storage\ole.py",
line 35, in init
self.impl = impl_class(*args, **kwargs) File "c:\users\uesrname\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\hwp5\plat\olefileio.py",
line 112, in init
if not isOleFile(olefile): File "c:\users\username\appdata\local\pypoetry\cache\virtualenvs\asiae-bok-nlp-xpmr0ew7-py3.7\lib\site-packages\olefile\olefile.py",
line 309, in isOleFile
with open(filename, 'rb') as fp: OSError: [Errno 22] Invalid argument:
'C:\Users\username\VSCodeProjects\asiae-BOK-nlp\data_files\BOK_minutes\hwp\2020-01-17_-_???????
???(2020?? ?1?).hwp'
As you can see, OS Error was raised because the command I sent to the prompt somehow didn't manage to pass the right Korean characters, which is now ????? instead of its proper name.
I tried it on the terminal manually but it also fails.
How do I pass string characters that is not properly passed to the module?
I'm using the latest version of VSCode with Git Bash terminal.
Also, I can check this information. If you need further information, please comment.
sys.stdout.encoding
>> 'UTF-8'
sys.stdin.encoding
>> 'cp1252'
sys.getfilesystemencoding
>> 'UTF-8'

Turned out, this wasn't Python's problem nor VSCode. It was just hwp5txt's issue where hwp5txt.exe won't digest Korean sys.argv. It works by trying:
$ hwp5txt-script.py 'C:\\...\\한국어가포함된파일.hwp'
However, one thing that bugs me is that this script would work on the terminal but not on Jupyter Lab or .py script.
i.e,
os.system(command) or subprocess.run(command, shell=True) won't run.
Instead, they will raise an error popup that says:
"This file does not have an app associated with it for performing this
action. Please install an app or, if one is already installed, create
an association in the Default Apps Settings page."

How to make a Python script "tab-complete" directories in terminal?

I have a Shell Script, let's say run.sh, which reads a user input from keyboard and then does some specific tasks. For some technical reasons I'm migrating this script to Python, e.g run.py, in order to achieve the exact same goal.
In the run.sh file I ask the user a input, which is typically a file in the file system, so I gave the option of "tab-completing" it and I achieved it simply through the line:
read -e -p "Choose a file: " file
The -e flag does the job of tab-completing users input. For example, if user's current directory is project, which follows the structure:
project
-- src
-- shared
-- lib
-- imgs
-- image.png
-- include
-- README.txt
and the input file is image.png they could proceed as follow:
sh<tab>i<tab><tab>
the result would be shared/imgs/image.png.
Now how do I achieve it inside a Python script? You may believe there are tons of related questions but I haven't been able to reproduce this exact same result in run.py.
What I have tried so far:
1. Python's os module:
import os
os.system("read -e -p 'Choose a file:'")
Output: sh: 1: read: Illegal option -e
2. Python's subprocess module
import subprocess
subprocess.run(['read', '-e', '-p', 'Choose a file'])
Output:
Traceback (most recent call last):
File "run.py", line 26, in <module>
subprocess.run(['read', '-e', '-p', 'Choose a file'])
File "/usr/lib/python3.7/subprocess.py", line 453, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.7/subprocess.py", line 756, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.7/subprocess.py", line 1499, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'read': 'read'
3. Python's readline module
import readline
readline.parse_and_bind("tab:complete")
file = input("Choose a file: ")
This one almost seems to work, but there is one big issue: it completes only the files in user's current directory. If user hit s<tab> then src and shared show up, but if they hit sh<tab> the liband imgs directory do not show up.
I'd like some elegant and simple way to achieve this, but I am convinced this might be a little more difficult than expected. Are there any other approaches that can solve this problem?

Set sensible completion delimiters:
import readline
readline.set_completer_delims(' \t\n=')
readline.parse_and_bind("tab: complete")
option = input("Tab complete a file: ")
By default, readline will delimit based on any of the following:
>>> import readline
>>> readline.get_completer_delims()
' \t\n`~!##$%^&*()-=+[{]}\\|;:\'",<>/?'
Since / is part of this set, anything after a / will be completed independently of anything before it. This obviously makes no sense when you're trying to complete a file path.

Get a line count of grep results for image names in a directory with Python

I have seen a lot of solutions, but I am not seeing one that works I am trying to grep every file in a directory in Python for a specific string, count the number of lines that the grep returns, and record this in python. Here's what I have tried most recently:
for f in try_files:
print("trying %s"%f)
s = subprocess.Popen("grep -r '%s' ../dir/*"%f)
print(s)
I am getting this error:
trying accept_button_off_transparent.png
Traceback (most recent call last):
File "findImages.py", line 17, in <module>
s = subprocess.Popen("grep -r %s '../dir/*'"%f)
File "/Users/agsrn/anaconda3/lib/python3.5/subprocess.py", line 950, in __init__
restore_signals, start_new_session)
File "/Users/agsrn/anaconda3/lib/python3.5/subprocess.py", line 1544, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: "grep -r accept_button_off_transparent.png '../dir/*'"
Agsrn-MacBook-Pro:images agsrn$ emacs findImages.py
Agsrn-MacBook-Pro:images agsrn$ python findImages.py
['accept_button_off_transparent.png', 'accept_button_on.png', 'accept_button_on_food.png', 'accept_button_on_transparent.png']
trying accept_button_off_transparent.png
Traceback (most recent call last):
File "findImages.py", line 17, in <module>
s = subprocess.Popen("grep -r '%s' ../dir/*"%f)
File "/Users/agsrn/anaconda3/lib/python3.5/subprocess.py", line 950, in __init__
restore_signals, start_new_session)
File "/Users/agsrn/anaconda3/lib/python3.5/subprocess.py", line 1544, in _execute_child
raise child_exception_type(errno_num, err_msg)
Ultimately I want to execute this query from within Python:
grep -r "filename" ../dir/* | wc -l
...And get that line count back as a # I can use for other logic. What's the best way to do this?
To be clear, my ultimate goal is to count how many times a particular string is mentioned by any/all files in a directory for a list of a bunch of strings. I am looking for strings inside files, not just file names. I suspect grep is a much faster solution to do this than Python, but it's inside a larger Python routine, hence the proposed hybrid solution.

If you accept another solution, here it is. Counting files can be performed easily with glob:
import glob
files = glob.glob("filename")
nfiles = len(files)
In which "filename" has the patter you want. Then, you can use nfiles for your logic.

Alternatively to my other answer, you may want to try and do it entirely in python this way:
import re # regex module
for filename in files:
n = 0
for line in open(filename, 'r'):
if re.match(r"...", line):
n += 1

Probably because of this, from the docs:
"If args is a string, the interpretation is platform-dependent [...]. On POSIX, if args is a string, the string is interpreted as the name or path of the program to execute."
The error you see says that your string is interpreted as a file name, so it fits this description. Try instead to pass args as a list:
subprocess.Popen(["grep", "-r", f, "../dir/*"], shell=True)

The following shell command will output the count that you want:
find ../dir -type f -exec cat {} + | grep -c 'filename'
The find command will print the contents of all the files in the directory, and the -c option to grep tells it to print the count of matches instead of the matching lines.
You can run this command with subprocess.Popen(). You need to use the shell=True option so it processes this as a shell command, not the name of a program to run. And to get the output of the command, you need to specify stdout=PIPE and use communicate to read from it.
pipe = subprocess.Popen("find ../dir -type f -exec cat {} + | grep -c '%s'"%f, shell=True, stdout=PIPE)
count = int(pipe.communicate()[0]);
See Store output of subprocess.Popen call in a string

subprocess.call() fails on Mac and Linux

I'm running into a weird issue with subprocess.call() function. I am trying to execute Java's 'jar' command using subprocess.call(). Here's the code:
import os
import subprocess
def read_war():
war_file_path = "jackrabbit-webapp-2.6.5.war"
java_home = os.environ['JAVA_HOME']
jar_path = os.path.join(java_home, 'bin', 'jar')
jar_cmd = jar_path + ' tvf ' + war_file_path
print "command to be executed is : " + jar_cmd
subprocess.call(jar_cmd)
read_war()
I'm using Python v2.7.3 on both Windows and Linux (Oracle Enterprise Linux).
On Windows 7, I see the contents of the war file being displayed. On Linux, however, I see a 'no such file or directory' error.:
$ python example.py
command to be executed is : /usr/local/tools/jdk1.7.0_15/bin/jar tvf jackrabbit-webapp-2.6.5.war
Traceback (most recent call last):
File "example.py", line 24, in <module>
read_war()
File "example.py", line 23, in read_war
subprocess.call(jar_cmd)
File "/usr/local/tools/Python-2.7.3/Lib/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/local/tools/Python-2.7.3/Lib/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/local/tools/Python-2.7.3/Lib/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
$
I've tried the command '/usr/local/tools/jdk1.7.0_15/bin/jar tvf jackrabbit-webapp-2.6.5.war' from command prompt and it works fine. So, nothing's wrong with the command.
I've tried various combinations of subprocess.call() - passing a string, passing a list etc. None of them worked. Any help at all would be appreciated.

Add shell=True to the call. On windows, the CreateProcess command does string parsing to separate the command and its various arguments. On linux, you only get string processing if you specifically tell subprocess to call the shell. Otherwise, it treats that entire string you handed in as the command and you don't get very far.
subprocess.call(jar_cmd, shell=True)

Use a list (sequence) argument instead of a string as the docs say:
args is required for all calls and should be a string, or a sequence
of program arguments. Providing a sequence of arguments is generally
preferred, as it allows the module to take care of any required
escaping and quoting of arguments (e.g. to permit spaces in file
names). If passing a single string, either shell must be True (see
below) or else the string must simply name the program to be executed
without specifying any arguments.
Example:
import os
import subprocess
def read_war():
war_file_path = "jackrabbit-webapp-2.6.5.war"
jar_path = os.path.join(os.environ['JAVA_HOME'], 'bin', 'jar')
jar_cmd = [jar_path, 'tvf', war_file_path]
print("command to be executed is: %s" % jar_cmd)
subprocess.check_call(jar_cmd)
read_war()
I've used check_call to raise an exception if the command returns non-zero exit status.

rar file module not working in python 2.7

i have a code like this
import rarfile
pwd = None
rar = rarfile.RarFile(source_filename)
rar.extractall(dest_dir,None,pwd) # error from here
this code in working in ubuntu.
when i run this on windows i get error like this
Traceback (most recent call last):
File "1_bete_rar.pyw", line 132, in extract
File "1_bete_rar.pyw", line 176, in unrar_file
File "rarfile.pyc", line 586, in extractall
File "rarfile.pyc", line 1112, in _extract
File "rarfile.pyc", line 1704, in custom_popen
File "subprocess.pyc", line 711, in __init__
File "subprocess.pyc", line 948, in _execute_child
WindowsError: [Error 2] The system cannot find the file specified
what is the problem with my code? how can i extract rar file with python in windows?

As the rarfile FAQ states (and as is made evident by traces of subprocess in the stack trace),
[rarfile] depends on unrar command-line utility to do the actual decompression.
Note that by default it expect it to be in PATH. If unrar launching fails, you need to fix this.
So get UnRAR from http://www.rarlab.com/rar_add.htm and put it somewhere in your PATH (such as the directory you're running your script from).

Looks like source_filename isn't pointing to a valid RAR file, do this little check before, just to be sure:
import os.path
os.path.isfile(source_filename) # what's the value returned?
If the file exists, then check if the path is in the correct format. For example, this won't work:
source_filename = 'c:\documents\file.rar'
Try this instead:
source_filename = 'c:\\documents\\file.rar'
Or even better, use raw strings:
source_filename = r'c:\documents\file.rar'

One common problem using python with Windows is that the path separator is \, but this is a special character that needs to be escaped in Python. If you print the source_filename you should be able to see if this is set correctly.
e.g.
source_filename = 'c:\users\prosserc\documents\myfile.txt'
will not work correctly. Here are a couple of alternatives:
Use a raw string:
source_filename = r'c:\users\prosserc\documents\myfile.txt'
or use os.path.join to join to an eviornment variable such as user_profile
source_filename = os.path.join(os.getenv('userprofile'), 'documents', 'myfile.txt')