We Keep Coding

subprocess.run simple scenario fails

I am trying to run python subprocess.run function to execute following command:
pdftoppm -jpeg -f 1 -scale-to 200 data/andromeda.pdf and-page
pdftoppm - is part of poppler utility and it generates images from pdf files.
File data/andromeda.pdf exists. Folder data is on same level with python script and/or where I run command from.
Command basically will generate a jpeg file, from page 1 (-f 1) 200px wide (-scale-to) from given file of and-page-1.jpeg format (so called ppmtroot).
Long story short: from command line it works as expected i.e. if I call the above command either from zsh or bash shell, manually - it generates thumbnail as expected. However if I run it from python subprocess module - it fails it returns 99 error code!
Following is python code (file name is sc_02_thumbnails.py):
import subprocess
import sys
def main(filename, ppmroot):
cmd = [
'pdftoppm',
'-f 1',
'-scale-to 200',
'-jpeg',
filename,
ppmroot
]
result = subprocess.run(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
if result.returncode:
print("Failed to generate thumbnail. Return code: {}. stderr: {}".format(
result.returncode,
result.stderr
))
print("Used cmd: {}".format(' '.join(cmd)))
sys.exit(1)
else:
print("Success!")
if __name__ == "__main__":
if len(sys.argv) > 2:
filename = sys.argv[1]
ppmroot = sys.argv[2]
else:
print("Usage: {} <pdffile> <ppmroot>".format(sys.argv[0]))
sys.exit(1)
main(filename, ppmroot)
And here is repo which includes data/andromeda.pdf file as well.
I call my script with as (from zsh):
$ chmod +x ./sc_02_thumbnauils.py
$ ./sc_02_thumbnails.py data/andromeda.pdf and-page
and ... thumbnail generating fails!
I have tried executing python script from both, from zsh and bash shells :(
What I am doing wrong?

The quoting is wrong, you should have '-f', '1', etc

How can I use an executable jar file with a mainClass in python? [duplicate]

I have been looking for an answer for how to execute a java jar file through python and after looking at:
Execute .jar from Python
How can I get my python (version 2.5) script to run a jar file inside a folder instead of from command line?
How to run Python egg files directly without installing them?
I tried to do the following (both my jar and python file are in the same directory):
import os
if __name__ == "__main__":
os.system("java -jar Blender.jar")
and
import subprocess
subprocess.call(['(path)Blender.jar'])
Neither have worked. So, I was thinking that I should use Jython instead, but I think there must a be an easier way to execute jar files through python.
Do you have any idea what I may do wrong? Or, is there any other site that I study more about my problem?

I would use subprocess this way:
import subprocess
subprocess.call(['java', '-jar', 'Blender.jar'])
But, if you have a properly configured /proc/sys/fs/binfmt_misc/jar you should be able to run the jar directly, as you wrote.
So, which is exactly the error you are getting?
Please post somewhere all the output you are getting from the failed execution.

This always works for me:
from subprocess import *
def jarWrapper(*args):
process = Popen(['java', '-jar']+list(args), stdout=PIPE, stderr=PIPE)
ret = []
while process.poll() is None:
line = process.stdout.readline()
if line != '' and line.endswith('\n'):
ret.append(line[:-1])
stdout, stderr = process.communicate()
ret += stdout.split('\n')
if stderr != '':
ret += stderr.split('\n')
ret.remove('')
return ret
args = ['myJarFile.jar', 'arg1', 'arg2', 'argN'] # Any number of args to be passed to the jar file
result = jarWrapper(*args)
print result

I used the following way to execute tika jar to extract the content of a word document. It worked and I got the output also. The command I'm trying to run is "java -jar tika-app-1.24.1.jar -t 42250_EN_Upload.docx"
from subprocess import PIPE, Popen
process = Popen(['java', '-jar', 'tika-app-1.24.1.jar', '-t', '42250_EN_Upload.docx'], stdout=PIPE, stderr=PIPE)
result = process.communicate()
print(result[0].decode('utf-8'))
Here I got result as tuple, hence "result[0]". Also the string was in binary format (b-string). To convert it into normal string we need to decode with 'utf-8'.

With args: concrete example using Closure Compiler (https://developers.google.com/closure/) from python
import os
import re
src = test.js
os.execlp("java", 'blablabla', "-jar", './closure_compiler.jar', '--js', src, '--js_output_file', '{}'.format(re.sub('.js$', '.comp.js', src)))
(also see here When using os.execlp, why `python` needs `python` as argv[0])

How about using os.system() like:
os.system('java -jar blabla...')
os.system(command)
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.

Quote spaces in filename when calling subprocess

Somebody came up with the brilliant idea of putting spaces in a filename. I need to do scp from python using that filename, which is problematic because the shell parses the command, and scp has also some quircks regarding spaces. This is my test code:
import subprocess
import shlex
def split_it(command):
return shlex.split(command)
#return command.split(" ")
def upload_file(localfile, host, mypath):
command = split_it('scp {} {}:"{}"'.format(localfile, host, mypath))
print(command)
res = subprocess.run(command, stdout=subprocess.PIPE)
return res.stdout.decode()
upload_file("localfile.txt", "hostname", "/some/directory/a file with spaces.txt")
Which gives:
['scp', 'localfile.txt', 'hostname:/some/directory/a file with spaces.txt']
scp: ambiguous target
Using the naive version with command.split(" "):
['scp', 'localfile.txt', 'hostname:"/some/directory/a', 'file', 'with', 'spaces.txt"']
spaces.txt": No such file or directory
The right, working scp command would be:
['scp', 'localfile.txt', 'hostname:"/some/directory/a file with spaces.txt"']
Is there a ready solution for this?
If not, what would be the robust way of doing:
split_it('scp localfile.txt hostname:"/some/directory/a file with spaces.txt"')
# returns ['scp', 'localfile.txt', 'hostname:"/some/directory/a file with spaces.txt"']

command = split_it('scp {} {}:"{}"'.format(localfile, host, mypath))
Instead of building a command string, only to split_it again, directly build a list of arguments.
In order to add one layer of quoting to the remote file path, use shlex.quote (or pipes.quote if using older Python versions).
command = ['scp', localfile, '{}:{}'.format(host, shlex.quote(mypath))]
Sources/related posts:
How to escape spaces in path during scp copy in linux?
Python scp copy file with spaces in filename
https://docs.python.org/3/library/subprocess.html#popen-constructor

Path not being found in Windows via bash/python script

This is .csv file containing target,path:
[root#server ~]# head /usr/local/CustomAppResults/App/targets.txt
server1.com,C:\\Program Files\\App\\
I loop through the .csv, line by line in a Bash script ..
IFS=","
while read target path
do
# Windows target
result=$(python /root/apache-tomcat-7.0.37/webapps/App/tools/remoteStafCommand.py "$target" "dir $path /B")
resultCode=$?
echo "$result"
done
The bash script passes the contents of the csv. file into a Python script, it's contents are below.
The Python script runs a staf subprocess and parses the results, which is returned and stored inside result of the Bash script.
[root#server ~]# cat App/tools/remoteStafCommand.py
import subprocess,re,sys
from subprocess import CalledProcessError, check_output
print "1st arg: %s" % (sys.argv[1])
print "2nd arg: %s" % (sys.argv[2])
try:
output = subprocess.check_output(["staf", "%s" % (sys.argv[1]) , "PROCESS", "START", "SHELL", "COMMAND", "%s" % (sys.argv[2]), "WAIT", "RETURNSTDOUT", "STDERRTOSTDOUT"])
result = re.findall(r'Data\s*:\s*((?:[^\n]*(?:[\r\n]+(?!\s*}))?)+)', output, re.DOTALL)[0]
print result
except CalledProcessError as e:
print(e.returncode)
sys.exit(e.returncode)
This is the output I get, why can it not find the path? It exists.
1st arg: server1.com
2nd arg: dir C:\Program Files\App\ /B
The system cannot find the path specified.

The read command tokenizes on whitespace, not comma. You can override this by changing the value of IFS. Or if the text file isn't used for anything else, change its format to have a space instead of a comma between the fields. Or better still, have your Python script read and parse the file directly -- the simple Bash script begs to be factored away.

Wrapping path with double quotes directly in .csv file for escaping ..
[root#server ~]# head /usr/local/CustomAppResults/App/targets.txt
server1.com,"C:\\Program Files\\App\\"

python copy file in local network (linux -> linux) and output

I'm trying to write a script to copy files in my RaspberryPi, from my Desktop PC.
Here is my code: (a part)
print "start the copy"
path_pi = '//192.168.2.2:22/home/pi/Stock/'
file_pc = path_file + "/" + file
print "the file to copy is: ", file_pc
shutil.copy2(file_pc, path_pi + file_pi)
Actually I have this error: (in french)
IOError: [Errno 2] Aucun fichier ou dossier de ce type: '//192.168.2.2:22/home/pi/Stock/exemple.txt'
So, how could I proceed? Must I connect the 2 machines before trying to copy?
I have tryed with:
path_pi = r'//192.168.2.2:22/home/pi/Stock'
But the problem is the same. (And file_pc is a variable)
Thanks
Edit:
Ok, I found this:
command = 'scp', file_pc, file_pi
p = subprocess.Popen(command, stdout=subprocess.PIPE)
But no way to have the output... (work with Shell=False)

shutil.copy2() works with local files. 192.168.2.2:22 suggests that you want to copy files over ssh. You could mount the remote directory (RaspberryPi) onto a local directory on your desktop machine (sshfs) so that shutil.copy2() would work.
If you want to see the output of a command then don't set stdout=PIPE (note: if you set stdout=PIPE then you should read from p.stdout otherwise the process may block forever):
from subprocess import check_call
check_call(['scp', file_pc, file_pi])
scp will print to whatever places your parent Python script prints.
To get the output as a string:
from subprocess import check_output
output = check_output(['scp', file_pc, file_pi])
Though It looks like scp doesn't print anything by default if the output is redirected.
You could use pexpect to make scp think that it runs in a terminal:
import pipes
import re
import pexpect # $ pip install pexpect
def progress(locals):
# extract percents
print(int(re.search(br'(\d+)%[^%]*$', locals['child'].after).group(1)))
command = "scp %s %s" % tuple(map(pipes.quote, [file_pc, file_pi]))
status = pexpect.run(command, events={r'\d+%': progress}, withexitstatus=1)[1]
print("Exit status %d" % status)

Do you have SSH enabled? Something like this could help you:
import os
os.system("scp FILE USER#SERVER:PATH")