We Keep Coding

Python - shell command causes InterfaceError on file download

Recently, we replaced curl with aria2c in order to download files faster from our backend servers for later conversion to different formats.
Now for some reason we ran into the following issue with aria2c:
Pool callback raised exception: InterfaceError(0, '')
It's not clear to us where this InterfaceError occurs or what it actually could mean. Besides, we can trigger the executed command manually without any problems.
Please also have a look at our download function:
def download_file(descriptor):
"""
creates the WORKING_DIR structure and Download the descriptor.
The descriptor should be a URI (processed via aria2c)
returns the created resource path
"""
makedirs(WORKING_DIR + 'output/', exist_ok=True)
file_path = WORKING_DIR + decompose_uri(descriptor)['fileNameExt']
print(file_path)
try:
print(descriptor)
exec_command(f'aria2c -x16 "{descriptor}" -o "{file_path}"')
except CalledProcessError as err:
log('DEBUG', f'Aria2C error: {err.stderr}')
raise VodProcessingException("Download failed. Aria2C error")
return file_path
def exec_command(string):
"""
Shell command interface
Returns returnCode, stdout, stderr
"""
log('DEBUG', f'[Command] {string}')
output = run(string, shell=True, check=True, capture_output=True)
return output.returncode, output.stdout, output.stderr
Is stdout here maybe misunderstood by python which then drop into this InterfaceError?
Thanks in advance

As I just wanted to use aria2c to download files faster, as it support multiple connection, I now switched over to a tool called "axel". It also supports multiple connections without the excessive overhead aria2c has, at least for me in this situation.

Executing DevCon CMD command from python

I want to restart driver with DevCon from python script. It works from command line with this command:
devcon restart \"sd0007322081041363_kcanv\"
I tried this:
os.system("devcon restart \"sd0007322081041363_kcanv\"")
with result:
'devcon' is not recognized as an internal or external command
I read that os.system is obsolete and i need to use subprocess.check_output so i try this:
subprocess.check_output(['devcon', 'restart', '"sd0007322081041363_kcanv"'])
with result:
WindowsError:[Error 2] The system cannot find the file specified
and this:
subprocess.check_output('devcon restart "sd0007322081041363_kcanv"', shell=True)
with result:
subprocess.CalledProcessError: Command 'devcon restart "sd0007322081041363_kcanv"' returned non-zero exit status 1
and this:
subprocess.Popen("devcon restart \"sd0007322081041363_kcanv\"", shell=True, stdout=subprocess.PIPE).stdout.read()
result:
'devcon' is not recognized as an internal or external command
and this:
try:
subprocess.check_output('devcon disable "sd0007322081041363_kcanv" /f',shell=True,stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
with result:
RuntimeError: command 'devcon disable "sd0007322081041363_kcanv" /f' return with errpr (cpde 1): 'devcon' is not recognized as an internal or external command, operable program or batch file
devcon.exe is under Windows/System32 and it is set in system path.
I know that this can be duplicate question but I have tried many solution on stackoverflow but i van't resolve this issue.

Finally, I came up with a solution. I tried many things but this is what works for me:
copy devcon.exe from C:\Windows\System32 and put it to C:\Windows\SysWOW64.
my code:
try:
subprocess.check_output('C:\\Windows\\SysWOW64\\devcon.exe restart "sd0007322081041363_kcanv" /f',shell=True,stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))

Python Script not running ".sh" file

I have written a simple python script that is supposed to run a "sh" file when it is executed.
The problem is, that the script runs but it does not start the ".sh" file. When I execute the ".sh" file manually using "puffy" it does the job, but not when I use my python script. So, what do I have to change in my script in order for it to work?
I will post the methods below so you could get a better idea. I also am using python 3.3.5, Oracle Linux 6.8.
The method that calls the ".sh" file, I have used Popen.
def runPrestage2Stage(self):
import time
time.sleep(60)
reload(Queries)
if "FINISHED" in Queries.statusS2S:
#run batch file
p = Popen(["sh", "/u01/app/Oracle_ODI_11/oracledi/agent/bin/start_prestage2stage_simple.sh"], stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()
print("Prestage2Stage has started")
elif 'ERROR' in Queries.statusS2S:
print("Can not start Prestage Converter, please check runtime status of Stage Converter")
else:
print("Prestage2Stage Converter cannot be started")
Part of main method, that calls the method runPRestage2Stage.
DailyLoadsAutomation.DailyLoads.runPrestage2Stage(self)
load_paraprak_prestage = True
count2 = 0
while load_paraprak_prestage:
reload(Queries)
sleep(300) #waits 300 seconds (5 minutes) and re-checks load status.
if "FINISHED" in Queries.statusPreStage:
load_paraprak_prestage = False
else:
load_paraprak_prestage = True
if count2 == 8:
import sys
sys.exit()
else:
count2 += 1
print("PreStage is finished")
When I run the script,
It will print "Prestage2Stage has started", and "Prestage is finished", as it is supposed to, but It will not run the ".sh" file.
Any idea what is wrong?

usually it is related to user rights or errors in path. You can replace you .sh script with some simple one like "echo "I am running"" and see if it can be accessed and executed. If it is under linux, I hope you are are giving execution rights to you sh script via chmod.

You can run sh file by importing os module like this:
import os
os.system('sh filename.sh')

How to know if a service is installed

I'm looking for a way to check with a Python script if a service is installed. For example, if I want to check than a SSH server in installed/running/down in command line, I used :
service sshd status
If the service is not installed, I have a message like this:
sshd.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
So, I used a subprocess check_output to get this three line but the python script is not working. I used shell=True to get the output but it doen't work. Is it the right solution to find if a service is installed or an another method is existing and much more efficient?
There is my python script:
import subprocess
from shlex import split
output = subprocess.check_output(split("service sshd status"), shell=True)
if "Loaded: not-found" in output:
print "SSH server not installed"
The probleme with this code is a subprocess.CalledProcessError: Command returned non-zero exit status 1. I know that's when a command line return something which doesn't exist but I need the result as I write the command in a shell

Choose some different systemctl call, which differs for existing and non-existing services. For example
systemctl cat sshd
will return exit code 0 if the service exists and 1 if not. And it should be quite easy to check, isn't it?

Just catch the error and avoid shell=True:
import subprocess
try:
output = subprocess.check_output(["service", "sshd", "status"], stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
print(e.output)
print(e.returncode)

Command program not executing from PYTHON subprocess

I downloaded a python program (PYPDFOCR) that runs through the command line. PYPDFOCR has several dependencies (ghost script, image magick, etc..)
When I execute the program in the command line, it fails. The code executes GhostScript with all its arguments but I get the error that the command is not recognized.
If I go to the command line, I can execute ghost script "C:\Programs.....\gswin64c.exe" and pass the arguments and get the result. However, when I run PYPDFOCR, it fails everytime.
def _run_gs(self, options, output_filename, pdf_filename):
try:
cmd = '%s -q -dNOPAUSE %s -sOutputFile="%s" "%s" -c quit' % (self.binary, options, output_filename, pdf_filename)
logging.info(cmd)
out = subprocess.check_output(cmd, shell=True)
except subprocess.CalledProcessError as e:
print e.output
if "undefined in .getdeviceparams" in e.output:
error(self.msgs['GS_OUTDATED'])
else:
error (self.msgs['GS_FAILED'])
The error I get in the command is "C\Program" is not recognized as an internal or external command, operable program or batch file.
When I print the contents of the command, it shows:
c:\Program File\gs\gs9.16\bin\gswin64c.exe" -q -dNOPAUSE -sDEVICE=j[ecgray -dJPEG=75 -r300 -sOutputFILE="C:\test\a3_%d.jpg "c:\test\a3.pdf" -c quit
Again, I can run the c:...gswin64.exe command without the program.
Any help will be deeply appreciated.

The problem is probably that the space after Program in 'Program Files' is not correctly escaped. Use some function like
def shellquote(s):
return "'" + s.replace("'", "'\\''") + "'"
to escape your command before starting the process.
(Example function taken from here)

I had the same problem on Windows with windres.
It turned out to be a windres problem (MinGW64 4.8.3.20141208 from chocolatey).
I debugged into it and found that subprocess.Popen(cmd,**kw),
cmd being a list with first entry the executable's path,
mapped to _winapi.CreateProcess(executable, cmd, ...), with executable=None.
So the error originated from winapi.
In the Posix branch in subprocess there is
if executable is None:
executable = args[0]
which is missing in the windows branch.
As a try I setexecutable in **kw: Still the same problem.
Then I found https://amindlost.wordpress.com/2012/06/09/mingw-windres-exe-cant-popen-error/.
Indeed it worked with msys2's ming64's windres.