I have been testing the stderr with the subprocess module. If I write a simple test with shell=True with the linux shell command ls intentionally bad typed:
p=subprocess.Popen(["lr"],stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=True)
out, err=p.communicate()
print ("standard error")
print(err)
it outputs the usual from the shell: lr: command not found.
But if shell=False, I don't quite understand why the program has an error executing
Traceback (most recent call last):
File "importInteresantes.py", line 6, in <module>
p=subprocess.Popen(["lr"],stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=False)
File "/usr/lib/python2.7/subprocess.py", line 390, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1024, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I thought that it would give me the same output. Is the code wrong or is the point of view that I should obtain the same stderr?
NOTE: Just in case I also tried with python3
With shell=True, Python launches a shell and tells the shell to run lr. The shell runs just fine, fails to find an lr program, and produces error output reporting this failure.
With shell=False, Python tries to run lr directly. Since there is no lr program to run, Python can't find an executable file corresponding to lr. Python can't launch the subprocess at all, and there are no stdout or stderr streams to read from. Python raises an exception reporting its failure to find the file.
This behavior is normal and expected.
Related
I have a command saved in a text file that I wish to execute using subprocess.run() from R using the reticulate package.
I have a directory with three files:
test_command.txt which contains the command touch foo.txt
run_command.py:
import subprocess
import os
subprocess.check_output('bash test_command.txt')
print(os.path.isfile("foo.txt")) # Check if the command was actually executed properly
run_from_r.R:
library(reticulate)
use_condaenv("my_env") # Same conda environment as used for python
source_python("run_command.py")
When I run run_command.py directly, foo.txt is created, and True is returned.
However, when I run from R using run_from_r.R, I get the following message:
Error in py_run_file_impl(file, local, convert) :
OSError: [WinError 6] The handle is invalid
Detailed traceback:
File "<string>", line 5, in <module>
File "C:\Users\Danie\miniconda3\envs\wildcats_summer_env\lib\subprocess.py", line 411, in check_output
**kwargs).stdout
File "C:\Users\Danie\miniconda3\envs\wildcats_summer_env\lib\subprocess.py", line 488, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\Danie\miniconda3\envs\wildcats_summer_env\lib\subprocess.py", line 753, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "C:\Users\Danie\miniconda3\envs\wildcats_summer_env\lib\subprocess.py", line 1054, in _get_handles
p2cread = _winapi.GetStdHandle(_winapi.STD_INPUT_HANDLE)
system("bash test_command.txt") runs properly in R.
Any idea what this error message means, and how I can make the command run properly when running using subprocess.check_output/run and reticulate?
Thanks!
I seem to have fixed it by using:
subprocess.run(['bash', 'test_command.txt'], stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL)
From Python running as Windows Service: OSError: [WinError 6] The handle is invalid
What is needed to use subprocess.run() inside a Flask app?
Even a simple example from https://docs.python.org/3.6/library/subprocess.html fails.
process = subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE)
output = process.stdout
app.logger.info(f"Process output: {output}")
File "./main.py", line 209, in process_pdf
process = subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE)
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ls': 'ls'
Python 3.6.8 (on Ubuntu 18.04LTS)
Flask is being served by uwsgi (from nginx)
I started from more complicated examples, trying with shell=True and other arguments, but nothing seems to be working.
subprocess.run() works just fine when called from command line sub.py
process = subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE)
output = process.stdout
print(f"Results {output}")
Results b'crw-rw-rw- 1 root root 1, 3 Nov 28 15:10 /dev/null\n'
Within Flask I can use old os.popen - no results though
stream = os.popen('ls -l /dev/null')
output = stream.readlines()
app.logger.info(f"Process output: {output}")
EDIT: Thanks to #furas and #Dursug for pointing me in the right direction. It seems like lack of shell issue for www-data.
So what would be the most Pythonic / Flask way of solving this?
PS I want to run specific external programs such as imagemagick, pdftotext, but I want to avoid wrappers/bindings (sometimes there are none).
This did turn out to be an enviroment issue for www-data which only had access to virtualenv path where Flask app was residing.
Solved by editing
/etc/systemd/system/myproject.service and adding :/usr/bin:/bin
as in
Environment="PATH=/home/myname/myproject/myprojectenv/bin:/usr/bin:/bin"
Then restarted nginx and service
As suggested by one of the answers on
uWSGI python subprocess chrome/firefox failed.
The question remains open whether this is the best practice for Flask.
Theoretically this opens up a potential vulnerability if client can find a way to run an arbitrary command.
This question already has answers here:
Python subprocess.run('ls',shell=True) not working on windows
(2 answers)
Closed 1 year ago.
I want to run shell scripts and git-bash commands using subprocess.run(), in python 3.7.4. When I run the simple example on the subprocess documentation page this happens:
import subprocess
subprocess.run(["ls", "-l"])
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "C:\pycharm\project\envs\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\pycharm\project\envs\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:\pycharm\project\envs\lib\subprocess.py", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
# it also fails with shell=True
subprocess.call(["ls", "-l"], shell=True)
'ls' is not recognized as an internal or external command,
operable program or batch file.
1
The message from shell=True is a message from windows cmd, which suggests subprocess is not sending commands to git-bash.
I am using a conda environment located in the project/envs/ folder for python. I have also installed git-bash.
I also tried setting the env and got the same error.
import os
import subprocess
my_env = os.environ.copy()
my_env["PATH"] = 'C:\Program Files\Git\;' + my_env["PATH"]
subprocess.run(['git-bash.exe', 'ls', '-l'], env=my_env)
Traceback (most recent call last):
File "<input>", line 3, in <module>
File "C:\pycharm\project\envs\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\pycharm\project\envs\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:n\pycharm\project\envs\lib\subprocess.py", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
I can get it to run by pointing at the git-bash.exe, but it returns an empty string instead of the files in my directory
import subprocess
subprocess.run(['C:\Program Files\Git\git-bash.exe', 'ls', '-l'], capture_output=True)
CompletedProcess(args=['C:\\Program Files\\Git\\git-bash.exe', 'ls', '-l'], returncode=0, stdout=b'', stderr=b'')
I would appreciate any advice on the best way to get this working as shown on the subprocess documentation page.
I found that I can run commands using ...Git\bin\bash.exe instead of the ...\Git\git-bash.exe, like this:
import subprocess
subprocess.run(['C:\Program Files\Git\\bin\\bash.exe', '-c','ls'], stdout=subprocess.PIPE)
CompletedProcess(args=['C:\\Program Files\\Git\\bin\\bash.exe', '-c', 'ls'], returncode=0, stdout=b'README.md\n__pycache__\nconda_create.sh\nenvs\nmain.py\ntest.sh\nzipped\n')
Try this
p = subprocess.Popen(("ls", "-l"), stdout=subprocess.PIPE)
nodes = subprocess.check_output(("grep"), stdin=p.stdout)
p.wait()
ls is Linux shell command for listing files and directories
dir is Windows command line command for listing files and directories
Try to run dir in Windows command line. If it works, try to run the same command using python subprocess:
import subprocess
subprocess.run(["dir"])
For a machine with Windows Operating System, Try the following
import subprocess
subprocess.run(["dir", "/p"], shell=True)
"ls" is replaced with "dir", "-l" is replaced with "/l" and the "shell" is set to true
For a machine with Linux/Mac Operating System, Try the following
import subprocess
subprocess.run(["ls", "-l"])
I have a compiled binary for Linked list written in c. I placed the executable in /usr/bin/ as /usr/bin/app where app is the name of the executable. This was compiled using gcc.
Can anyone help me to invoke this (app) using a python script.
I have written a script below to do this but seems to give errors. I am very new to python and have very basic knowledge on this. I am just exploring pythons features.
Below is the script code:
#!/usr/bin/env python
import subprocess
proc = subprocess.Popen(['\usr\bin\app'],
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE
)
(out, err) = proc.communicate()
print out
Here are the errors:
Traceback (most recent call last):
File "./LinkedList.py", line 7, in <module>
stderr = subprocess.PIPE
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Thank you for your assistancce
Per Comments the answer was:
Use forward slashes '/usr/bin/app'
Personally though I would strongly consider using os.path.join or str.join and os.sep so you don't have to remember which way the slashes should go.
http://docs.python.org/2/library/os.html
http://docs.python.org/2/library/os.path.html
So i am running into a problem using subprocess.call() and i think i may just be calling it wrong. I am using:
subprocess.call('testingosfile.py')
and i get the traceback:
Traceback (most recent call last):
File "<pyshell#14>", line 1, in <module>
subprocess.call('testingosfile.py')
File "C:\Python27\lib\subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Python27\lib\subprocess.py", line 679, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 893, in _execute_child
startupinfo)
WindowsError: [Error 193] %1 is not a valid Win32 application
and the contents of testingosfile.py is:
print "hello world!"
raw_input('....')
how do i manage to get this running?
Thank you in advance for your replies.
The error message makes perfect sense - with subprocess, you can only start an executable. So, to fix it, you should start an executable. Specifically, you should start the Python interpreter and tell it to run your script. Something like
subprocess.call(['python.exe', 'testingosfile.py'])
should work, although you might have to provide the full path to the Python interpreter (I can't test at the moment).
However, have you considered importing testingosfile.py instead? Whenever you import a Python script, all the commands in that script are run. Using
import testingosfile
inside a function in order to execute the commands would be poor style, but you could package up the useful commands of testingosfile.py into some function. Then, you could use
import testingosfile
at the top of your main script, and just call that function whenever you want to print Hello World and get the user's input.
try subprocess.call("myfile.ext", shell=True)