I would like to import a function (function.py) from a given module in python (MOD.py), whose location I do not know. For it, I have performed two steps:
First step, I get the path to the directory that contains the module:
path = subprocess.check_output(['find', 'home/scripts','-iname','MOD.py','|','sed','s/\/MOD.py//g']).rstrip()
Secondly, I point at this directory to get the function from the module:
sys.path.insert(0,'{0}'.format(path))
from MOD import function
The code written is failing in the first step, particularly in the sed. Why is it not working? Is there a clearer way to do the first step? Is it necessary to do two steps, or is it possible to do it with one python instruction?
Thanks!
First note that you could not use pipe like that ! for using pipe you have to pass shell=True so instead of check_output use Popen, also your code failed in path argument of find add a / before home .
If the executed command returns a nonzero exit code, an exception is raised. you can use a try-except with subprocess.CalledProcessError to catch errors and getting the output created along with the exit code :
import subprocess
try:
ps = subprocess.Popen(['find', '/home/scripts','-iname','MOD.py','|','sed','s/\/MOD.py//g'],shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
path= ps.communicate()[0]
except subprocess.CalledProcessError as e:
out_bytes = e.output
code= e.returncode
in addition as a more secure method i suggest don't use shell=True instead use tow command :
ps = subprocess.Popen(['find', '/home/scripts','-iname','MOD.py'], stdout=subprocess.PIPE)
path = subprocess.check_output(['sed','s/\/MOD.py//g'], stdin=ps.stdout)
Related
I try to run Pylint from Python script using the library "subprocess"; however, I always receive the following error:
Command '['pylint', 'E:\python_projects\machine_learning_projects\alibi\', '--recursive=True']' returned non-zero exit status 30.
The source code is:
import os
import pandas as pd
import subprocess
def pylint_project(project_path):
if not os.path.exists(project_path):
raise ValueError("Path not valid.")
output = subprocess.check_output(['pylint', project_path, "--recursive=True"]).decode('utf-8')
rating = output.split('Your code has been rated at ')[1].split('/')[0]
except Exception as e:
print(e)
rating = None
df = pd.DataFrame({'project': [os.path.basename(project_path)], 'rating': [rating]})
return df
path = "E:\python_projects\machine_learning_projects\\alibi\\"
print(pylint_project(path))
I tried to run the same command using the CMD without errors (I suppose that always is correctly installed)
The Operative System that I'm using is "Windows 11".
How can I fix it?
Thats how pylint works. If it detects warnings/errors, it will return a bitmapped exit code of non-zero value to indicate what sort of things it detected.
Now, you are using subprocess.check_output() - on its documentation, following is said:
If the return code was non-zero it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute and any output in the output attribute.
(emphasis mine)
If you do pylint --help, you will see following:
--exit-zero Always return a 0 (non-error) status code, even if
lint errors are found. This is primarily useful in
continuous integration scripts. (default: False)
So, add --exit-zero to your arguments passed to pylint or add exception handling for CalledProcessError or switch to subprocess.run() ?
Do not use subprocess for that, pylint has an API:
from pylint import run_pylint
run_pylint(argv=[project_path, "--recursive=True"])
See https://pylint.readthedocs.io/en/latest/development_guide/api/pylint.html
I am running Jupyter notebook on a server (python 3).
Want to see output of OS command (any OS command - this is just example):
output = os.system("pwd")
When I do print of it:
print (output)
Response is 0.
How to get a simple output (like in CLI)?
Thanks.
Just found it on internet and wanted to post.
It needs to be:
print(os.popen('ls').read())
(or any other OS command).
This works fine.
import os
print(os.getcwd())
print(os.system("pwd"))
But this question is a duplicate:
how to show current directory in ipython promp
Note that os.system() calls are not the preferred way to run any commands and do not ensure capturing the output (see here for more on this).
The preferred and safer mechanism which will capture the output of commands is subprocess.run() which has a capture_output parameter and returns a CompletedProcess object that has members including stdout and stderr which, if capture_output=True contain the output stream contents.
It is worth mentioning that for portability it is usually better to use the methods from the os, shutil, path & glob libraries, etc. This is because calls such as ls, pwd, dir, etc., will work on some platforms but not others.
Example:
import subprocess
result = subprocess.run(['cwd',], capture_output=True)
# Returns the current directory as a string for processing in result.stdout on Mac/Linux but raises an exception on Windows
print(result.stdout)
result = subprocess.run(['ls', '*.docx'], capture_output=True)
# Returns the *.docx files in the current directory as a string for processing in result.stdout on Mac/Linux but raises an exception on Windows
print(result.stdout)
However:
import pathlib
cwd = pathlib.Path.cwd() # Returns the current directory as a Path object on any platform.
print(cwd)
docs = cwd.glob('*.docx') # Returns an generator giving path for to any docx files in cwd on any platform
print(', '.join(p.name for p in docs)) # Print comma seperated list of filenames
Note that for long running or very verbose calls it is often better to use the subprocess.POpen constructor and communicate or wait.
If you want to start an external process and not wait for it to finish then use the asynco create_subprocess_exec() call instead.
In Python 3.7 running on Windows, what specific syntax is required to:
1. Navigate to a directory containing a terraform program
2. Execute "terraform apply -auto-approve" in that target directory
3. Extract the resulting output variables into a form usable in python
The output variables might take the form:
security_group_id_nodes = sg-xxxxxxxxxx
vpc_id_myvpc = vpc-xxxxxxxxxxxxx
Want to be using windows cmd style commands here, NOT powershell.
My first failed newbie attempt is:
import os
os.chdir('C:\\path\\to\\terraform\\code')
from subprocess import check_output
check_output("terraform apply -auto-approve", shell=True).decode()
Not sure about your output, but subprocess could definitely make the trick.
Try something like:
command = 'terraform apply -auto-approve'
TARGET_DIR = 'E:\Target\Directory'
subprocess_handle = subprocess.Popen(shlex.split(command), cwd=TARGET_DIR, shell=False, stdout=subprocess.PIPE)
subprocess_handle.wait()
result = subprocess_handle.communicate()[0]
print(result)
Worked for me once, just play around with params.
UPD: Here I assume that "terraform" is an executable.
Assume a unittest test, in which a multi-line output file is generated via a Python script (that uses argparse) and the equality of that file against an expected outcome is compared.
def test_actual_vs_expected_output(self):
actual_inp = '/path_to/actu_inp.txt'
expect_otp = '/path_to/expe_otp.txt'
actual_otp = '/path_to/actu_otp.txt'
myScript = '/path_to/myScript.py'
cmd_list = ['python2', myScript,
'-i', actual_inp,
'-o', actual_otp]
try:
subprocess.check_output(' '.join(cmd_list), shell=True)
except subprocess.CalledProcessError as e:
print e.output
if os.path.isfile(actual_otp):
expect_str = open(expect_otp).read()
actual_str = open(actual_otp).read()
self.assertMultiLineEqual(expect_str, actual_str)
How can I avoid hardcoding the calling of python2 (i.e., in cmd_list of the above example)? After all, the Python2 interpreter may be called differently on different systems.
To call Python in a subprocess, you can use the currently running Python interpreter. The full path of this interpreter is given by the global variable sys.executable.
So, you can write:
import sys
cmd_list = [sys.executable, myScript,
'-i', actual_inp,
'-o', actual_otp]
A comment: the subprocess.check_output function accept a list of arguments, so you can pass the cmd_list parameter as-is (you don't need to join):
subprocess.check_output(cmd_list, shell=True)
Another comment: A python script may write error messages in STDERR. You may consider an alternative to check_output to get the error message or use stderr=subprocess.STDOUT.
I'm trying to execute Google's cpplint.py on a group of my files and collect the results to one log file. However, I have not managed to beat the subprocess module. My current code is here:
import os, subprocess
rootdir = "C:/users/me/Documents/dev/"
srcdir = "project/src/"
with open(rootdir+srcdir+"log.txt", mode='w', encoding='utf-8') as logfile:
for subdir, dirs, files in os.walk(rootdir+srcdir):
for file in files:
if file.endswith(".h") or file.endswith(".cpp"):
filewithpath=os.path.join(subdir, file)
cmd=['c:/Python27/python.exe','C:/users/me/Documents/dev/cpplint.py','--filter=-whitespace,-legal,-build/include,-build/header_guard/', filewithpath]
output = subprocess.check_output(cmd)
logfile.write(output.decode('ascii'))
Trying to run the above code throws an error:
File "C:\Python32\lib\site.py", line 159
file=sys.stderr)
^ SyntaxError: invalid syntax Traceback (most recent call last): File "C:\Users\me\Documents\dev\project\src\verifier.py", line 19, in <module>
output = subprocess.check_output(cmd) File "C:\Python32\lib\subprocess.py", line 511, in check_output
raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['c:/Python27/python.exe', 'C:/users/me/Documents/dev/cpplint.py', '--filter=-whitespace,-legal,-build/include,-build/header_guard/', 'C:/users/me/Documents/dev/project/src/aboutdialog.cpp']' returned non-zero exit status 1
If I substitute the cmd with something simpler like:
cmd=['C:/WinAVR-20100110/bin/avr-gcc.exe','--version']
Then the script works as expected.
I have also tried to use a single command string instead of a list of strings as cmd, but the result is the same.
When debugging the code, I copied the list-of-strings-turned-into-the-command-line-command from the debugger and ran it in the Windows command line, and the command ran as expected.
The Python interpreter running my script is Python 3.2.
Any tips are greatly appreciated.
Looks like cpplint.py is simply exiting with a non-zero return code - which it might do, for instance, if it finds errors or "lint" in the source files it is checking.
See the documentation for subprocess.check_output. Note that if the command executed returns a non-zero exit code then a subprocess.CalledProcessError is raised.
You could work around it by watching for CalledProcessError, e.g.
try:
output = subprocess.check_output(cmd)
except subprocess.CalledProcessError as e:
# ack! cpplint.py failed... report an error to the user?
EDIT:
The SyntaxError seems to be the key here, and is probably caused by C:\Python32\lib being in your PYTHONPATH (either explicitly, or, this could happen if it is your current working directory).
The Python interpreter (since about 1.5.2-ish) automatically runs import site when started. So, when this is the case, and your script goes to execute:
c:/Python27/python.exe C:/users/me/Documents/dev/cpplint.py ...
then the Python 2.7 interpreter will find C:\Python32\lib\site.py first, and try to load that, instead of the one (presumably) at C:\Python27\lib\site.py. The issue is that Python 3's site.py contains syntax incompatible with Python 2, so the process launched by subprocess.check_output is failing with a SyntaxError before it even gets a chance to run cpplint, which propagates the CalledProcessError.
Solution? Make sure Python2 gets a bonafide Python2 "PYTHONPATH", and likewise for Python3! In other words, make sure C:\Python32\lib is not in the PYTHONPATH search path when running the Python2 interpreter.
One way to do this in your case is to set an explicit environment when launching the process, e.g.:
python2_env = {"PYTHONPATH": "path/to/python2/stuff:..."}
output = subprocess.check_output(cmd, env=python2_env)
I would request you to run this first
pipe = subprocess.Popen([cmd, options],stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout, stderr = pipe.communicate()
You will get to know what exactly is the error behind it, since CalledProcessError is raised only if the exit code was non-zero.
I did it by replacing the def main() with the following (I editet the errorfunction too to get a proper csv-file):
errorlog = sys.stderr
sys.stderr = open("errorlog.csv","w")
sys.stderr.write("File;Line;Message;Category;Confidence\n")
for filename in filenames:
ProcessFile(filename, _cpplint_state.verbose_level)
_cpplint_state.PrintErrorCounts()
sys.exit(_cpplint_state.error_count > 0)
sys.stdout = errorlog
sys.stderr.close()