I try to run Pylint from Python script using the library "subprocess"; however, I always receive the following error:
Command '['pylint', 'E:\python_projects\machine_learning_projects\alibi\', '--recursive=True']' returned non-zero exit status 30.
The source code is:
import os
import pandas as pd
import subprocess
def pylint_project(project_path):
if not os.path.exists(project_path):
raise ValueError("Path not valid.")
output = subprocess.check_output(['pylint', project_path, "--recursive=True"]).decode('utf-8')
rating = output.split('Your code has been rated at ')[1].split('/')[0]
except Exception as e:
print(e)
rating = None
df = pd.DataFrame({'project': [os.path.basename(project_path)], 'rating': [rating]})
return df
path = "E:\python_projects\machine_learning_projects\\alibi\\"
print(pylint_project(path))
I tried to run the same command using the CMD without errors (I suppose that always is correctly installed)
The Operative System that I'm using is "Windows 11".
How can I fix it?
Thats how pylint works. If it detects warnings/errors, it will return a bitmapped exit code of non-zero value to indicate what sort of things it detected.
Now, you are using subprocess.check_output() - on its documentation, following is said:
If the return code was non-zero it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute and any output in the output attribute.
(emphasis mine)
If you do pylint --help, you will see following:
--exit-zero Always return a 0 (non-error) status code, even if
lint errors are found. This is primarily useful in
continuous integration scripts. (default: False)
So, add --exit-zero to your arguments passed to pylint or add exception handling for CalledProcessError or switch to subprocess.run() ?
Do not use subprocess for that, pylint has an API:
from pylint import run_pylint
run_pylint(argv=[project_path, "--recursive=True"])
See https://pylint.readthedocs.io/en/latest/development_guide/api/pylint.html
Related
Problem
As the title says, I am trying to run sphinx-apidoc from subprocess.run() on Ubuntu 20.04 in Azure DevOps Build Pipeline.
My problem is that I seem to get an error but no message and nothing is really executed?
My code is
call = ['sphinx-apidoc']
try:
res = subprocess.run(call, text=True, check=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
print(res)
print("stdout: ", res.stdout)
print("stderr: ", res.stderr)
except subprocess.CalledProcessError as e:
print("CalledProcessError: " + e.output)
and my output is
CalledProcessError:
without any output.
What I tried
I can invoke sphinx-apidoc using a pipeline step
task: CmdLine#2
And I can also call for example python --version using the above subprocess.run(), using
call= ['python']
call.append('--version')
Why is it that I do not get an output from an error?
Why is it not working although other commands like running python works?
why can I execute the command from a pipeline step without a problem?
Update - Task Definitions
For the test command, I just use this:
- task: CmdLine#2
inputs:
script: |
sphinx-apidoc *putfolder *source
for my python script that should run Subprocess.run()
Python3.9.15
- task: PythonScript#0
inputs:
scriptSource: 'filePath'
scriptPath: '$(System.DefaultWorkingDirectory)/myScript.py'
p.s. I know that only calling sphinx-apidoc without arguments will lead to an error, this is just for the sake of simplicity. And it should still give me a proper error message, so I know the subprocess was run properly.
Ok, so after trying a lots of different things, I want to answer myself:
There was a magnitude of problems.
1. sphinx-apidoc module path
I provided an absolute module path that sphinx-apidoc does not like. Need to pass in a relative path
2. properly use capture_output-option
In the end, I removed the stdout=subprocess.PIPE and just set capture_output. I also removed shell=True that I had used temporarily
3. Check command string
I used subprocess.list2cmdline(*listOfArgs*) to verify my command, I also had to remove quotes/double quotes that were not needed.
So had nothing to do with Azure DevOps or some wrong environment setup, just me not being able to properly develop in python + handling strings and commands under Ubuntu :D But maybe this still is of help to someone
Final Code
(not perfect, also shows you the command that was sent and gives you the output from the command)
cmd = ['sphinx-apidoc']
cmd.append('-f')
...
try:
res = subprocess.run(callList, text=True, capture_output=True, check=True)
print(res)
print(res.stdout)
except subprocess.CalledProcessError as e:
print("CalledProcessError: " + str(e.output))
name = input('name? ')
if len(name) == 0:
print('error.\n')
raise SystemExit
I receive an error when using python 3.3.2 (which is the version is school sadly) but it works fine on other versions e.g. 2.7.10 and 3.5
This is the error
Looking at the screenshot I can see Python prompt at the bottom:
This means the script is run in an interactive session (IDLE on Windows I guess). I haven't found any documentation, but other users have discovered that raising SystemExit in an interactive session does print the traceback.
So you should check and ensure that you are not launching the script in an interactive session.
Old answer:
Looks like it's a bug (or a particularity) in Python 3.3.2. According to this blog post:
If nothing catches the exception, then the python interpreter catches
it at the end, does not print a stack trace, and then calls exit.
I tried to raise SystemExit('asd') and the program just printed asd, so looks like it's true.
Either upgrade Python or try os._exit(1).
Not sure if this is what you want, but if you wanna exit use this:
import sys
name = raw_input('name? ')
if len(name) == 0:
print('error.\n')
sys.exit()
This exits the interpreter by raising SystemExit.
Why don't you use sys.exit()?
sys.exit([arg])
Exit from Python. This is implemented by raising the SystemExit exception, so cleanup actions specified by finally clauses of try statements are honored, and it is possible to intercept the exit attempt at an outer level.
You probably have an handler set for excepthook:
https://docs.python.org/3/library/sys.html#sys.excepthook
You should be able to reset it by doing
sys.excepthook = sys.__excepthook__
Nevermind, the hook works correctly for BaseException, but weirdly enough not for SystemExit (which is a subclass of BaseException).
You're probably executing your program with
python3 -i whatever.py
This gives me the same behavior you witnessed:
dario#feynman ~> python3 -i /tmp/a.py
Traceback (most recent call last):
File "/tmp/a.py", line 11, in <module>
raise SystemExit()
SystemExit
>>>
Note the >>> at the end.
Just remove the -i flag, from whatever is executing your program
Alternatively, it's bad practice, but you can also use os._exit(1)
I would like to import a function (function.py) from a given module in python (MOD.py), whose location I do not know. For it, I have performed two steps:
First step, I get the path to the directory that contains the module:
path = subprocess.check_output(['find', 'home/scripts','-iname','MOD.py','|','sed','s/\/MOD.py//g']).rstrip()
Secondly, I point at this directory to get the function from the module:
sys.path.insert(0,'{0}'.format(path))
from MOD import function
The code written is failing in the first step, particularly in the sed. Why is it not working? Is there a clearer way to do the first step? Is it necessary to do two steps, or is it possible to do it with one python instruction?
Thanks!
First note that you could not use pipe like that ! for using pipe you have to pass shell=True so instead of check_output use Popen, also your code failed in path argument of find add a / before home .
If the executed command returns a nonzero exit code, an exception is raised. you can use a try-except with subprocess.CalledProcessError to catch errors and getting the output created along with the exit code :
import subprocess
try:
ps = subprocess.Popen(['find', '/home/scripts','-iname','MOD.py','|','sed','s/\/MOD.py//g'],shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
path= ps.communicate()[0]
except subprocess.CalledProcessError as e:
out_bytes = e.output
code= e.returncode
in addition as a more secure method i suggest don't use shell=True instead use tow command :
ps = subprocess.Popen(['find', '/home/scripts','-iname','MOD.py'], stdout=subprocess.PIPE)
path = subprocess.check_output(['sed','s/\/MOD.py//g'], stdin=ps.stdout)
I'm trying to execute Google's cpplint.py on a group of my files and collect the results to one log file. However, I have not managed to beat the subprocess module. My current code is here:
import os, subprocess
rootdir = "C:/users/me/Documents/dev/"
srcdir = "project/src/"
with open(rootdir+srcdir+"log.txt", mode='w', encoding='utf-8') as logfile:
for subdir, dirs, files in os.walk(rootdir+srcdir):
for file in files:
if file.endswith(".h") or file.endswith(".cpp"):
filewithpath=os.path.join(subdir, file)
cmd=['c:/Python27/python.exe','C:/users/me/Documents/dev/cpplint.py','--filter=-whitespace,-legal,-build/include,-build/header_guard/', filewithpath]
output = subprocess.check_output(cmd)
logfile.write(output.decode('ascii'))
Trying to run the above code throws an error:
File "C:\Python32\lib\site.py", line 159
file=sys.stderr)
^ SyntaxError: invalid syntax Traceback (most recent call last): File "C:\Users\me\Documents\dev\project\src\verifier.py", line 19, in <module>
output = subprocess.check_output(cmd) File "C:\Python32\lib\subprocess.py", line 511, in check_output
raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['c:/Python27/python.exe', 'C:/users/me/Documents/dev/cpplint.py', '--filter=-whitespace,-legal,-build/include,-build/header_guard/', 'C:/users/me/Documents/dev/project/src/aboutdialog.cpp']' returned non-zero exit status 1
If I substitute the cmd with something simpler like:
cmd=['C:/WinAVR-20100110/bin/avr-gcc.exe','--version']
Then the script works as expected.
I have also tried to use a single command string instead of a list of strings as cmd, but the result is the same.
When debugging the code, I copied the list-of-strings-turned-into-the-command-line-command from the debugger and ran it in the Windows command line, and the command ran as expected.
The Python interpreter running my script is Python 3.2.
Any tips are greatly appreciated.
Looks like cpplint.py is simply exiting with a non-zero return code - which it might do, for instance, if it finds errors or "lint" in the source files it is checking.
See the documentation for subprocess.check_output. Note that if the command executed returns a non-zero exit code then a subprocess.CalledProcessError is raised.
You could work around it by watching for CalledProcessError, e.g.
try:
output = subprocess.check_output(cmd)
except subprocess.CalledProcessError as e:
# ack! cpplint.py failed... report an error to the user?
EDIT:
The SyntaxError seems to be the key here, and is probably caused by C:\Python32\lib being in your PYTHONPATH (either explicitly, or, this could happen if it is your current working directory).
The Python interpreter (since about 1.5.2-ish) automatically runs import site when started. So, when this is the case, and your script goes to execute:
c:/Python27/python.exe C:/users/me/Documents/dev/cpplint.py ...
then the Python 2.7 interpreter will find C:\Python32\lib\site.py first, and try to load that, instead of the one (presumably) at C:\Python27\lib\site.py. The issue is that Python 3's site.py contains syntax incompatible with Python 2, so the process launched by subprocess.check_output is failing with a SyntaxError before it even gets a chance to run cpplint, which propagates the CalledProcessError.
Solution? Make sure Python2 gets a bonafide Python2 "PYTHONPATH", and likewise for Python3! In other words, make sure C:\Python32\lib is not in the PYTHONPATH search path when running the Python2 interpreter.
One way to do this in your case is to set an explicit environment when launching the process, e.g.:
python2_env = {"PYTHONPATH": "path/to/python2/stuff:..."}
output = subprocess.check_output(cmd, env=python2_env)
I would request you to run this first
pipe = subprocess.Popen([cmd, options],stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout, stderr = pipe.communicate()
You will get to know what exactly is the error behind it, since CalledProcessError is raised only if the exit code was non-zero.
I did it by replacing the def main() with the following (I editet the errorfunction too to get a proper csv-file):
errorlog = sys.stderr
sys.stderr = open("errorlog.csv","w")
sys.stderr.write("File;Line;Message;Category;Confidence\n")
for filename in filenames:
ProcessFile(filename, _cpplint_state.verbose_level)
_cpplint_state.PrintErrorCounts()
sys.exit(_cpplint_state.error_count > 0)
sys.stdout = errorlog
sys.stderr.close()
Where can I find information about meaning of exit codes of "python" process on Unix? For instance, if I do "python thisfiledoesntexist.py", I get exit code 2
Summary:
from errno import errorcode
print errorcode[2]
As stated, mostly the error codes come from the executed script and sys.exit().
The example with a non-existing file as an argument to the interpreter fall in a different category. Though it's stated nowhere I would guess, that these exit codes are the "standard" Linux error codes. There is a module called errno that provides these error numbers (the exit codes come from linux/include/errno.h.
I.e.: errno.ENOENT (stands for for "No such file or directory") has the number 2 which coincides with your example.
The Python manual states this regarding its exit codes:
Some systems have a convention for assigning specific meanings to specific exit codes, but these are generally underdeveloped; Unix programs generally use 2 for command line syntax errors and 1 for all other kind of errors.
So, since you specified thisfiledoesntexist.py as a command line argument, you get a return code of 2 (assuming the file does not, in fact, exist. In that case I'd recommend renaming it to thisfiledoesexist.py. ;) )
Other that such parsing errors, the return code is determined by the Python program run. 0 is returned unless you specify another exit code with sys.exit. Python itself does not interfere.
Maybe exit code constants from os module can help you. Also have a look at sys.exit documentation.
Unfortunately, there is no 100% guarantee that Python's exit codes will be what the documentation claims they will be: os._exit allows the Python programmer to define which exit code is supposed to be used, which means python file_exists_but_claims_that_it_does_not.py could exit with os.EX_DATAERR.