I am trying to make a basic linter script which I can run on Python files in the current directory. So far my script looks like this:
import subprocess
from os import listdir
from os.path import isfile, join
if __name__ == "__main__":
subprocess.check_output(["black", "-l", "100", "./"])
files = [f for f in listdir("./") if isfile(join("./", f))]
for file in files:
if file.endswith(".py"):
subprocess.check_output(["flake8", file])
I am wanting to run the code via the command line with a call such as main.py. Black performs fine and finds the .py files in the current directory and formats them without a problem. However, when trying to run a similar command with flake8, it also runs on children of the directory such as the venv folder which I am not interested in.
Therefore, the script includes a check to get the files in the current directory and then find the .py files. However, once I get those files, I cannot seem to use my flake8 command with subprocess.check_output. The error I get is as follows:
Traceback (most recent call last):
File "linter.py", line 18, in <module>
subprocess.check_output(["flake8", file], shell=False)
File "C:\Users\calum\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 411, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "C:\Users\calum\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['flake8', 'flask_server_controller.py']' returned non-zero exit status 1.
Could someone please explain the error and/or provide a solution to my problem. I would also like to add other linting tools to the script such as pylint, however, I am worried I will run into the same problem without understanding it properly.
Thanks in advance.
subprocess.check_output is giving you that because of the check_ aspect
this means that when the executable you're running returns nonzero (for example, flake8 returns nonzero when it detects lint failures) an exception will be raised
To avoid this behaviour, I'd suggest using subprocess.run instead, and forwarding along the return code. Something like this:
import os
import subprocess
import sys
def main():
ret = 0
output = b''
for filename in os.listdir('.'):
if filename.endswith('.py'):
proc_ret = subprocess.run(
('flake8', filename),
stdout=subprocess.PIPE,
)
ret |= proc_ret.returncode
output += proc_ret.stdout
sys.stdout.buffer.write(output)
return ret
if __name__ == '__main__':
exit(main())
Note that this is going to be prohibitively slow, you have to incur the startup cost of flake8 for every file.
One way you can improve this is by passing all the filenames to flake8 at once:
import os
import subprocess
import sys
def main():
filenames = (fname for fname in os.listdir('.') if fname.endswith('.py'))
proc_ret = subprocess.run(('flake8', *filenames), stdout=subprocess.PIPE)
sys.stdout.buffer.write(proc_ret.stdout)
return proc_ret.returncode
if __name__ == '__main__':
exit(main())
but also this brings up another interesting point, why are you collecting the output at all? if you let the output go to stdout it will be printed automatically:
import os
import subprocess
def main():
filenames = (fname for fname in os.listdir('.') if fname.endswith('.py'))
return subprocess.call(('flake8', *filenames))
if __name__ == '__main__':
exit(main())
and hmmm, you probably don't need to do this at all since flake8 has its own inclusion / exclusion code -- you probably just want to configure exclude properly
# setup.cfg / tox.ini / .flake8
[flake8]
# for example, exclude recursing into the venv
exclude = venv
and then you can use flake8 . as normal
(disclaimer: I am the current maintainer of flake8)
Related
I have been on this problem for quite a while now. I have this command line that I want run trough python:
Users\name.lastname\Desktop\TESTER\Latitude 5431\Latitude-5431-46KCM_Win10_1.0_A01.exe /s /e=C:Users\name.lastname\Desktop\TESTER\Latitude 5431
this should run the .exe and then extract the files to the specified folder. I tried this with os.system and it worked but when I run it with
import subprocess
x = '"' + "\\Users\\name.lastname\\Desktop\\TESTER\\Latitude 5431\\Latitude-5431-46KCM_Win10_1.0_A01.exe" + '" ' + "/s /e=C:Users\\name.lastname\\Desktop\\TESTER\\Latitude 5431"
p1 = subprocess.run(x, shell=True)
it only shows me 'tips' like these but no error message and the .exe is not executed.
Pass command line arguments directly to vendor installer.
Turn the return code to success if required
Latitude-5431-46KCM_Win10_1.0_A01.exe /factoryinstall /passthrough D:\Sample.xml C:\log\FI.log
Change from the default log location to C:\my path with spaces\log.txt
Latitude-5431-46KCM_Win10_1.0_A01.exe /l="C:\my path with spaces\log.txt"
Force update to continue, even on "soft" qualification errors
Latitude-5431-46KCM_Win10_1.0_A01.exe /s /f
Try running without shell=True as it makes things more complicated than it helps:
import subprocess
prog = r"C:\Users\name.lastname\Desktop\TESTER\Latitude 5431\Latitude-5431-46KCM_Win10_1.0_A01.exe"
args = ["/s", r"/e=C:\Users\name.lastname\Desktop\TESTER\Latitude 5431"]
subprocess.run([prog]+args)
I have a python script that I want to call from the windows file browser context menu (https://www.howtogeek.com/107965/how-to-add-any-application-shortcut-to-windows-explorers-context-menu/)
I am currently debugging calling it from the non-specific context (HKEY_CLASSES_ROOT\Directory\Background\shell) with the command "python "D:\toolbox\mineAudio.py" 0"
(note python3 is on the path as python and the script is at D:\toolbox\mineAudio.py)
When I call the script from cmd it works as expected with that command, and when I make debug modifications to the script (adding os.system("pause") to random lines) I can verify it is running correctly up to the point it hits the line meta=cmd(['ffmpeg','-i',target]) (line 46) where it instantly and silently fails (note ffmpeg is also on the path)
EDIT: it actually gets as far as line 15 result = subprocess.run(command, stdout=subprocess.PIPE,stderr=subprocess.PIPE,startupinfo=startupinfo)
I cant figure out why the program is failing there as that line works fine everywhere else I have tested the script from other than the context menu.
Here is the full script if you want to brows through it
import subprocess
import os
import sys
from sys import argv
from tree import tree
#for command line use:
#mineAudo.py [prompt=1] [dir=cwd]
#first arg prompt will prompt user for dir if 1, otherwise it wont
#second arg is the directory to use, if specified this will override prompt, if not and prompt=0, current working dir is used
def cmd(command):
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE
result = subprocess.run(command, stdout=subprocess.PIPE,stderr=subprocess.PIPE,startupinfo=startupinfo)
return result.stderr.decode("utf-8")
def stripStreams(meta):
i=1;
lines=[]
while i>0 :
i=meta.find("Stream",i+1)
lineEnd=meta.find("\n",i)
lines.append(meta[i:lineEnd])
return lines
def mineAudio(streams):
ret=[]
for stream in streams:
if "Audio:" in stream:
start =stream.find("#")+1
end=stream.find("(",start)
ret.append(stream[start:end])
return ret
def convDir(dirTarget):
targets=tree(dirTarget)
convList(targets,dirTarget)
def convList(targets,dirTarget):
print(targets)
#target="2018-05-31 06-16-39.mp4"
i=0
for target in targets:
i+=1
if(target[target.rfind("."):]==".mp4"):
print("("+str(i)+"/"+str(len(targets))+") starting file "+target)
meta=cmd(['ffmpeg','-i',target])
streams=stripStreams(meta)
streams=mineAudio(streams)
count=0
output=target[target.rfind("/")+1:target.rfind(".")]
file=target[target.rfind("/")+1:]
#print (output)
try:
os.mkdir(dirTarget+"\\"+output)
except:
pass
for s in streams:
print("converting track "+str(count+1)+" of "+str(len(streams)));
count+=1
cmd("ffmpeg -i \""+target+"\" -vn -sn -c:a mp3 -ab 192k -map "+s+" \""+dirTarget+"\\"+output+"\\"+output+" Track "+str(count)+".mp3\"")
print("moving "+target+" to "+dirTarget+"\\"+output+"\\"+file)
os.rename(target,dirTarget+"\\"+output+"\\"+file)
print("Finished file "+target)
else:
print("("+str(i)+"/"+str(len(targets))+") skiping non mp4 file "+target)
def prompt():
while True:
dirTarget=input("input target dir: ")
convDir(dirTarget)
if __name__ == "__main__":
sys.setrecursionlimit(2000)
if len(argv)>2:
if os.path.isdir(argv[2]):
convDir(argv[2])
else:
convList([argv[2]],os.path.dirname(argv[2]))
elif(len(argv)>1):
if int(argv[1])==1:
prompt()
else:
convDir(os.getcwd())
else:
prompt()
os.system("pause")
Note that I am not married to this particular implementation, any implementation with the same effect (extracting the .mp3 tracks from an .mp4 file automatically) would be fine too
also, here is the file Tree
#Returns the paths of all files in a directory and all sub directories relative to start directory
import os
def tree(directory,target="f"):
paths=[]
for currentDir,dirs,files in os.walk(directory):
if target=="f":
for file in files:
paths.append(currentDir+"/"+file)
if target=="d":
#paths.append(currentDir)
for dir in dirs:
paths.append(currentDir+"/"+dir)
for i in range(len(paths)):
paths[i]=paths[i].replace("\\","/")
return paths
Can anyone help me get this working?
Edit:
here is a shorter example code that crashes in the same way (still uses ffmpeg though)
import subprocess
import os
def cmd(command):
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE
result = subprocess.run(command,stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,stderr=subprocess.PIPE,startupinfo=startupinfo)
return result.stderr.decode("utf-8")
os.system("pause")
out=cmd(['ffmpeg','-i','D:\\ffmpeg test\\test\\2018-05-31 06-16-39\\2018-05-31 06-16-39.mp4'])
print(out)
os.system("pause")
(note the file is hard coded, program output should be
)
I managed to "solve" the problem by the round about hackish way of creating a batch file that calls the python script command, but this does seem a bit of a hack, and I would think there would be a better way.
I have been looking for an answer for how to execute a java jar file through python and after looking at:
Execute .jar from Python
How can I get my python (version 2.5) script to run a jar file inside a folder instead of from command line?
How to run Python egg files directly without installing them?
I tried to do the following (both my jar and python file are in the same directory):
import os
if __name__ == "__main__":
os.system("java -jar Blender.jar")
and
import subprocess
subprocess.call(['(path)Blender.jar'])
Neither have worked. So, I was thinking that I should use Jython instead, but I think there must a be an easier way to execute jar files through python.
Do you have any idea what I may do wrong? Or, is there any other site that I study more about my problem?
I would use subprocess this way:
import subprocess
subprocess.call(['java', '-jar', 'Blender.jar'])
But, if you have a properly configured /proc/sys/fs/binfmt_misc/jar you should be able to run the jar directly, as you wrote.
So, which is exactly the error you are getting?
Please post somewhere all the output you are getting from the failed execution.
This always works for me:
from subprocess import *
def jarWrapper(*args):
process = Popen(['java', '-jar']+list(args), stdout=PIPE, stderr=PIPE)
ret = []
while process.poll() is None:
line = process.stdout.readline()
if line != '' and line.endswith('\n'):
ret.append(line[:-1])
stdout, stderr = process.communicate()
ret += stdout.split('\n')
if stderr != '':
ret += stderr.split('\n')
ret.remove('')
return ret
args = ['myJarFile.jar', 'arg1', 'arg2', 'argN'] # Any number of args to be passed to the jar file
result = jarWrapper(*args)
print result
I used the following way to execute tika jar to extract the content of a word document. It worked and I got the output also. The command I'm trying to run is "java -jar tika-app-1.24.1.jar -t 42250_EN_Upload.docx"
from subprocess import PIPE, Popen
process = Popen(['java', '-jar', 'tika-app-1.24.1.jar', '-t', '42250_EN_Upload.docx'], stdout=PIPE, stderr=PIPE)
result = process.communicate()
print(result[0].decode('utf-8'))
Here I got result as tuple, hence "result[0]". Also the string was in binary format (b-string). To convert it into normal string we need to decode with 'utf-8'.
With args: concrete example using Closure Compiler (https://developers.google.com/closure/) from python
import os
import re
src = test.js
os.execlp("java", 'blablabla', "-jar", './closure_compiler.jar', '--js', src, '--js_output_file', '{}'.format(re.sub('.js$', '.comp.js', src)))
(also see here When using os.execlp, why `python` needs `python` as argv[0])
How about using os.system() like:
os.system('java -jar blabla...')
os.system(command)
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.
Preface: I understand this question has been asked before, but I cannot find a solution to my error from looking at those previous answers.
All I want to do is call diff for the output of two different commands on the same file.
import os, sys
from subprocess import check_call
import shlex
ourCompiler = 'espressoc';
checkCompiler = 'espressocr';
indir = 'Tests/Espresso/GoodTests';
check_call(["pwd"]);
for root, dirs, filenames in os.walk(indir):
for f in filenames:
if len(sys.argv) == 2 and sys.argv[1] == f:
str1 = "<(./%s ./%s) " % (ourCompiler, os.path.join(root, f))
str2 = "<(./%s ./%s) " % (checkCompiler, os.path.join(root, f))
check_call(["diff", str1, str2])
Why is it that I receive following error?
diff: <(./espressoc ./Tests/Espresso/GoodTests/Init.java) : No such file or directory
diff: <(./espressocr ./Tests/Espresso/GoodTests/Init.java) : No such file or directory
Traceback (most recent call last):
File "runTest.py", line 21, in <module>
check_call(["diff", str1, str2])
File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['diff', '<(./espressoc ./Tests/Espresso/GoodTests/Init.java) ', '<(./espressocr ./Tests/Espresso/GoodTests/Init.java) ']' returned non-zero exit status 2
If I were to run this command from my shell it works fine.
diff is complaining that it can't find the file with the strange name <(./espressoc ./Tests/Espresso/GoodTests/Init.java) because that's the argument you fed it.
subprocess.Popen (check_call is a convenience function for it) is directly calling what you give it, there isn't a shell to interpret redirections or anything, unless you tell it shell=True, which will then call the command via /bin/sh (on POSIX). Note the security considerations before using it.
So basically:
subprocess.check_call(['diff', '<this', '<that'])` # strange files.
subprocess.check_call('diff <this <that', shell=True)` # /bin/sh does some redirection
If you wanted to be "pure" (probably more effort than it's worth), I think you could subprocess all three processes (diff, compiler 1 and 2) and handle the piping yourself. Does diff wait for 2 EOFs or something before closing stdin? Not sure how it actually deals with the double input redirection like your line has...
I have a Python script that needs to invoke another Python script in the same directory. I did this:
from subprocess import call
call('somescript.py')
I get the following error:
call('somescript.py')
File "/usr/lib/python2.6/subprocess.py", line 480, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I have the script somescript.py in the same folder though. Am I missing something here?
If 'somescript.py' isn't something you could normally execute directly from the command line (I.e., $: somescript.py works), then you can't call it directly using call.
Remember that the way Popen works is that the first argument is the program that it executes, and the rest are the arguments passed to that program. In this case, the program is actually python, not your script. So the following will work as you expect:
subprocess.call(['python', 'somescript.py', somescript_arg1, somescript_val1,...]).
This correctly calls the Python interpreter and tells it to execute your script with the given arguments.
Note that this is different from the above suggestion:
subprocess.call(['python somescript.py'])
That will try to execute the program called python somscript.py, which clearly doesn't exist.
call('python somescript.py', shell=True)
Will also work, but using strings as input to call is not cross platform, is dangerous if you aren't the one building the string, and should generally be avoided if at all possible.
Windows? Unix?
Unix will need a shebang and exec attribute to work:
#!/usr/bin/env python
as the first line of script and:
chmod u+x script.py
at command-line or
call('python script.py'.split())
as mentioned previously.
Windows should work if you add the shell=True parameter to the "call" call.
Check out this.
from subprocess import call
with open('directory_of_logfile/logfile.txt', 'w') as f:
call(['python', 'directory_of_called_python_file/called_python_file.py'], stdout=f)
import subprocess
command = 'home/project/python_files/run_file.py {} {} {}'.format(
arg1, arg2, arg3) # if you want to pass any arguments
p = subprocess.Popen(
[command],
shell=True,
stdin=None,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
close_fds=True)
out, err = p.communicate()
subprocess.call expects the same arguments as subprocess.Popen - that is a list of strings (the argv in C) rather than a single string.
It's quite possible that your child process attempted to run "s" with the parameters "o", "m", "e", ...
If you're on Linux/Unix you could avoid call() altogether and not execute an entirely new instance of the Python executable and its environment.
import os
cpid = os.fork()
if not cpid:
import somescript
os._exit(0)
os.waitpid(cpid, 0)
For what it's worth.
What's wrong with
import sys
from os.path import dirname, abspath
local_dir = abspath(dirname(__file__))
sys.path.append(local_dir)
import somescript
or better still wrap the functionality in a function, e.g. baz, then do this.
import sys
from os.path import dirname, abspath
local_dir = abspath(dirname(__file__))
sys.path.append(local_dir)
import somescript
somescript.baz()
There seem to be a lot of scripts starting python processes or forking, is that a requirement?
First, check if somescript.py is executable and starts with something along the lines of #!/usr/bin/python.
If this is done, then you can use subprocess.call('./somescript.py').
Or as another answer points out, you could do subprocess.call(['python', 'somescript.py']).
def main(argv):
host = argv[0]
type = argv[1]
val = argv[2]
ping = subprocess.Popen(['python ftp.py %s %s %s'%(host,type,val)],stdout = subprocess.PIPE,stderr = subprocess.PIPE,shell=True)
out = ping.communicate()[0]
output = str(out)
print output
The subprocess call is a very literal-minded system call. it can be used for any generic process...hence does not know what to do with a Python script automatically.
Try
call ('python somescript.py')
If that doesn't work, you might want to try an absolute path, and/or check permissions on your Python script...the typical fun stuff.