Hello I am trying to write a script to unzip bz2 files with 7zip and python.
First when I write
PS> & 'C:\Program Files\7-zip\7z.exe' e D:\path\example.bz2
with powershell it is working perfectly.
So I tried that with Python:
import glob
import subprocess
target = r"D:\path\*.bz2"
tab = glob.glob(target)
for i in range(len(tab)):
subprocess.call("& 'C:\\Program Files\\7-zip\\7z.exe' e %s" %tab[i], shell=True)
And I've got that error message: & was unexpected.
Does someone has an idea why ?
I am using Python 3.9.2
By using shell=True, you're opting to pass the command line to the platform-native shell, which on Windows is cmd.exe, not PowerShell, so your PowerShell command line fundamentally cannot be expected to work as-is.
If we take a step back: you don't need to involve the shell at all in your 7z.exe call, and not doing so also speeds up your operation.
By omitting shell=True, the target executable and its arguments must then be passed as the elements of an array rather than as a single command-line string.
for i in range(len(tab)):
exitCode = subprocess.call([ 'C:\\Program Files\\7-zip\\7z.exe', 'e', tab[i] ])
Note the use of exitCode = , which captures 7z.exe's exit code and therefore allows you to check for failure.
Alternatively, you could let Python raise an exception on failure automatically, by using subprocess.check_call() rather than subprocess.call()
Related
I am trying to make a python program(python 3.6) that writes commands to terminal to download a specific youtube video(using youtube-dl).
If I go on terminal and execute the following command:
cd; cd Desktop; youtube-dl "https://www.youtube.com/watch?v=b91ovTKCZGU"
It will download the video to my desktop. However, if I execute the below code, which should be doing the same command on terminal, it does not throw an error but also does not download that video.
import subprocess
cmd = ["cd;", "cd", "Desktop;", "youtube-dl", "\"https://www.youtube.com/watch?v=b91ovTKCZGU\""]
print(subprocess.call(cmd, stderr=subprocess.STDOUT,shell=True))
It seems that this just outputs 0. I do not think there is any kind of error 0 that exists(there are error 126 and 127). So if it is not throwing an error, why does it also not download the video?
Update:
I have fixed the above code by passing in a string, and have checked that youtube-dl is installed in my default python and is also in the folder where I want to download the videos, but its still throwing error 127, meaning command "youtube-dl" is not found.
cd; cd Desktop; youtube-dl "https://www.youtube.com/watch?v=b91ovTKCZGU" is not a single command; it's a list (delimited by ;) of three separate commands.
subprocess.call(cmd, ..., shell=True) is effectively the same as
subprocess.call(['sh', '-c'] + cmd)
which is almost never what you want. Instead, just pass a single string and let the shell parse it.
subprocess.call('cd; cd Desktop; youtube-dl "https://www.youtube.com/watch?v=b91ovTKCZGU"', shell=True)
If you really want to use the list form (which is always a good idea), use the cwd parameter instead of running cd.
subprocess.call(['youtube-dl', 'https://www.youtube.com/watch?v=b91ovTKCZGU'],
cwd=os.path.expanduser("~/Desktop"))
I'll answer this with an example:
>>> subprocess.call(["echo $0 $2", "foo", "skipped", "bar"], shell=True)
foo bar
0
The first element of the list is the shell command (echo $0 $2), and the remaining elements are the positional parameters that the command can optionally use ($0, $1, ...).
In your example, you are creating a subshell that only runs the cd; command. The positional parameters are ignored. See the Popen and bash docs for details.
As noted in the comments, you should make the command a string (not a list).
I'm currently trying to run an R script from the command line (my end goal is to execute it as the last line of a python script). I'm not sure what a batch file is, or how to make my R script 'executable'. Currently it is saved as a .R file. It works when I run it from R.
How do I execute this from the windows command prompt line? Do i need to download something called Rscript.exe? Do I just save my R script as an .exe file? Please advise on the easiest way to achieve this.
R: version 3.3 python: version 3.x os: windows
As mentioned, Rscript.exe the automated executable to run R scripts ships with any R installation (usually located in bin folder) and as #Dirk Eddelbuettel mentions is the recommended automated version. And in Python you can run any external program as a subprocess with various types including a call, check_output, check_call, or Popen and the latter of which provides more facility such as capturing errors in the child process.
If R directory is in your PATH environmental variable, you do not need to include full path to RScript.exe but just name of program, Rscript. And do note this is fairly the same process for Linux or Mac operating systems.
command = 'C:/R-3.3/bin/Rscript.exe' # OR command = 'Rscript'
path2script = 'C:/Path/To/R/Script.R'
arg = '--vanilla'
# CHECK_CALL VERSION
retval = subprocess.check_call([command, arg, path2script], shell=True)
# CALL VERSION
retval = subprocess.call(["'Rscript' 'C:/Path/To/R/Script.R'"])
# POPEN VERSION (W/ CWD AND OUTPUT/ERROR CAPTURE)
curdir = 'C:/Path/To/R/Script'
p = subprocess.Popen(['Rscript', 'Script.R'], cwd=curdir,
stdin = subprocess.PIPE, stdout = subprocess.PIPE,
stderr = subprocess.PIPE)
output, error = p.communicate()
if p.returncode == 0:
print('R OUTPUT:\n {0}'.format(output.decode("utf-8")))
else:
print('R ERROR:\n {0}'.format(error.decode("utf-8")))
You already have Rscript, it came with your version of R. If R.exe, Rgui.exe, ... are in your path, then so is Rscript.exe.
Your call from Python could just be Rscript myFile.R. Rscript is much better than R BATCH CMD ... and other very old and outdated usage patterns.
You probably already have R, since you can already run your script.
All you have to do is find its binaries (the Rscript.exe file).
Then open windows command line ([cmd] + [R] > type in : "cmd" > [enter])
Enter the full path to R.exe, followed by the full path to your script.
I have been looking through some similar posts and through the ImageMagick page, but I cannot seem to find a reason for my issue:
Note I am using a windows machine.
I have a .ps image in a folder and it works when running with the command to convert it from the cmd: convert saved.ps newsaved.png
However when I try to execute it from my python script with the following code:
args = ["convert","saved.ps newsave.png"]
subprocess.Popen(args)
#or this call(args)
os.system("start newsave.png")
The cmd window says that newsave.png is an invalid parameter. (The error message being: Invalid parameter - newsave.png in the cmd window, which then closes instantly)
Having the everything seperated by a comma in args has also not helped. os.getcwd() returns the current work directory as well, so I know I'm in the right dir. The error happens when the subprocess is called.
Make each command-line argument a separate element of args. Also, use subprocess.call to ensure that the convert function has completed before you call os.system("start newsave.png"):
args = ["convert", "saved.ps", "newsave.png"]
rc = subprocess.call(args)
if rc != 0:
print "rc =", rc
In the end I had to add shell=True in order for the conversion to work properly.
args = ["convert", "saved.ps", "newsave.png"]
subprocess.call(args, shell=True)
Thanks to Warren for the help.
I'm trying to execute Google's cpplint.py on a group of my files and collect the results to one log file. However, I have not managed to beat the subprocess module. My current code is here:
import os, subprocess
rootdir = "C:/users/me/Documents/dev/"
srcdir = "project/src/"
with open(rootdir+srcdir+"log.txt", mode='w', encoding='utf-8') as logfile:
for subdir, dirs, files in os.walk(rootdir+srcdir):
for file in files:
if file.endswith(".h") or file.endswith(".cpp"):
filewithpath=os.path.join(subdir, file)
cmd=['c:/Python27/python.exe','C:/users/me/Documents/dev/cpplint.py','--filter=-whitespace,-legal,-build/include,-build/header_guard/', filewithpath]
output = subprocess.check_output(cmd)
logfile.write(output.decode('ascii'))
Trying to run the above code throws an error:
File "C:\Python32\lib\site.py", line 159
file=sys.stderr)
^ SyntaxError: invalid syntax Traceback (most recent call last): File "C:\Users\me\Documents\dev\project\src\verifier.py", line 19, in <module>
output = subprocess.check_output(cmd) File "C:\Python32\lib\subprocess.py", line 511, in check_output
raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['c:/Python27/python.exe', 'C:/users/me/Documents/dev/cpplint.py', '--filter=-whitespace,-legal,-build/include,-build/header_guard/', 'C:/users/me/Documents/dev/project/src/aboutdialog.cpp']' returned non-zero exit status 1
If I substitute the cmd with something simpler like:
cmd=['C:/WinAVR-20100110/bin/avr-gcc.exe','--version']
Then the script works as expected.
I have also tried to use a single command string instead of a list of strings as cmd, but the result is the same.
When debugging the code, I copied the list-of-strings-turned-into-the-command-line-command from the debugger and ran it in the Windows command line, and the command ran as expected.
The Python interpreter running my script is Python 3.2.
Any tips are greatly appreciated.
Looks like cpplint.py is simply exiting with a non-zero return code - which it might do, for instance, if it finds errors or "lint" in the source files it is checking.
See the documentation for subprocess.check_output. Note that if the command executed returns a non-zero exit code then a subprocess.CalledProcessError is raised.
You could work around it by watching for CalledProcessError, e.g.
try:
output = subprocess.check_output(cmd)
except subprocess.CalledProcessError as e:
# ack! cpplint.py failed... report an error to the user?
EDIT:
The SyntaxError seems to be the key here, and is probably caused by C:\Python32\lib being in your PYTHONPATH (either explicitly, or, this could happen if it is your current working directory).
The Python interpreter (since about 1.5.2-ish) automatically runs import site when started. So, when this is the case, and your script goes to execute:
c:/Python27/python.exe C:/users/me/Documents/dev/cpplint.py ...
then the Python 2.7 interpreter will find C:\Python32\lib\site.py first, and try to load that, instead of the one (presumably) at C:\Python27\lib\site.py. The issue is that Python 3's site.py contains syntax incompatible with Python 2, so the process launched by subprocess.check_output is failing with a SyntaxError before it even gets a chance to run cpplint, which propagates the CalledProcessError.
Solution? Make sure Python2 gets a bonafide Python2 "PYTHONPATH", and likewise for Python3! In other words, make sure C:\Python32\lib is not in the PYTHONPATH search path when running the Python2 interpreter.
One way to do this in your case is to set an explicit environment when launching the process, e.g.:
python2_env = {"PYTHONPATH": "path/to/python2/stuff:..."}
output = subprocess.check_output(cmd, env=python2_env)
I would request you to run this first
pipe = subprocess.Popen([cmd, options],stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout, stderr = pipe.communicate()
You will get to know what exactly is the error behind it, since CalledProcessError is raised only if the exit code was non-zero.
I did it by replacing the def main() with the following (I editet the errorfunction too to get a proper csv-file):
errorlog = sys.stderr
sys.stderr = open("errorlog.csv","w")
sys.stderr.write("File;Line;Message;Category;Confidence\n")
for filename in filenames:
ProcessFile(filename, _cpplint_state.verbose_level)
_cpplint_state.PrintErrorCounts()
sys.exit(_cpplint_state.error_count > 0)
sys.stdout = errorlog
sys.stderr.close()
I am trying to execute a command inside a Python script:
import subprocess
output_process =
subprocess.Popen("javac -cp C:\Users\MyUsername\Desktop\htmlcleaner-2.2.jar Scrapping_lastfm.java",
shell=True, stdout=subprocess.PIPE)
But I am getting an error package org.htmlcleaner does not exist.
If I run the javac command independently, it executes fine..
My current working directry is C:\Users\MyUsername.
The error is not raised by python but by the java subprocess. Most likely the java machine is not finding some libraries, and that refines the problem to a PATH configuration problem, most likely
the variable CLASSPATH has not been set in the environment. to solve :
import shlex
JAVA_COMMAND=r"javac -cp C:\\Users\\MyUsername\\Desktop\\htmlcleaner-2.2.jar Scrapping_lastfm.java"
cmdline = shlex.split(JAVA_COMMAND)
output_process = subprocess.Popen(cmdline,shell=True, stdout=subprocess.PIPE, env={'CLASSPATH':'/path/to/java/packages'})
Try
output_process = subprocess.Popen(["javac", "-cp",
"C:\Users\MyUsername\Desktop\htmlcleaner-2.2.jar", "Scrapping_lastfm.java"],
shell=True, stdout=subprocess.PIPE, env={'ENVIRONMENTAL': '/variables/here'})
with whatever java-related environmental variables you have when you run javac normally as items in the env dictionary. asgs suggests you need CLASSPATH.
You don't have to split the command up into a list I just did that to make it easier to see the whole thing.
Be aware, that you have to escape the backslash (\) in the string. Your example is fine, however if your username is not actually MyUsername but maybe „nerd“ or any other string forming a valid escape-sequence, the command will fail.
Also make sure that you don't have spaces in the filename (or use the split syntax in the other example).
So you might want to do:
output_process = subprocess.Popen(["javac", "-cp",
"C:\\Users\\MyUsername\\Desktop\\htmlcleaner-2.2.jar", "Scrapping_lastfm.java"],
shell=True, stdout=subprocess.PIPE)