I am using pexpect.run to execute a command. See below:
cmd = "grep -L killed /dir/dumps/*MAC-66.log"
output = pexpect.run(cmd)
When I run this, output equals to:
grep: /dir/dumps/*MAC-66.log: No such file or directory
But when I run the same command in my shell, it works, everytime. I don't see the problem. Any help is appreciated! Does pexpect.run require the command to be split in some fancy way?
Your shell is interpreting the glob, pexpect is not. You could either use python's glob.glob() function to evaluate the glob yourself, or run it through your shell, for example:
cmd = "bash -c 'grep -L killed /dir/dumps/*MAC-66.log'"
Also, if all you're after is output of this command, you ought to check out the subprocess module.
Related
I am learning concurrency with Python 3 Standard Library, 2nd Edition. Is there a way to get the subprocess module to use variables in my .bashrc when I set shell=True?
I tried adding echo "something" in my .bashrc and ran the following but I couldn't see something in the output but I could see $HOME.
import subprocess
completed = subprocess.run('echo $HOME', shell=True)
print('returncode:', completed.returncode)
There are a few ways to go about this. But first, you need to execute Bash. Right now, you're executing /bin/sh, which could be Bash, but you don't want to assume. To fix that, pass in executable argument:
import subprocess
completed = subprocess.run('echo $HOME',
shell=True,
executable='/usr/bin/bash'
)
print('returncode:', completed.returncode)
Now we have that out of the way, there are two ways to ensure your .bashrc is sourced. Since Bash is not running interactively, you need to either,
(A) source it yourself:
import subprocess
completed = subprocess.run('source ~/.bashrc && echo "$HOME"',
shell=True,
executable='/usr/bin/bash'
)
print('returncode:', completed.returncode)
or,
(B) Use BASH_ENV:
import subprocess
completed = subprocess.run('echo "$HOME"',
env={'BASH_ENV': '~/.bashrc'},
shell=True,
executable='/usr/bin/bash'
)
print('returncode:', completed.returncode)
BASH_ENV is a variable used by Bash. If Bash sees the variable, and after expanding the variable (i.e. tilda expansion), the path is to a file that exists, it sources the variable before executing the rest of the script.
I need to use the wget in a Python script with the subprocess.call function, but it seems the "wget" command cannot be identified by the bash subprocess opened by python.
I have added the environment variable (the path where wget is):
export PATH=/usr/local/bin:$PATH
to the ~/.bashrc file and the ~/.bash_profile file on my mac and guaranteed to have sourced them.
And the python script looks like:
import subprocess as sp
cmd = 'wget'
process = sp.Popen(cmd ,stdout=sp.PIPE, stdin=sp.PIPE,
stderr=sp.PIPE, shell=True ,executable='/bin/bash')
(stdoutdata, stderrdata) = process.communicate()
print stdoutdata, stderrdata
The expected output should be like
wget: missing URL
Usage: wget [OPTION]... [URL]...
But the result is always
/bin/bash: wget: command not found
Interestingly I can get the help output if I type in wget directly in a bash terminal, but it never works in the python script. How could it be?
PS:
If I change the command to
cmd = '/usr/local/bin/wget'
then it works. So I am sure I got wget installed.
You can pass an env= argument to the subprocess functions.
import os
myenv = os.environ.copy
myenv['PATH'] = '/usr/local/bin:' + myenv['PATH']
subprocess.run(..., env=myenv)
However, you probably want to avoid running a shell at all, and instead augment the PATH that Python uses to find the binary to run in the subprocess call.
import subprocess as sp
import os
os.environ['PATH'] = '/usr/local/bin:' + os.environ['PATH']
cmd = 'wget'
# use run instead of Popen
# don't needlessly use a shell
# and thus put [cmd] as a list
process = sp.run([cmd], stdout=sp.PIPE, stdin=sp.PIPE,
stderr=sp.PIPE,
universal_newlines=True)
print(process.stdout, process.stderr)
Running Bash commands in Python explains the changes I made in more detail.
However, there is no good reason to use an external utility for this; Python requests does pretty everything wget does, often more naturally and with more control over what exactly it does.
I want to execute a command in terminal from a Python script.
./driver.exe bondville.dat
This command is getting printed in the terminal, but it is failing to execute.
Here are my steps:
echo = "echo"
command="./driver.exe"+" "+"bondville.dat"
os.system(echo + " " + command)
It should execute the command, but it's just printing it on terminal. When feeding the same thing manually it's executing. How do I do this from a script?
The echo terminal command echoes its arguments, so printing the command to the terminal is the expected result.
Are you typing echo driver.exe bondville.dat and is it running your driver.exe program?
If not, then you need to get rid of the echo in the last line of your code:
os.system(command)
You can use the subprocess.check_call module to run the command, you don't need to echo to run the command:
from subprocess import check_call
check_call(["./driver.exe", "bondville.dat"])
Which is equivalent to running ./driver.exe bondville.dat from bash.
If you wanted to get the output you would use check_outout:
from subprocess import check_output
out = check_output(["./driver.exe", "bondville.dat"])
In your own code you are basically echoing the string command not actually running the command i.e echo "./driver.exe bondville.dat" which would output ./driver.exe bondville.dat in your shell.
Try this:
import subprocess
subprocess.call("./driver.exe bondville.dat")
In Python I'm using subprocess to call gsutil copy and move commands, but am currently unable to select multiple extensions.
The same gsutil command works at the terminal, but not in python:
cmd_gsutil = "sudo gsutil -m mv gs://xyz-ms-media-upload/*.{mp4,jpg} gs://xyz-ms-media-upload/temp/"
p = subprocess.Popen(cmd_gsutil, shell=True, stderr=subprocess.PIPE)
output, err = p.communicate()
If say there are four filetypes to move but the bucket is empty, the returning gsutil error from terminal is:
4 files/objects could not be transferred.
Whereas the error returned when run through subprocess is:
1 files/objects could not be transferred.
So clearly subprocess is mucking up the command somehow...
I could always inefficiently repeat the command for each of the filetypes, but would prefer to get to the bottom of this!
It seems, /bin/sh (the default shell) doesn't support {mp4,jpg} syntax.
Pass executable='/bin/bash', to run it as a bash command instead.
You could also run the command without the shell e.g., using glob or fnmatch modules to get the filenames to construct the gsutil command. Note: you should pass the command as a list in this case instead.
I'd like Popen to execute:
grep -i --line-buffered "grave" data/*.txt
When run from the shell, this gives me the wanted result. If I start, in the very same directory where I test grep, a python repl and follow the instruction from the docs, I obtain what should be the proper argument list to feed Popen with:
['grep', '-i', '--line-buffered', 'grave', 'data/*.txt']
The result of p = subprocess.Popen(args) is
grep: data/*.txt: No such file or directory
and if I try p = subprocess.Popen(args, shell=True), I get:
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
Any help on how to perform the wanted process? I'm on MacOS Lion.
If you type * in bash the shell expands it to the files in the given directory before executing the command. Python's Popen does no such thing, so what you're doing when you call Popen like that is telling grep there is a file called *.txt in the data directory, instead of all the .txt files in the data directory. That file doesn't exist and you get the expected error.
To solve this you can tell python to run the command through the shell by passing shell=True to Popen:
subprocess.Popen('grep -i --line-buffered grave data/*.txt', shell=True)
Which gets translated to:
subprocess.Popen(['/bin/sh', '-c', 'grep -i --line-buffered "grave" data/*.txt'])
As explained in the documentation of Popen.
You have to use a string instead of a list here, because you want to execute /bin/sh -c "grep -i --line-buffered "grave" data/*.txt" (N.B. quotes around the command, making it a single argument to sh). If you use a list this command is run: /bin/sh -c grep -i --line-buffered "grave" data/*.txt, which gives you the output of simply running grep.
The problem is that shell makes file globbing for you: data/*.txt
You will need to do it youself, for example, by using glob module.
import glob
cmd_line = ['grep', '-i', '--line-buffered', 'grave'] + glob.glob('data/*.txt')