Hi all I am learning python and shell script and for GUI i am using wxpython.
I have said to make a automated tool which does some operation , deb creation is also one of that.
for deb creation , command is:
./myfile -u username
I have tried os.Popen, os.system, subprocess.Popen , subprocess.call.But no use everytime "-u" wont take effect, "-u " is must. I have tried by storing "-u" in variable and the passed it but still no use.
Please suggest me the exact way or where i am doing wrong
No error message but "myfile" output shows that "-u" has not given in command.
Code is:
1. cmd = ['./myfile', '-u', 'username']
Popen(cmd,shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
2. user = "-u"
name = "username"
sbprocess.call("./myfile %s %s" %(str(user),str(name)), shell=True)
same kind using "os" command also
The first code example in your question passes incorrect command because shell=True changes the meaning of the first parameter, from the subprocess docs:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
..[snip]..
If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
The second code example from you question should work if the following command works in a shell:
$ /bin/sh -c './myfile -u username'
To fix your command, you could omit possibly unnecessary shell=True and use check_call():
import subprocess
subprocess.check_call(["./myfile", "-u", "username"])
Try 'bash' instead of './'. If its bash script or else use respective. Its work only if your myfile is shell script.
subprocess.call("bash myfile -u %s" % str(name)), shell=True)
Related
I'm trying to use subprocess.Popen to run a check on Kafka consumer groups and log their state, but it doesn't appear to be waiting for all the commands to run. It isn't giving me any stdout, but its also returning an exit code of 0.
prompt = ["cd", "~/path/to/kafka_2.11-2.1.0;", "pwd;", "./bin/kafka-consumer-groups.sh",
"--bootstrap-server", "localhost:9092", "--describe", "--group", "groupname"]
response = subprocess.run(prompt, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True, check=True)
print(response)
Prints:
CompletedProcess(args=['cd', '~/path/to/kafka_2.11-2.1.0;', 'pwd;', './bin/kafka-consumer-groups.sh', '--bootstrap-server', 'localhost:9092', '--describe', '--group', 'groupname'], returncode=0, stdout=b'', stderr=b'')
The pwd command was to primarily test if it would return any kind of stout, it won't be kept.
I've looked through the docs for subprocess, and I haven't see anything that suggests that it is unable to capture multiple stdouts. Also, according to the logs, the CompletedProcess is returned in less than 10ms, while running cd ~path/to/kafka_2.11-2.1.0; pwd; ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group groupname takes about 10-15s on my machine.
Please note that I'm using python3.5.2
Your error is more fundamental than you think. Your code runs
sh -c 'cd'
with $0 set to the directory, $1 set to pwd;, etc; so very far from what you want. (Just cd simply switches to your home directory; then the shell exits, without doing anything with all those arguments you passed in, and Python continues back in whichever directory was current before you launched the subprocess.)
Generally, pass a single string as the first argument with shell=True, and a list of strings when you don't have a shell.
subprocess.run(r"cd foo; pwd; use shell commands to your heart\'s content; run as many processes as you like as subprocesses of your shell subprocess", shell=True)
subprocess.run(['/bin/echo', 'one', 'single', 'process', 'with', 'arguments])
I found my error. cd doesn't work with subprocess, but subprocess offers the cwd named argument that accepts the path you need to run arguments. The reason it was returning a CompletedProcess so quickly is that it was successfully changing directories and then exiting the subprocess.
Sorry, I wasn't thinking about cd being the culprit until I came across this question which answered my problem too.
I am new to the subprocess module, and wondering why the first subprocess failed while the second one worked. I am on py3.7 and macOS.
>>> from subprocess import PIPE, Popen, STDOUT
>>> Popen(['touch', '/Users/me/fail.txt'], stdout=PIPE, stderr=STDOUT, shell=True)
>>> Popen(['touch /Users/me/ok.txt'], stdout=PIPE, stderr=STDOUT, shell=True)
According to the docs:
The shell argument (which defaults to False) specifies whether to use the shell as the program to execute. If shell is True, it is recommended to pass args as a string rather than as a sequence.
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
So in the first case the 2nd element of the list is passed as an argument to /bin/sh itself, not the touch command. So you are basically running:
user#name ~$ touch
Which produces the following error:
touch: missing file operand
Try 'touch --help' for more information.
And if you read the stdout of your first command, you will find the same:
>>> Popen(['touch', '/Users/me/fail.txt'], stdout=PIPE, stderr=STDOUT, shell=True).stdout.read()
b"touch: missing file operand\nTry 'touch --help' for more information.\n"
So while shell=True, it is better to pass string.
In subprocess.run which is a high-level function, you need to pass the arguments as a list but for Popen, which is a low-level function needs a direct command hence the first one failed but the second one worked.
I am facing difficulties calling a command line from my script.I run the script but I don't get any result. Through this command line in my script I want to run a tool which produces a folder that has the output files for each line.The inputpath is already defined. Can you please help me?
for line in inputFile:
cmd = 'python3 CRISPRcasIdentifier.py -f %s/%s.fasta -o %s/%s.csv -st dna -co %s/'%(inputpath,line.strip(),outputfolder,line.strip(),outputfolder)
os.system(cmd)
You really want to use the Python standard library module subprocess. Using functions from that module, you can construct you command line as a list of strings, and each would be processed as one file name, option or value. This bypasses the shell's escaping, and eliminates the need to massage you script arguments before calling.
Besides, your code would not work, because the body block of the for statement is not indented. Python would simply not accept this code (could be you pasted into the questiong without the proper indentations).
as mentioned before, executing command vias: os.system(command) is not recomended. please use subprocess (read in python docs about this modulesubprocess_module_docs). see the code here:
for command in input_file:
p = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
# use this if you want to communicate with child process
# p = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate()
# --- do the rest
I usually do like this for static command
from subprocess import check_output
def sh(command):
return check_output(command, shell=True, universal_newlines=True)
output = sh('echo hello world | sed s/h/H/')
BUT THIS IS NOT SAFE!!! It's vunerable to shell injection you should do
from subprocess import check_output
from shlex import split
def sh(command):
return check_output(split(command), universal_newlines=True)
output = sh('echo hello world')
The difference is subtle but important. shell=True will create a new shell, so pipes, etc will work. I use this when I have a big command line with pipes and that is static, I mean, it do not depend on user input. This is because this variant is vunerable to shell injection, a user can input something; rm -rf / and it will run.
The second variant only accepts one command, it will not spawn a shell, instead it will run the command directly. So no pipes and such shell things will work, and is safer.
universal_newlines=True is for getting output as string instead of bytes. Use it for text output, if you need binary output just ommit it. The default is false.
So here is the full example
from subprocess import check_output
from shlex import split
def sh(command):
return check_output(split(command), universal_newlines=True)
for line in inputFile:
cmd = 'python3 CRISPRcasIdentifier.py -f %s/%s.fasta -o %s/%s.csv -st dna -co %s/'%(inputpath,line.strip(),outputfolder,line.strip(),outputfolder)
sh(cmd)
Ps: I didn't test this
I would like to run a subprocess from Python. Inside the command string are several subcommands with backticks:
subprocess = subprocess.Popen(["echo", "COMMAND [`date +%%s`] SCHEDULE_HOST_DOWNTIME;%s;`date +%%s`;`date -d 'now + %d sec' +%%s`;1;;;%s;Downtime comment" % (hostname, 300, username)], stdout=subprocess.PIPE)
Though the date commands in the backticks are not executed. The stdout of this command is:
COMMAND [`date +%s`] SCHEDULE_HOST_DOWNTIME;example.com;`date +%s`;`date -d 'now + 300 sec' +%s`;1;;;my-username;Downtime comment
I also tried to use $(date +%s) instead of backticks and explicitly sent it to bash via subprocess.Popen(["/bin/bash", "-c", "echo", "..."], with the same result.
How can this be solved? I know I can of course use Pythons datetime module in this specific case. But I want to know why this is not working and how to solve it without tearing apart the command. While I'm here able to run the timestamp calculation easily in Python, in other cases it might be more complicated, where I would be forced to run several subprocesses which gets quickly very ugly.
Backticks are a shell syntax feature, but you are not running your command in a shell. The subprocess module runs the command directly.
Provide one string rather than a list, and set shell=True if the shell needs to process the command as one entry:
subprocess = subprocess.Popen(
'echo "COMMAND [`date +%%s`] SCHEDULE_HOST_DOWNTIME;%s;`date +%%s`;`date -d \'now + %d sec\' +%%s`;1;;;%s;Downtime comment"' % (hostname, 300, username),
stdout=subprocess.PIPE
shell=True)
I am trying to mirror the following shell command using subprocess.Popen():
echo "SELECT employeeid FROM Users WHERE samaccountname=${1};" | bsqldb -S mdw2k8sqlp02.dow.com -D PhoneBookClient -U PortManUser -P plum45\\torts -q
It currently looks like:
stdout = subprocess.Popen(["echo", "\"SELECT", "employeeid", "FROM", "Users", "WHERE", "samaccountname=${1};\"", "|", "bsqldb", "arg1etc"], stdout=subprocess.PIPE)
for line in stdout.stdout.readlines():
print line
It seems that this is wrong, it returns the following standard out:
"SELECT employeeid FROM Users WHERE samaccountname=${1};" | bsqldb arg1etc
Does anyone know where my syntax for subprocess.Popen() has gone wrong?
The problem is that you're trying to run a shell command without the shell. What happens is that you're passing all of those strings—including "|" and everything after is—as arguments to the echo command.
Just add shell=True to your call to fix that.
However, you almost definitely want to pass the command line as a string, instead of trying to guess at the list that will be joined back up into the string to pass to the shell.
Or, even better, don't use the shell, and instead pipe within Python. The docs have a nice section about Replacing shell pipeline (and all kinds of other things) with subprocess code.
But in your case, the thing you're trying to pipe is just echo, which is quite silly, since you already have exactly what echo would return, and can just feed it as the input to the second program.
Also, I'm not sure what you expect that ${1} to get filled in with. Presumably you're porting a shell script that took some arguments on the command line; your Python script may have the same thing in sys.argv[1], but without knowing more about what you're doing, that's little more than a guess.
The analog of echo some string | command arg1 arg2 shell pipeline in Python is:
from subprocess import Popen, PIPE
p = Popen(["command", "arg1", "arg2"], stdin=PIPE)
p.communicate("some string")
In your case, you could write it as:
import shlex
import sys
from subprocess import Popen, PIPE
cmd = shlex.split("bsqldb -S mdw2k8sqlp02.dow.com -D PhoneBookClient "
"-U PortManUser -P plum45\\torts -q")
sql = """SELECT employeeid FROM Users
WHERE samaccountname={name};""".format(name=sql_escape(sys.argv[1]))
p = Popen(cmd, stdin=PIPE)
p.communicate(input=sql)
sys.exit(p.returncode)