We Keep Coding

Python - subprocess with creationflags (0x00000008) - How to get output after process ends? (Windows)

I wrote a Python script to run a terminal command that belongs to a 3rd party program.
import subprocess
DETACHED_PROCESS = 0x00000008
command = 'my cmd command'
process = subprocess.Popen(
args=command,
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
encoding="utf-8",
creationflags=DETACHED_PROCESS
)
code = process.wait()
print(process.stdout.readlines())
# Output: []
This script basically runs the command successfully. However, I'd like to print the output but process.stdout.readlines() prints an empty list.
I need to run the subprocess with creationflags due to 3rd party program's terminal command.
I've also tried creationflags=subprocess.CREATE_NEW_CONSOLE. It works but process takes too long because of 3rd party program's terminal command.
Is there a way to print the output of subprocess by using creationflags=0x00000008 ?
By the way, I can use subprocess.run etc to run the command also but I'm wondering if I can fix this.
Thank you for your time!
Edit:
I'm sorry I forgot to say I can get output if i write "dir" etc. as a command. However, I can't get any output when I write a command such as: command = '"program.exe" test'

I'm not sure that this works for your specific case, but I use subprocess.check_output when I need to capture subprocess output.
import subprocess
DETACHED_PROCESS = 0x00000008
command = 'command'
process = subprocess.check_output(
args=command,
shell=True,
stderr=subprocess.STDOUT,
encoding="utf-8",
creationflags=DETACHED_PROCESS
)
print(process)
This just returns a string of stdout.

call a command line from script using python, Ubuntu OS

I am facing difficulties calling a command line from my script.I run the script but I don't get any result. Through this command line in my script I want to run a tool which produces a folder that has the output files for each line.The inputpath is already defined. Can you please help me?
for line in inputFile:
cmd = 'python3 CRISPRcasIdentifier.py -f %s/%s.fasta -o %s/%s.csv -st dna -co %s/'%(inputpath,line.strip(),outputfolder,line.strip(),outputfolder)
os.system(cmd)

You really want to use the Python standard library module subprocess. Using functions from that module, you can construct you command line as a list of strings, and each would be processed as one file name, option or value. This bypasses the shell's escaping, and eliminates the need to massage you script arguments before calling.
Besides, your code would not work, because the body block of the for statement is not indented. Python would simply not accept this code (could be you pasted into the questiong without the proper indentations).

as mentioned before, executing command vias: os.system(command) is not recomended. please use subprocess (read in python docs about this modulesubprocess_module_docs). see the code here:
for command in input_file:
p = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
# use this if you want to communicate with child process
# p = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate()
# --- do the rest

I usually do like this for static command
from subprocess import check_output
def sh(command):
return check_output(command, shell=True, universal_newlines=True)
output = sh('echo hello world | sed s/h/H/')
BUT THIS IS NOT SAFE!!! It's vunerable to shell injection you should do
from subprocess import check_output
from shlex import split
def sh(command):
return check_output(split(command), universal_newlines=True)
output = sh('echo hello world')
The difference is subtle but important. shell=True will create a new shell, so pipes, etc will work. I use this when I have a big command line with pipes and that is static, I mean, it do not depend on user input. This is because this variant is vunerable to shell injection, a user can input something; rm -rf / and it will run.
The second variant only accepts one command, it will not spawn a shell, instead it will run the command directly. So no pipes and such shell things will work, and is safer.
universal_newlines=True is for getting output as string instead of bytes. Use it for text output, if you need binary output just ommit it. The default is false.
So here is the full example
from subprocess import check_output
from shlex import split
def sh(command):
return check_output(split(command), universal_newlines=True)
for line in inputFile:
cmd = 'python3 CRISPRcasIdentifier.py -f %s/%s.fasta -o %s/%s.csv -st dna -co %s/'%(inputpath,line.strip(),outputfolder,line.strip(),outputfolder)
sh(cmd)
Ps: I didn't test this

How to use linux command in python with python functions (sys.argv)

In my python script, I need to use 'awk' but I want to pass file using the sys.argv.
My current code is like this:
import sys
import os
cmd="awk '/regex/ {print}' sys.argv[1] | sed 's/old/new/g'"
x=os.popen(cmd).read()
Now the problem is that 'sys.argv' is a python thing but cmd variable is using a linux command.
So my question is - Is there any way to include sys.argv in my linux command?

You really don't need Awk or sed for this. Python can do these things natively, elegantly, flexibly, robustly, and naturally.
import sys
import re
r = re.compile(r'regex')
s = re.compile(r'old')
with open(sys.argv[1]) as input:
for line in input:
if r.search(line):
print(s.sub('new', line))
If you really genuinely want to use subprocesses for something, simply use Python's general string interpolation functions where you need to insert the value of a Python variable into a string.
import subprocess
import sys
import shlex
result = subprocess.run(
"""awk '/regex/ {print}' {} |
sed 's/old/new/g'""".format(shlex.quote(sys.argv[1])),
stdout=subprocess.PIPE,
shell=True, check=True)
print(subprocess.stdout)
But really, don't do this. If you really can't avoid a subprocess, keep it as simple as possible (avoid shell=True and peel off all the parts which can be done in Python).

Just try like this
cmd="awk '/regex/ {print}' " + str(sys.argv[1]) + " | sed 's/old/new/g'"
x=os.popen(cmd).read()

Your best choice is to implement your logic as pure Python logic, as described in the first part of the answer by #tripleee. Your second best choice is to keep the external tools, but eliminate the need for a shell in invoking them and connecting them together.
See the Python documentation section Replacing Shell Pipelines.
import sys
from subprocess import Popen, PIPE
p1 = Popen(['awk', '/regex/ {print}'], stdin=open(sys.argv[1]), stdout=PIPE)
p2 = Popen(['sed', 's/old/new/g'], stdin=p1.stdout, stdout=PIPE)
x = p2.communicate()[0]
Your third best choice is to keep the shell, but pass the data out-of-band from the code:
p = subprocess.run([
"""awk '/regex/ {print}' <"$1" | sed 's/old/new/'""", # code to run
'_', # $0 in context of that code
sys.argv[1] # $1 in context of that code
], shell=True, check=True, stdout=subprocess.PIPE)
print(p.stdout)

Using Popen in Python in ssh to Run "ls -lh INPUTS" Bash Commands

INPUTS is the variable I gave for the absolute path of a directory of possible input files. I want to check their status before going through my pipeline. So I tried:
import subprocess
import argparse
INPUTS = '/home/username/WinterResearch/Inputs'
status = subprocess.Popen(['ls', '-lh', INPUTS], shell=True, stdout=subprocess.PIPE)
stdout = status.communicate()
status.stdout.close()
I have also tried the often used
from shlx import split
import subprocess
import argparse
cmd = 'ls -lh INPUTS'
status = subprocess.Popen(cmd.split(), shell=True, stdout=subprocess.PIPE)
and
cmd = "ls -lh 'INPUTS'"
I do not receive an error code. The process simply does not output anything to the terminal window. I am not sure why the python script simply skips over this instead of stating there is an error. I do receive an error when I include close_fds=True that states int cannot use communicate(). So how can I receive an output from some ls -lh INPUTS equivalent using subprocess.Popen()?

You don't see any output because you're not printing to console stdout — it's saved into a variable (named "stdout"). Popen is overkill for this task anyway since you aren't piping the command to another. check_output should work fine with subprocess for this purpose:
import subprocess
subprocess.check_output("ls -lh {0}".format(INPUTS), shell=True)
subprocess.check_output(args, *, stdin=None, stderr=None, shell=False,
universal_newlines=False)
Run command with arguments and return its
output as a byte string.
METHOD WITH LESSER SECURITY RISK: (see warnings plastered throughout this page)
EDIT: Using communicate() can avoid the potential
shell=True security risk:
output = subprocess.Popen(["ls", "-lh", INPUTS]).communicate()[0]
print(output)

From your first snippet:
stdout = status.communicate()
status.stdout.close()
Nothing is being printed here. You may need to change it to the following (or your preferred form/format)
stdout = status.communicate()
print stdout
status.stdout.close()

python - subprocess.Popen() synax errors

I am trying to mirror the following shell command using subprocess.Popen():
echo "SELECT employeeid FROM Users WHERE samaccountname=${1};" | bsqldb -S mdw2k8sqlp02.dow.com -D PhoneBookClient -U PortManUser -P plum45\\torts -q
It currently looks like:
stdout = subprocess.Popen(["echo", "\"SELECT", "employeeid", "FROM", "Users", "WHERE", "samaccountname=${1};\"", "|", "bsqldb", "arg1etc"], stdout=subprocess.PIPE)
for line in stdout.stdout.readlines():
print line
It seems that this is wrong, it returns the following standard out:
"SELECT employeeid FROM Users WHERE samaccountname=${1};" | bsqldb arg1etc
Does anyone know where my syntax for subprocess.Popen() has gone wrong?

The problem is that you're trying to run a shell command without the shell. What happens is that you're passing all of those strings—including "|" and everything after is—as arguments to the echo command.
Just add shell=True to your call to fix that.
However, you almost definitely want to pass the command line as a string, instead of trying to guess at the list that will be joined back up into the string to pass to the shell.
Or, even better, don't use the shell, and instead pipe within Python. The docs have a nice section about Replacing shell pipeline (and all kinds of other things) with subprocess code.
But in your case, the thing you're trying to pipe is just echo, which is quite silly, since you already have exactly what echo would return, and can just feed it as the input to the second program.
Also, I'm not sure what you expect that ${1} to get filled in with. Presumably you're porting a shell script that took some arguments on the command line; your Python script may have the same thing in sys.argv[1], but without knowing more about what you're doing, that's little more than a guess.

The analog of echo some string | command arg1 arg2 shell pipeline in Python is:
from subprocess import Popen, PIPE
p = Popen(["command", "arg1", "arg2"], stdin=PIPE)
p.communicate("some string")
In your case, you could write it as:
import shlex
import sys
from subprocess import Popen, PIPE
cmd = shlex.split("bsqldb -S mdw2k8sqlp02.dow.com -D PhoneBookClient "
"-U PortManUser -P plum45\\torts -q")
sql = """SELECT employeeid FROM Users
WHERE samaccountname={name};""".format(name=sql_escape(sys.argv[1]))
p = Popen(cmd, stdin=PIPE)
p.communicate(input=sql)
sys.exit(p.returncode)