We Keep Coding

How to apply string formatting to a bash command (incorporated into Python script via subprocess)?

I would like to add a bash command to my Python script, which linearises a FASTA sequence file while leaving sequence separation intact (hence the specific choice of command). Below is the command, with the example input file of "inputfile.txt":
awk '/^>/ {printf("\n%s\n",$0);next; } { printf("%s",$0);} END {printf("\n");}' < inputfile.txt
The aim is to allow the user to specify the file which is to be modified in the command line, for example:
$ python3 program.py inputfile.txt
I have tried to use string formatting (i.e. %s) in conjunction with sys.argv in order to achieve this. However, I have tried many different locations of " and ', and still cannot get this to work and accept a user input from the command line here.
(The command contains escapes such as \n and so I have tried to counteract this by adding additional backslashes, as well as additional % for the existing %s in the command.)
import sys
import subprocess
path = sys.argv[1]
holder = subprocess.Popen("""awk '/^>/ {printf("\\n%%s\\n",$0);next; } { printf("%%s",$0);} END {printf("\\n");}' < %s""" % path , shell=True, stdout=subprocess.PIPE).stdout.read()
print(holder)
I would very much appreciate any help with identifying the syntax error here, or suggestions for how I could add this user input.

TL;DR: Don't shell out to awk! Just use Python. But let's go step by step...
Your instinct of using triple quotes here is good, then at least you don't need to escape both single and double quotes, that you need in your shell string.
The next useful device you can use is raw strings, using r'...' or r"..." or r"""...""". Raw strings don't expand backslash escapes, so in that case you can leave the \ns intact.
Last is the %s, which you need to escape if you use the % operator, but here I'm going to suggest that instead of using the shell to redirect input, just use Python's subprocess to send stdin from the file! Much simpler and you end up with no substitution.
I'll also recommend that you use subprocess.check_output() instead of Popen(). It's much simpler to use and it's a lot more robust, since it will check that the command exited successfully (with a zero exit status.)
Putting it all together (so far), you get:
with open(path) as inputfile:
holder = subprocess.check_output(
r"""awk '/^>/ {printf("\n%s\n",$0);next; } { printf("%s",$0);} END {printf("\n");}'""",
shell=True,
stdin=inputfile)
But here you can go one step further, since you don't really need a shell anymore, it's only being used to split the command line into two arguments, so just do this split in Python (it's almost always possible and easy to do this and it's a lot more robust since you don't have to deal with the shell's word splitting!)
with open(path) as inputfile:
holder = subprocess.check_output(
['awk', r'/^>/ {printf("\n%s\n",$0);next; } { printf("%s",$0);} END {printf("\n");}'],
stdin=inputfile)
The second string in the list is still a raw string, since you want to preserve the bacsklash escapes.
I could go into how you can do this without using printf() in awk, using print instead, which should get rid of both \ns and %s, but instead I'll tell you that it's much easier to do what you're doing in Python directly!
In fact, everything that awk (or sed, tr, cut, etc.) can do, Python can do better (or, at least, in a more readable and maintainable way.)
In the case of your particular code:
with open(path) as inputfile:
for line in inputfile:
if line.startswith('>'):
# Insert a blank line before this one.
print()
print(line)
if line.startswith('>'):
# Also insert a blank line after this.
print()
# And a blank line at the end.
print()
Isn't this better?
And you can put this into a function, into a module, and reuse it anywhere you'd like. It's easy to store the result in a string, save it into a variable if you like, much more flexible...
Anyways, if you still want to stick to shelling out, see my previous code, I think that's the best you can do while still shelling out, without significantly changing the external command.

How can I use ssh with print of python?

I would like to run ssh with print of python.
The followings are my test code.
import subprocess
# case1:
command_str = "\"print(\'test\')\""
# case 2:
# command_str = "\\\"print(\'test\')\\\""
ssh_command = ['ssh', 'USER_X#localhost', 'python', '-c']
ssh_command.append(command_str)
process = subprocess.run(ssh_command, stdout=subprocess.PIPE)
print(process.stdout)
case 1 and case 2 did not work.
The outputs are followings,
case 1:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `python -c print('test')'
b''
case 2:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `python -c \"print('test')\"'
b''
Please let me know how it works.

It should work with
command_str = "'print(\"test\")'"
or equivalently
command_str = '\'print("test")\''
Explanation
The outermost quotes and the escaping are for the local Python. So in either case, the local Python string will be 'print("test")'.
There is no quoting or escaping required for the local shell, as subcommand.run(...) won't invoke it unless shell=True is passed.
Thus the single quotes within the python string are for the remote shell (presumably bash or other sh-compatible shell). The argument passed to the remote Python is thus print("test"). (And the double quotes in there are to signify the string literal to print to the remote python.)
Can we do without escaping (without \)?
As there are three levels involved (local Python, remote shell, remote Python), I don't think so.
Can we do with a single type of quotes?
Yes, with a bit more escaping. Let's build this from behind (or inside-out).
We want to print
test
This needs to be escaped for the remote Python (to form a string literal instead of an identifier):
"test"
Call this with the print() function:
print("test")
Quite familiar so far.
Now we want to pass this as an argument to python -c on a sh-like shell. To protect the ( and ) to be interpreted by that, we quote the whole thing. For the already present " not to terminate the quotation, we escape them:
"print(\"test\")"
You can try this in a terminal:
$> echo "print(\"test\")"
print("test")
Perfect!
Now we have to represent the whole thing in (the local) Python. We wrap another layer of quotes around it, have to escape the four(!) existing quotation marks as well as the two backslashes:
"\"print(\\\"test\\\")\""
(Done. This can also be used as command_str.)
Can we do with only single quotes (') and escaping?
I don't know, but at least not as easily. Why? Because, other than to Python, double and single quotes aren't interchangeable to sh and bash: Within single quotes, these shells assume a raw string without escaping until the closing ' occurs.
My brain hurts!
If literally, go see a doctor. If figuratively, yeah, mine too. And your code's future readers (including yourself) will probably feel the same, when they try to untangle that quoting-escaping-forest.
But there's a painless alternative in our beloved Python standard library!
import shlex
command_str = shlex.quote('print("test")')
This is much easier to understand. The inner quotes (double quotes here, but doesn't really matter: shlex.quote("print('test')") works just as fine) are for the remote Python. The outer quotes are obviously for the local Python. And all the quoting and escaping beyond that for the remote shell is taken care of by this utility function.

The correct syntax for python 2 and 3 is:
python -c 'print("test")'

os.system() executes wrong command [duplicate]

I'm trying to run a .exe that requires some parameters by using system().
If there's a space in the .exe's path AND in the path of a file passed in parameters, I get the following error:
The filename, directory name, or volume label syntax is incorrect.
Here is the code that generates that error:
#include <stdlib.h>
#include <conio.h>
int main (){
system("\"C:\\Users\\Adam\\Desktop\\pdftotext\" -layout \"C:\\Users\\Adam\\Desktop\\week 4.pdf\"");
_getch();
}
If the "pdftotext"'s path doesn't use quotation marks (I need them because sometimes the directory will have spaces), everything works fine. Also, if I put what's in "system()" in a string and output it and I copy it in an actual command window, it works.
I thought that maybe I could chain some commands using something like this:
cd C:\Users\Adam\Desktop;
pdftotext -layout "week 4.pdf"
So I would already be in the correct directory, but I don't know how to use multiple commands in the same system() function.
Can anyone tell me why my command doesn't work or if the second way I thought about would work?
Edit: Looks like I needed an extra set of quotation marks because system() passes its arguments to cmd /k, so it needs to be in quotations. I found it here:
C++: How to make a my program open a .exe with optional args
so I'll vote to close as duplicate since the questions are pretty close even though we weren't getting the same error message, thanks!

system() runs command as cmd /C command. And here's citation from cmd doc:
If /C or /K is specified, then the remainder of the command line after
the switch is processed as a command line, where the following logic is
used to process quote (") characters:
1. If all of the following conditions are met, then quote characters
on the command line are preserved:
- no /S switch
- exactly two quote characters
- no special characters between the two quote characters,
where special is one of: &<>()#^|
- there are one or more whitespace characters between the
two quote characters
- the string between the two quote characters is the name
of an executable file.
2. Otherwise, old behavior is to see if the first character is
a quote character and if so, strip the leading character and
remove the last quote character on the command line, preserving
any text after the last quote character.
It seems that you are hitting case 2, and cmd thinks that the whole string C:\Users\Adam\Desktop\pdftotext" -layout "C:\Users\Adam\Desktop\week 4.pdf (i.e. without the first and the last quote) is the name of executable.
So the solution would be to wrap the whole command in extra quotes:
//system("\"D:\\test\" nospaces \"text with spaces\"");//gives same error as you're getting
system("\"\"D:\\test\" nospaces \"text with spaces\"\""); //ok, works
And this is very weird. I think it's also a good idea to add /S just to make sure it will always parse the string by the case 2:
system("cmd /S /C \"\"D:\\test\" nospaces \"text with spaces\"\""); //also works

I got here looking for an answer, and this is the code that I came up with (and I was this explicit for the benefit of next person maintaining my code):
std::stringstream ss;
std::string pathOfCommand;
std::string pathOfInputFile;
// some code to set values for paths
ss << "\""; // command opening quote
ss << "\"" << pathOfCommand << "\" "; // Quoted binary (could have spaces)
ss << "\"" << pathOfInputFile << "\""; // Quoted input (could have spaces)
ss << "\""; // command closing quote
system( ss.str().c_str() ); // Execute the command
and it solved all of my problems.

Good learning from here on the internals of System call.Same issue reproducible(of course) with C++ string, TCHARs etc.
One approach that always helped me is SetCurrentDirectory() call. I first set current path and then execute. This has worked for me so far. Any comments welcome.
-Sreejith. D. Menon

call awk from inside python generate error

Ive to run awk from the python. When I run the script from the terminal, gives the desired output but showing error when
executing from inside the python.
runAwk = '''awk '{printf $1}{for(i=2;i<=NF;i++)printf "|"$i}{printf "\n"}' final.txt'''
os.system(runAwk)
gives the error:
awk: line 1: runaway string constant " ...
when I surfed from the web, I found that awk can not be used with os module and there are not much contents. I am confused how to proceed ahead.

The \n in your runAwk string is being interpreted by Python as a literal newline character, rather than being passed through to awk as the two characters \ and n. If you use a raw string instead, by preceding the opening triple-quotes with an r:
runAwk = r'''awk '{printf $1}{for(i=2;i<=NF;i++)printf "|"$i}{printf "\n"}' final.txt'''
... then Python won't treat \n as meaning "newline", and awk will see the string you intended.

Passing double quotation marks to a command from Python using the subprocess module

I have a command line program I'm using from within a Python script to take care of stamping a build number onto an executable.
This is the command line program: http://www.elphin.com/downloads/stampver/
My problem is that the program takes double quotes (") in one of its arguments and the Python subprocess module I'm using to call this program keeps prepending a backslash onto the double quotes when executing the program. This causes the program to fail due to incorrect syntax.
It is expecting a command like: StampVer.exe -nopad -k -f"0.1.4491.0" <ExeFile>.exe
And instead Python is executing: StampVer.exe -nopad -k -f\"0.1.4491.0\" <ExeFile>.exe
I've tried a few things that I've found here on StackOverflow for similar sounding problems such as marking the string as raw or adding backslashes before the quotes in Python; which just results in triple backslashes on the command line instead of one, because Python then tries to escape the backslash as well as the double quote.
Should I be using something other than the subprocess module to accomplish this or do all these types of modules in Python follow the same rules for escaping characters? Is there something I can do to tell subprocess to strip escape characters or to not add them at all?
EDIT
This is how I'm calling subprocess from Python:
def set_file_version(version_number, filepath):
try:
file_version_arg = '-f"{0}"'.format(version_number)
cmd_line = ["StampVer.exe", "-nopad", "-k", file_version_arg, filepath]
subprocess.check_output(cmd_line)
except subprocess.CalledProcessError as e:
if e.returncode == 1:
pass
else:
raise e
StampVer then returns this:
error: Invalid -f parameter. Try -f"0.0.0.0" Use StampVer -? for help

try this script sub.py:
#! /usr/bin/python
import sys
from subprocess import check_output
if len(sys.argv) > 1:
print sys.argv
else:
print check_output((sys.argv[0], '-f"1234"'))
then run it:
./sub.py
it return what we gave:
['./sub.py', '-f"1234"']
So I guess check_output works just fine, the problem may came from how StampVer.exe handle parameter, you can try
file_version_arg = '-f{0}'.format(version_number)

My solution ended up being kind of a cop-out. Despite the documentation for StampVer showing the format above for the version number in all examples, it turns out you can just leave the quotes out all together and even space it out from the -f switch and it will still be accepted.
I'm going to call this my answer but I still think being able to pass quotes through subprocess is a worthwhile problem to figure out. If anyone has an answer that will actually solve the initial problem then please post it and I'll mark it instead.