We Keep Coding

In python, how to pass an array argument to powershell script

I have a PowerShell script, which has two parameters, the first one is a string, the second one is an array of string.
I would like to call this PowerShell script from my python code. How to pass the array type parameter to PowerShell?
If I write something like this:
subprocess.run(['powershell.exe', 'script.ps1', 'arg1', '#("str1", "str2")'])
Powershell think '#("str1", "str2")' is a string, not an array.
Edit
I found a workaround
subprocess.run(['powershell.exe', 'script.ps1 arg1 #("str1", "str2")'])
It doesn't look beautiful, but works. and in this way, I can't use -File after powershell.exe

Your original command does work as written (except that you must use .\script.ps1 rather than script.ps1, unless the script is in the system's path), as does the second one you added later, because it implicitly uses the PowerShell CLI's -Command parameter rather than its
-File parameter.
In short:
Passing arrays is fundamentally only supported with -Command, which interprets the subsequent arguments as PowerShell code, where the usual PowerShell syntax applies.
With -File, by contrast, all arguments after the target-script argument are passed verbatim, as strings, so there is no concept of an array.
I suggest using the following approach, for increased robustness and conceptual clarity:
subprocess.run(['powershell.exe', '-noprofile', '-c', '.\script.ps1 arg1 #("str1", "str2")'])
Note: You can omit #(...) around the array elements - #() is never needed for array literals in PowerShell.
Note:
-noprofile ensures that PowerShell doesn't load the $PROFILE file(s), which avoids potential slow-downs and side effects.
-c (-Command) makes it explicit that you're passing PowerShell code rather than a script file with literal arguments (-File)
Do note that -Command arguments are subject to additional interpretation by PowerShell, so if you pass, say, a token $foo$ you intend to be a literal, PowerShell will expand it to just $ (if no $foo variable is defined), because it expands $foo as a variable reference; passing `$foo`$ (backtick-escaping) prevents that.
Note the .\ before script.ps1: Since you're using -Command you cannot execute a script by file name only (unless the script happens to be located in a directory listed in $env:PATH); as from inside PowerShell, executing scripts from the current directory requires .\ for security reasons; by contrast, file-name-only invocation does work with -File.
The script file as well as its arguments are passed as a single argument, which reflects how PowerShell will process the command.
-Command is the default in Windows PowerShell, but no longer in PowerShell Core (pwsh.exe), which defaults to -File; it is generally a good idea to explicitly use -Command (-c) or -File (-f) to make it obvious how PowerShell will interpret the arguments.
How subprocess.run() builds the command line and how PowerShell parses it:
Your original Python command passes #("str1", "str2") as an individual argument to subprocess.run():
subprocess.run(['powershell.exe', '.\script.ps1', 'arg1', '#("str1", "str2")'])
This results in the following command line executed behind the scenes:
powershell.exe .\script.ps1 arg1 "#(\"str1\", \"str2\")"
Note how only #("str1", "str2") is double-quoted, and how the embedded " chars. are escaped as \".
As an aside: PowerShell's CLI (arguments passed to powershell.exe) uses the customary \-escaping of literal " chars.; inside PowerShell, however, it is ` (backtick) that serves as the escape character.
Your second command combines the script.ps1 and #("str1", "str2") into a single argument:
subprocess.run(['powershell.exe', '.\script.ps1 arg1 #("str1", "str2")'])
This results in the following command line:
powershell.exe ".\script.ps1 arg1 #(\"str1\", \"str2\")"
Note how the single argument passed is double-quoted as a whole.
Generally, subprocess.run() automatically encloses a given argument in "..." (double quotes) if it contains spaces.
Independently, it escapes embedded (literal) " chars. as \".
Even though these command lines are obviously different, PowerShell's (implied) -Command logic processes them the same, because it uses the following algorithm:
First, enclosing double quotes around each argument, if present, are removed.
The resulting strings, if there are multiple, are concatenated with spaces.
The resulting single string is then executed as PowerShell code.
If you apply this algorithm to either of the above command lines, PowerShell ends up executing the same code, namely:
.\script.ps1 arg1 #("str1", "str2")

Lets say your python array is arr
try to do this:
subprocess.run(['powershell.exe', 'script.ps1', 'arg1', '\"{}\"'.format(','.join(arr))])
To send array in powershell script you can send it as "item1,item2,item3"
and the function str.join allow you to get this format easly
If this doesn't work, i would try to edit the script to use the $args argument in the powershell script to change the way you using your arguments

You can use single quotes on the command line - e.g. #('str1', 'str2') or escape the double quotes with backslashes - e.g. #(\"str1\", \"str2\")
For example with this script:
script.ps1
param( [string[]] $s )
write-host $s.GetType().FullName
write-host $s.Length
write-host ($s | fl * | out-string)
You can call it from a command prompt like this:
C:\> powershell.exe .\script.ps1 #('str1', 'str2')
System.String[]
2
str1
str2
or like this:
C:> powershell.exe .\script.ps1 #(\"str1\", \"str2\")
System.String[]
2
str1
str2
You might need to apply some python escape characters to get the desired result in your code though.

Python subprocess.run shell kwarg behavior

There has been a lot of questions regarding shell keyword argument. But I still don't really get it especially if we use the sequence argument instead of the string.
My understanding is that if shell=False, the subprocess module will run the executable in args[0] and pass the rest as arguments to the executable. But if we run it with shell=True, it will be ran as something like "sh -c {}".format(format_escaping(args)).
But why does this happen?
# Ran in OSX
subprocess.run(["touch", "12; touch 34"]) # successfuly make the file '12; touch 34'
subprocess.run(["touch", "56; touch 78"], shell=True) # does not work:
# usage:
# touch [-A [-][[hh]mm]SS] [-acfhm] [-r file] [-t [[CC]YY]MMDDhhmm[.SS]] file ...
# CompletedProcess(args=['touch', '123; touch 456'], returncode=1)
What actually happen in subprocess.run(["touch", "56; touch 79"], shell=True)?

I think with shell=True the supbrocess runs first only the first parameter which is touch therefore you become the help mesage from the command , try it like this instead:
subprocess.run("touch 56; touch 78", shell=True)

The behavior when you pass in a list with shell=True is platform-dependent. On Unix-like platforms, Python simply passes in the first argument of the list. On Windows, it happens to work, though it probably shouldn't.

scp with Python3 subprocess [duplicate]

When using subprocess.Popen(args, shell=True) to run "gcc --version" (just as an example), on Windows we get this:
>>> from subprocess import Popen
>>> Popen(['gcc', '--version'], shell=True)
gcc (GCC) 3.4.5 (mingw-vista special r3) ...
So it's nicely printing out the version as I expect. But on Linux we get this:
>>> from subprocess import Popen
>>> Popen(['gcc', '--version'], shell=True)
gcc: no input files
Because gcc hasn't received the --version option.
The docs don't specify exactly what should happen to the args under Windows, but it does say, on Unix, "If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional shell arguments." IMHO the Windows way is better, because it allows you to treat Popen(arglist) calls the same as Popen(arglist, shell=True) ones.
Why the difference between Windows and Linux here?

Actually on Windows, it does use cmd.exe when shell=True - it prepends cmd.exe /c (it actually looks up the COMSPEC environment variable but defaults to cmd.exe if not present) to the shell arguments. (On Windows 95/98 it uses the intermediate w9xpopen program to actually launch the command).
So the strange implementation is actually the UNIX one, which does the following (where each space separates a different argument):
/bin/sh -c gcc --version
It looks like the correct implementation (at least on Linux) would be:
/bin/sh -c "gcc --version" gcc --version
Since this would set the command string from the quoted parameters, and pass the other parameters successfully.
From the sh man page section for -c:
Read commands from the command_string operand instead of from the standard input. Special parameter 0 will be set from the command_name operand and the positional parameters ($1, $2, etc.) set from the remaining argument operands.
This patch seems to fairly simply do the trick:
--- subprocess.py.orig 2009-04-19 04:43:42.000000000 +0200
+++ subprocess.py 2009-08-10 13:08:48.000000000 +0200
## -990,7 +990,7 ##
args = list(args)
if shell:
- args = ["/bin/sh", "-c"] + args
+ args = ["/bin/sh", "-c"] + [" ".join(args)] + args
if executable is None:
executable = args[0]

From the subprocess.py source:
On UNIX, with shell=True: If args is a string, it specifies the
command string to execute through the shell. If args is a sequence,
the first item specifies the command string, and any additional items
will be treated as additional shell arguments.
On Windows: the Popen class uses CreateProcess() to execute the child
program, which operates on strings. If args is a sequence, it will be
converted to a string using the list2cmdline method. Please note that
not all MS Windows applications interpret the command line the same
way: The list2cmdline is designed for applications using the same
rules as the MS C runtime.
That doesn't answer why, just clarifies that you are seeing the expected behavior.
The "why" is probably that on UNIX-like systems, command arguments are actually passed through to applications (using the exec* family of calls) as an array of strings. In other words, the calling process decides what goes into EACH command line argument. Whereas when you tell it to use a shell, the calling process actually only gets the chance to pass a single command line argument to the shell to execute: The entire command line that you want executed, executable name and arguments, as a single string.
But on Windows, the entire command line (according to the above documentation) is passed as a single string to the child process. If you look at the CreateProcess API documentation, you will notice that it expects all of the command line arguments to be concatenated together into a big string (hence the call to list2cmdline).
Plus there is the fact that on UNIX-like systems there actually is a shell that can do useful things, so I suspect that the other reason for the difference is that on Windows, shell=True does nothing, which is why it is working the way you are seeing. The only way to make the two systems act identically would be for it to simply drop all of the command line arguments when shell=True on Windows.

The reason for the UNIX behaviour of shell=True is to do with quoting. When we write a shell command, it will be split at spaces, so we have to quote some arguments:
cp "My File" "New Location"
This leads to problems when our arguments contain quotes, which requires escaping:
grep -r "\"hello\"" .
Sometimes we can get awful situations where \ must be escaped too!
Of course, the real problem is that we're trying to use one string to specify multiple strings. When calling system commands, most programming languages avoid this by allowing us to send multiple strings in the first place, hence:
Popen(['cp', 'My File', 'New Location'])
Popen(['grep', '-r', '"hello"'])
Sometimes it can be nice to run "raw" shell commands; for example, if we're copy-pasting something from a shell script or a Web site, and we don't want to convert all of the horrible escaping manually. That's why the shell=True option exists:
Popen(['cp "My File" "New Location"'], shell=True)
Popen(['grep -r "\"hello\"" .'], shell=True)
I'm not familiar with Windows so I don't know how or why it behaves differently.

How to use subprocess to fire and forget in python 2.7

I need to execute a command line in the bakground in python 2.7. I need to fire and forget.
Here is the command:
cmd = "/usr/local/bin/fab -H %s aws_bootstrap initial_chef_run:%s,%s,%s -w" % (...)
How do I use the subproccess module?
e.g. is it
subprocess.call([cmd])
or
subprocess.call(["/usr/local/bin/fab", "-H %s aws_bootstrap initial_chef_run:%s,%s,%s -w"])
I dont get how to use the list. Or is every element of the list what would be a white space.
Thanks

each thing that would be seperated by whitespace is a seperate entity of the list
subprocess.call is blocking however
subprocess.popen is non-blocking
cmd = ["/usr/local/bin/fab", "-H",var1,"aws_bootstrap initial_chef_run:%s,%s,%s"%(var2,var3,var4), "-w"]
subprocess.popen(cmd) # dopnt wait just keep going
#or
subprocess.call(cmd) # wait until the command returns
you may however alternatively pass the command as one big string
cmd = "/usr/local/bin/fab -H %s aws_bootstrap initial_chef_run:%s,%s,%s -w" % (...)
subprocess.call(cmd)
in general this method(passing a single string) is frowned upon for some reason that has never been explained sufficiently to me

I used this recently to fire a perl script, like so:
var = "C:\Users\user\Desktop"
retcode = subprocess.call(["perl", '.\hgncDL.pl',var])

Working code
Define hParam and runParams in following code and you're good to go:
hParam = 'hParam'
runParams = (a,b,c)
args = ('/usr/local/bin/fab', '-H', hParam, 'aws_bootstrap', 'initial_chef_run:%s,%s,%s' % runParams, '-w')
subprocess.Popen(args)
Details
How do I use <any python module> module?
https://docs.python.org is a good starting point.
In particular, docs for subprocess module available here.
I can't provide direct links for each case later in this answer due to restriction imposed by low reputation. Each time I will be referring to 'docs', look for a section in docs on the module.
I need to execute a command line in the background in python 2.7. I need to fire and forget
Consider subprocess.Popen(args). Note capital 'P'.
See docs for more details.
subprocess.call(args) works in similar way, but it would block until the command completes. As stated in docs:
Run the command described by args. Wait for command to complete, then return the returncode attribute.
How to use the sequence form of args parameter?
This is covered in "Frequently used arguments" section of docs:
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).
Also, passing an args in a string form has its limitation:
If passing a single string, either shell must be True or else the string must simply name the program to be executed without specifying any arguments.
Despite mentioned limitation, subprocess.Popen('cmd.exe /?') works for me. Win7, Python 2.7.8 64bit.
HTH, cheers.

Advantage of list over string in subprocess methods

What are the advantages of using list over string in subprocess methods? The ones I understand so far:
Security if input comes from external sources
Portability over different operating systems
Are there any others?
In my particular case, I'm using subprocess library to run tests on a software. Input does not come from external source. Tests are run only on Linux. Therefore, I see no benefit of lists over strings.

On POSIX, list and string arguments have different meaning and are used in different contexts.
You use a string argument and shell=True to run a shell command e.g.:
from subprocess import check_output
output = check_output("dmesg | grep hda", shell=True)
A list argument is used to run a command without the shell e.g.:
from subprocess import check_call
check_call(["ls", "-l"])
One exception is that call("ls") is equivalent to call(["ls"]) (a command with no arguments).
You should use a list argument with shell=False (default) except in those cases when you need the shell so the string argument is used.
It is almost always an error to use a list argument and shell=True (the arguments are interpreted as arguments to the shell itself instead of the command in this case). Don't use it.
If your question: what are the advantages of shell=False and hence the list argument over a string argument:
you don't need to escape the arguments, no shell interpolation such as word splitting, parameter expansion, command substitution occurs: what you see is what you get
support for arguments with spaces
support for arguments with special characters such as quotes, dollar sign, etc
it is clear where arguments boundaries are. They are explicitely separated.
it is clear what program is executed: it is the first item in the list
an argument that is populated from an untrusted source won't be able to execute arbitrary commands
why run a superfluous shell process unless you need it
Sometimes, it might be more convenient/readable to specify an argument as a string in the source code; shlex.split() could be used to convert it to a list:
import shlex
from subprocess import check_call
cmd = shlex.split('/bin/vikings -input eggs.txt -output "spam spam.txt" '
'''-cmd "echo '$MONEY'"''')
check_call(cmd)
See the docs.
On Windows, the arguments are interpreted differently. The native format is a string and the passed list is converted to a string using subprocess.list2cmdline() function that may not work for all Windows programs. shell=True is only necessary to run builtin shell commands.
If list2cmdline() creates a correct command line for your executable (different programs may use different rules for interpreting the command line) then a list argument could be used for portability and to avoid escaping separate arguments manually.