In pwsh call the following:
Write-Host '{"drop_attr": "name"}'
Result ok:
{"drop_attr": "name"}
Now do the same via pwsh:
pwsh -Command Write-Host '{"drop_attr": "name"}'
Result is missing quotation marks and square brackets?
drop_attr: name
Update:
PowerShell 7.3.0 mostly fixed the problem, with selective exceptions on Windows, and it seems that in some version after 7.3.1 the fix will require opt-in - see this answer for details.
For cross-version, cross-edition code, the Native module discussed at the bottom may still be of interest.
Unfortunately, PowerShell's handling of passing arguments with embedded " chars. to external programs - which includes PowerShell's own CLI (pwsh) - is fundamentally broken (and always has been), up to at least PowerShell 7.2.x:
You need to manually \-escape " instances embedded in your arguments in order for them to be correctly passed through to external programs (which happens to be PowerShell in this case as well):
# Note: The embedded '' sequences are the normal and expected
# way to escape ' chars. inside a PowerShell '...' string.
# What is *unexpected* is the need to escape " as \"
# even though " can normally be used *as-is* inside a '...' string.
pwsh -Command ' ''{\"drop_attr\": \"name\"}'' '
Note that I'm assuming your intent is to pass a JSON string, hence the inner '' ... '' quoting (escaped single quotes), which ensures that pwsh ultimately sees a single-quoted string ('...'). (No need for an explicit output command; PowerShell implicitly prints command and expression output).
Another way to demonstrate this on Windows is via the standard choice.exe utility, repurposed to simply print its /m (message) argument (followed by verbatim [Y,N]?Y):
# This *should* preserve the ", but doesn't as of v7.2
PS> choice /d Y /t 0 /m '{"drop_attr": "name"}'
{drop_attr: name} [Y,N]?Y # !! " were REMOVED
# Only the extra \-escaping preserves the "
PS> choice /d Y /t 0 /m '{\"drop_attr\": \"name\"}'
{"drop_attr": "name"} [Y,N]?Y # OK
Note that from inside PowerShell, you can avoid the need for \-escaping, if you call pwsh with a script block ({ ... }) - but that only works when calling PowerShell itself, not other external programs:
# NOTE: Works from PowerShell only.
pwsh -Command { '{"drop_attr": "name"}' }
Background info on PowerShell's broken handling of arguments with embedded " in external-program calls, as of PowerShell 7.2.1:
This GitHub docs issue contains background information.
GitHub issue #1995 discusses the problem and the details of the broken behavior as well as manual workarounds are summarized in this comment; the state of the discussion as of PowerShell [Core] 7 seems to be:
A fix is being considered as an experimental feature, which may become an official feature, in v7.3 at the earliest. Whether it will become a regular feature - i.e whether the default behavior will be fixed or whether the fix will require opt-in or even if the feature will become official at all - remains to be seen.
Fixing the default behavior would substantially break backward compatibility; as of this writing, this has never been allowed, but a discussion as to whether to allow breaking changes in the future and how to manage them has begun: see GitHub issue #13129.
See GitHub PR #14692 for the relevant experimental feature, which, however, as of this writing is missing vital accommodations for batch files and msiexec-style executables on Windows - see GitHub issue #15143.
In the meantime, you can use the PSv3+ ie helper function from the Native module (in PSv5+, install with Install-Module Native from the PowerShell Gallery), which internally compensates for all broken behavior and allows passing arguments as expected; e.g.,
ie pwsh -Command ' ''{"drop_attr": "name"}'' ' would then work properly.
Another way. Are you in Windows or Unix?
pwsh -c "[pscustomobject]#{drop_attr='name'} | convertto-json -compress"
{"drop_attr":"name"}
Another way is to use "encoded commands".
> $cmd1 = "Write-Host '{ ""description"": ""Test program"" }'"
> pwsh -encoded ([Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($cmd1)))
{ "description": "Test program" }
Related
I was trying to pass some arguments via PyCharm when I noticed that it's behaving differently that my console. When I pass arguments with no space in between all works fine, but when my arguments contains spaces inside it the behavior diverge.
def main():
"""
Main function
"""
for i, arg in enumerate(sys.argv):
print('Arg#{}: {}'.format(i, arg))
If I run the same function:
python3 argumnents_tester.py 'argument 1' argument2
Run in PyCharm:
Arg#0: /home/gorfanidis/PycharmProjects/test1/argparse_test.py
Arg#1: 'argument
Arg#2: 1'
Arg#3: argument2
Run in Console:
Arg#0: argparse_test.py
Arg#1: argument 1
Arg#2: argument2
So, PyCharm tends to ignore quotes altogether and splits the arguments using the spaces regardless of any quotes. Also, arguments with quotes are treated differently than the same arguments without quotes.
Question is why it this happening and at a practical level how am I suppose to pass an argument that contains spaces using PyCharm for example?
I am using Ubuntu 16.04 by the way.
What you are complaining about is a shell issue. The shell applies its convention of single quotes to parse arguments. Actually I find the PyCharm behaviour as understandable and consistent; if no shell is involved, nobody does the job you expect.
If you insist on running that from pycharm, I'd suggest another passing method (e.g. via file) or to apply some unquoting mechanism as urllib.parse.unquote.
See also here for a dated but still correct description of command line parameters in general, and specifically:
The ANSI C standard does not specify what constitutes a command-line argument, because operatring systems vary considerably on this point. However, the most common convention is as follows:
Each command-line argument must be separated by a space or a tab character. Commas, semicolons, and the like are not considered separators.
Disclaimer: What turns out to be the correct solution to #Eypros question was a passing suspicion that it is a matter of difference between how PyCharm implements its arguments parsing in its command-line run & how actual system console/shell behaves, which was also pointed out by #guidot in his answer. I provide more thoughts in Postscript below.
To circumvent the behaviour exhibited by PyCharm in how it interprets the argument 1 part in python3 argumnents_tester.py 'argument 1' argument2 (as 2 arguments), use a different type of quote in the code, a double quote " for str.format(), and a single quote ' for the argument in the run command.
PS:
While this seems like a simple workaround, I do think in case of any possibility the code would be executed in any other system, one should choose to adhere to most common/widely accepted standard behaviour of system shells (bash, zsh, sh, any *nix flavors) in interpreting the argument passing instead of PyCharm's implementation. This way the code will be much more portable and users don't have to figure out a different way to pass/feed an argument.
As consequence of that, I offer no guarantee this will work aside from this specific way the code formulated & in configuration similar to #Eypros system.
(Background info) Well, an original comment from #cryptonome seemed to work for me but since the provided answer by the same user is not exactly the same I summarized the solution that worked for me.
PyCharm for some reason treats different single (') and double quotes (") when parsing arguments. Programming in python this may or may not seem natural. Anyway double quotes (") seem to work exactly the same both in console and PyCharm. So, when arguments are passed using double quotes (") the same behavior is expected.
Single quotes should be avoided in PyCharm but seem to work in console (at least in mine: bash in Ubunut 16.04) because the argument splitting occurs in spaces and not quote boundaries.
I can check the presence of a file or folder using OS library very easily.
The following two links have described that
directoryExistance fileExistance
I am attempting to use the subprocess library to do the same
and, I tried a couple of approaches already
1- status = subprocess.call(['test','-e',<path>]), which is always returning 1, no matter what I pass in path.
2- Using getstatusoutput,
/bin/sh: 1: : Permission denied
status, result = subprocess.getstatusoutput([<path>])
print(status)
print(result)
which is working fine because status variable returns 126 if the file/folder exist and 127 when the file/folder doesn't exist. Also the result variable contains message but the "result" variable contains the message : Permission denied
But the second solution looks like a hack to me. Is their a better way, of doing this ?
The test command is a shell builtin, and on many platforms doesn't exist as an independent command you can run.
If you use shell=True to use the shell to run this command, you should pass in a single string, not a list of tokens.
status = subprocess.call("test -e '{}'".format(path), shell=True)
This will produce a malformed command if path contains any single quotes; try path.replace("'", r"\'") if you want to be completely correct and robust, or use one of the existing quoting functions to properly escape any shell metacharacters in the command you pass in.
The subprocess library now offers a function run() which is slightly less unwieldy than the old legacy call() function; if backwards compatibility is not important, you should probably switch to that... or, as several commenters have already implored you, not use subprocess for this task when portable, lightweight native Python solutions are available.
As pointed in the comments section
status = subprocess.call(['test','-e',<path>])
can be made to work with a shell expansion if we use "shell=True"
Although using os.path might be much more efficient anyways.
In my python script I need to execute a command over SSH that also takes a heredoc as an argument. The command calls an interactive script that can be also called as follows:
dbscontrol << EOI
HELP
QUIT
EOI
I also found this Q&A that describes how to do it using subprocess but I really like pexpect.pxssh convenience.
Code example would be greatly appreciated
I don't have pexpect handy to test my answer to your question, but I have a suggestion that should work and, if not, may at least get you closer.
Consider this command:
$ ssh oak 'ftp << EOF
lpwd
quit
EOF'
Local directory: /home/jklowden
$
What is happening? The entire quoted string is passed as a single argument to ssh, where it is "executed" on the remote. While ssh isn't explicit about what that means, exactly, we know what execv(2) does: if execve(2) fails to execute its passed arguments, the execv function will invoke /bin/sh with the same arguments (in this case, our quoted string). The shell then evaluates the quoted string as separate arguments, detects the HereDoc redirection, and executes per usual.
Using that information, and taking a quick look at the pexpect.pxssh documentation, it looks like you want:
s = pxssh.pxssh()
...
s.sendline('ftp << EOF\nlpwd\nquit\nEOF')
If that doesn't work, something is munging your data. Five minutes with strace(1) will tell you what happened to it, and you can start pointing fingers. ;-)
HTH.
I need to execute a command line in the bakground in python 2.7. I need to fire and forget.
Here is the command:
cmd = "/usr/local/bin/fab -H %s aws_bootstrap initial_chef_run:%s,%s,%s -w" % (...)
How do I use the subproccess module?
e.g. is it
subprocess.call([cmd])
or
subprocess.call(["/usr/local/bin/fab", "-H %s aws_bootstrap initial_chef_run:%s,%s,%s -w"])
I dont get how to use the list. Or is every element of the list what would be a white space.
Thanks
each thing that would be seperated by whitespace is a seperate entity of the list
subprocess.call is blocking however
subprocess.popen is non-blocking
cmd = ["/usr/local/bin/fab", "-H",var1,"aws_bootstrap initial_chef_run:%s,%s,%s"%(var2,var3,var4), "-w"]
subprocess.popen(cmd) # dopnt wait just keep going
#or
subprocess.call(cmd) # wait until the command returns
you may however alternatively pass the command as one big string
cmd = "/usr/local/bin/fab -H %s aws_bootstrap initial_chef_run:%s,%s,%s -w" % (...)
subprocess.call(cmd)
in general this method(passing a single string) is frowned upon for some reason that has never been explained sufficiently to me
I used this recently to fire a perl script, like so:
var = "C:\Users\user\Desktop"
retcode = subprocess.call(["perl", '.\hgncDL.pl',var])
Working code
Define hParam and runParams in following code and you're good to go:
hParam = 'hParam'
runParams = (a,b,c)
args = ('/usr/local/bin/fab', '-H', hParam, 'aws_bootstrap', 'initial_chef_run:%s,%s,%s' % runParams, '-w')
subprocess.Popen(args)
Details
How do I use <any python module> module?
https://docs.python.org is a good starting point.
In particular, docs for subprocess module available here.
I can't provide direct links for each case later in this answer due to restriction imposed by low reputation. Each time I will be referring to 'docs', look for a section in docs on the module.
I need to execute a command line in the background in python 2.7. I need to fire and forget
Consider subprocess.Popen(args). Note capital 'P'.
See docs for more details.
subprocess.call(args) works in similar way, but it would block until the command completes. As stated in docs:
Run the command described by args. Wait for command to complete, then return the returncode attribute.
How to use the sequence form of args parameter?
This is covered in "Frequently used arguments" section of docs:
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).
Also, passing an args in a string form has its limitation:
If passing a single string, either shell must be True or else the string must simply name the program to be executed without specifying any arguments.
Despite mentioned limitation, subprocess.Popen('cmd.exe /?') works for me. Win7, Python 2.7.8 64bit.
HTH, cheers.
How can I reverse the results of a shlex.split? That is, how can I obtain a quoted string that would "resemble that of a Unix shell", given a list of strings I wish quoted?
Update0
I've located a Python bug, and made corresponding feature requests here.
We now (3.3) have a shlex.quote function. It’s none other that pipes.quote moved and documented (code using pipes.quote will still work). See http://bugs.python.org/issue9723 for the whole discussion.
subprocess.list2cmdline is a private function that should not be used. It could however be moved to shlex and made officially public. See also http://bugs.python.org/issue1724822.
How about using pipes.quote?
import pipes
strings = ["ls", "/etc/services", "file with spaces"]
" ".join(pipes.quote(s) for s in strings)
# "ls /etc/services 'file with spaces'"
.
There is a feature request for adding shlex.join(), which would do exactly what you ask. As of now, there does not seem any progress on it, though, mostly as it would mostly just forward to shlex.quote(). In the bug report, a suggested implementation is mentioned:
' '.join(shlex.quote(x) for x in split_command)
See https://bugs.python.org/issue22454
It's shlex.join() in python 3.8
subprocess uses subprocess.list2cmdline(). It's not an official public API, but it's mentioned in the subprocess documentation and I think it's pretty safe to use. It's more sophisticated than pipes.open() (for better or worse).
While shlex.quote is available in Python 3.3 and shlex.join is available in Python 3.8, they will not always serve as a true "reversal" of shlex.split. Observe the following snippet:
import shlex
command = "cd /home && bash -c 'echo $HOME'"
print(shlex.split(command))
# ['cd', '/home', '&&', 'bash', '-c', 'echo $HOME']
print(shlex.join(shlex.split(command)))
# cd /home '&&' bash -c 'echo $HOME'
Notice that after splitting and then joining, the && token now has single quotes around it. If you tried running the command now, you'd get an error: cd: too many arguments
If you use subprocess.list2cmdline() as others have suggested, it works nicer with bash operators like &&:
import subprocess
print(subprocess.list2cmdline(shlex.split(command)))
# cd /home && bash -c "echo $HOME"
However you may notice now that the quotes are now double instead of single. This results in $HOME being expanded by the shell rather than being printed verbatim as if you had used single quotes.
In conclusion, there is no 100% fool-proof way of undoing shlex.split, and you will have to choose the option that best suites your purpose and watch out for edge cases.