I want to embed a python script in a Makefile
I built a python -c script (below), and it works well in my MacBook Hyper terminal:
% python -c $'from subprocess import getstatusoutput\noutput=getstatusoutput("open --background -a Docker")\nif int(output[0])>0:\n print("Docker desktop failed to launch: exit-code:{}".format(output[0]))'
For reasons I can't yet figure out, this seems to fail if I build a Makefile with it (note: a tab is four spaces... I used tab indentation in the Makefile).
all:
$(shell python -c $'from subprocess import getstatusoutput\noutput=getstatusoutput("open --background -a Docker")\nif int(output[0])>0:\n print("Docker desktop failed to launch: exit-code:{}".format(output[0]))')
Running the make all target...
% make all
/bin/sh: -c: line 0: syntax error near unexpected token `('
/bin/sh: -c: line 0: `python -c from subprocess import getstatusoutput\noutput=getstatusoutput("open --background -a Docker")\nif int(output[0])>0:\n print("Docker desktop failed to launch: exit-code:{}".format(output[0]))''
make: `all' is up to date.
%
I have been struggling with this for a while...
Can someone help explain why make all fails and the best way to fix this python -c command? My shell CLI python -c ... command successfully launches Docker desktop on my MacBook.
I understand there are non-python ways to solve this specific problem... I need a general python Makefile solution.
Using python -c in a Makefile is tricky because of Python's indentation requirements. One simple solution is to use SHELL=/bin/bash if you want to use a Bash "C-style" string:
SHELL=/bin/bash
all:
# python command must be wrapped in single quotes _and_ have double dollar sign in front
python -c $$'from subprocess import getstatusoutput\noutput=getstatusoutput("open --background -a Docker")\nif int(output[0])>0:\n print("Docker desktop failed to launch: exit-code:{}".format(output[0]))'
(Notice how the dollar sign needs to be doubled to escape it. And obviously, this restricts the portability of your Makefile to systems where Bash is available. The $'...' syntax lets you use escape codes like \n and \t within a string, and have them expanded to newline and tab, respectively. This construct specifically requires a leading dollar sign and single quotes around the string - merely '...' does someting slightly different, and $"..." does something entirely different.)
You could also define a make multi-line variable. But in this isolated case, Python is not playing any useful role anyway.
all:
open --background -a Docker
make will terminate with an error message if open fails; printing essentinally the same message from Python seems superfluous. If you want to proceed in spite of the error, you can do
all:
open --background -a Docker || \
echo "Docker desktop failed to launch: exit-code: $$?"
... though I assume failing to fail (sic) from the Python script was just a mistake.
I found a fairly simple way to embed a multiline python script in a Makefile...
Save this as Makefile...
define MULTILINE_PYTHON_SCRIPT
###########################################
# Start multiline python string here...
###########################################
from subprocess import getstatusoutput as gso
print("Here we go:")
for hello_int in [1, 2, 3,]:
print(' Hello World %i' % hello_int)
retval, _ = gso("ls -la")
assert retval==0, "ls command execution failed"
###########################################
# End of multiline python string...
###########################################
endef
export MULTILINE_PYTHON_SCRIPT
EMBEDDED_PY := python -c "$$MULTILINE_PYTHON_SCRIPT"
.PHONY: nothing
nothing:
echo "raw makefile command"
.PHONY: test
test:
$(EMBEDDED_PY)
.PHONY: all
all:
open --background -a Docker
Testing the output:
% make nothing
echo "raw makefile command"
raw makefile command
%
% make test
python -c "$MULTILINE_PYTHON_SCRIPT"
Here we go:
Hello World 1
Hello World 2
Hello World 3
%
%
Related
The command
python -c "print('hello')"
runs the code inside the quotes successfully, both in Linux (bash) and Windows (cmd.exe).
But how to pass code with newlines, with python -c?
Example: both
python -c "for i in range(10): if i % 2 == 0: print('hello')"
python -c "for i in range(10):\n if i % 2 == 0:\n print('hello')"
fail.
Example use case: I need to send a SSH command (with paramiko) to execute a short Python code on a remote server, so I need to pass one command like
ssh.exec_command('python -c "..."').
You can use bash's $'foo' string syntax to get newlines:
python -c $'for i in range(10):\n if i % 2 == 0:\n print("hello")'
(I'm using single space indents here)
For windows, you really should be using powershell, which has `n as a newline:
python -c "for i in range(10):`n if i % 2 == 0:`n print('hello')"
In cmd.exe, it seems that you can use ^ to escape a newline, however I'm unable to test this currently so you should refer to this question's answers.
You can use a bash "heredoc" (also available ksh, and POSIX shells):
python <<EOF
import numpy as np
print(dir(np))
EOF
While not using -c it is worth mentioning that, if using Bash, you can pipe echoed code directly to python.
echo -e 'for i in range(10):\n if i % 2 == 0:\n print("hello")' | python
Following Aplet123's lead and using using single space indents.
One way could be with exec(), what make strings executable. It looks bad, but it works.
python -c "exec(\"for i in range(10):\n if i % 2 == 0:\n print('hello')\")"
I am trying to access a python function from the command line, and I would like to write such a command that will print the output in the terminal. The below doesn't work. What could I change?
python -c 'from laser import Laser; laser = Laser();l = laser.embed_sentences("hello", lang = "en").shape == (1, 1024); print(l)'
(base) ~ % python -c 'print("hello, world")'
hello, world
Printing works fine for me when running python through python -c. Are you sure your terminal isn't truncating your output by omitting the last (and in this case, only) line? You could try creating a single line file (no newline at the end) and then running cat [filename] (which is how I sometimes discover that my terminal is doing this)
-c cmd : program passed in as string (terminates option list)
That is the correct flag to be used. This must be a CLI config issue. Or the script is taking longer than you are expecting to run and it appears no output is generated.
Does python -c 'print("hello")' work?
I'm making a python script for Travis CI.
.travis.yml
...
script:
- support/travis-build.py
...
The python file travis-build.py is something like this:
#!/usr/bin/env python
from subprocess import check_call
...
check_call(r"mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder", shell=True)
...
When Travis building achieves that line, I'm getting an error:
/bin/sh: 1: Syntax error: "(" unexpected
I just tried a lot of different forms to write it, but I get the same result. Any idea?
Thanks in advance!
Edit
My current directory layout:
- my_project/final_folder/
- cmake-3.0.2-Darwin64-universal/
- fileA
- fileB
- fileC
I'm trying with this command to move all the current files fileA, fileB and fileC, excluding my_project and cmake-3.0.2-Darwin64-universal folders into ./my_project/final_folder. If I execute this command on Linux shell, I get my aim but not through check_call() command.
Note: I can't move the files one by one, because there are many others
I don't know which shell Travis are using by default because I don't specify it, I only know that if I write the command in my .travis.yml:
.travis.yml
...
script:
# Here is the previous Travis code
- mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder
...
It works. But If I use the script, it fails.
I found this command from the following issue:
How to use 'mv' command to move files except those in a specific directory?
You're using the bash feature extglob, to try to exclude the files that you're specifying. You'll need to enable it in order to have it exclude the two entries you're specifying.
The python subprocess module explicitly uses /bin/sh when you use shell=True, which doesn't enable the use of bash features like this by default (it's a compliance thing to make it more like original sh).
If you want to get bash to interpret the command; you have to pass it to bash explicitly, for example using:
subprocess.check_call(["bash", "-O", "extglob", "-c", "mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder"])
I would not choose to do the job in this manner, though.
Let me try again: in which shell do you expect your syntax !(...) to work? Is it bash? Is it ksh? I have never used it, and a quick search for a corresponding bash feature led nowhere. I suspect your syntax is just wrong, which is what the error message is telling you. In that case, your problem is entirely independent form python and the subprocess module.
If a special shell you have on your system supports this syntax, you need to make sure that Python is using the same shell when invoking your command. It tells you which shell it has been using: /bin/sh. This is usually just a link to the real shell executable. Does it point to the same shell you have tested your command in?
Edit: the SO solution you referenced contains the solution in the comments:
Tip: Note however that using this pattern relies on extglob. You can
enable it using shopt -s extglob (If you want extended globs to be
turned on by default you can add shopt -s extglob to .bashrc)
Just to demonstrate that different shells might deal with your syntax in different ways, first using bash:
$ !(uname)
-bash: !: event not found
And then, using /bin/dash:
$ !(uname)
Linux
The argument to a subprocess.something method must be a list of command line arguments. Use e.g. shlex.split() to make the string be split into correct command line arguments:
import shlex, subprocess
subprocess.check_call( shlex.split("mv !(...)") )
EDIT:
So, the goal is to move files/directories, with the exemption of some file(s)/directory(ies). By playing around with bash, I could get it to work like this:
mv `ls | grep -v -e '\(exclusion1\|exclusion2\)'` my_project
So in your situation that would be:
mv `ls | grep -v -e '\(myproject\|cmake-3.0.2-Darwin64-universal\)'` my_project
This could go into the subprocess.check_call(..., shell=True) and it should do what you expect it to do.
Hey guys I'm trying to integrate some PY into my shell script and running across
the following error, I though quote should have quoted my variable but it looks
like it's not doing what I expected, can someone help me troubleshoot this?
#!/bin/bash
host='user#localhost'
path='/home/user/file'
python -c "return subprocess.call(['ssh', '$host', 'test -e ' + pipes.quote($path)]) == 0"
File "<string>", line 1
return subprocess.call(['ssh', "user#localhost", 'test -e ' + pipes.quote(/home/jdaniel/sent)]) == 0
^
SyntaxError: invalid syntax
python -c "return subprocess.call(['ssh', '$host', 'test -e ' + pipes.quote(\"$path\")]) == 0"
I would assume
as an aside .. why are you not just calling ssh from the bash? what benefit are you getting by using python here in this fashion? and do you not need to use import subprocess when you use the -c flag?
I would choose to do this whole program either in python or bash ... but mixing them like this feels slightly silly (especially given what your python code does)
You'll need to change this
pipes.quote($path)
to
pipes.quote('$path')
as pipes.quote() is expecting a string
I'd say Its better to use shell instead of python
#!/bin/bash
host='user#localhost'
path='/home/user/file'
ssh -q $host "test -e $path"
Assume I have a file at http://mysite.com/myscript.sh that contains:
#!/bin/bash
echo "Hello $1"
From the command line, I can execute my script (without downloading it) using the following command:
bash <(curl -s http://mysite.com/myscript.sh) World
Now, instead of executing the above command from the command line, I want to execute it from a python script. I tried doing the following:
import os
os.system('bash <(curl -s http://mysite.com/myscript.sh) World')
...but I get the following error:
sh: -c: line 0: syntax error near unexpected token `('
How do I make this execute correctly in python?
Evidently, os.system runs its command through /bin/sh, which usually causes whichever shell it's linked to to drop to a compatibility mode that doesn't include the <(...) construction. You can get around it by either storing the result in a temporary file or using another level of shell. Ugly, but it works.
os.system('bash -c "bash <(curl -s http://mysite.com/myscript.sh) World"')
There is a libcurl for python so you don't have to go the way around to command line behaviour. Here's the function list that should really do it - have never run remote scripts myself though. If you need installing the python binding, the instructions are here.
import curl