We Keep Coding

How to shorten long python commands in console?

I have a python script that I start with
python /home/USER/path/tagging.py -i /missions/YYYY/YYYYMMDD/HHMM/jpg -o /missions2/YYYY/YYYYMMDD/HHMM/jpg -l /missions/YYYY/YYYYMMDD/HHMM/info/*.bin
I want to make a bash script for the bashrc that uses that exact command with:
tagging YYYY/YYYYMMDD/HHMM
because the only thing changing is YYYY/YYYYMMDD/HHMM
Any hint?

When you call a bash script with arguments, you can access those arguments inside your script as positional parameters $1, $2,.. where $1 is the first argument, $2 is the second argument, etc. ($0 is special, it stores the script name).
You can then create a bash script like this:
#!/bin/bash
python /home/USER/path/tagging.py -i /missions/${1}/jpg -o /missions2/${1}/jpg -l /missions/${1}/info/*.bin
where all the YYYY/YYYYMMDD/HHMM is replaced by ${1} that is expected to be passed to the script.
Then call it like this:
/path/to/tagging.sh YYYY/YYYYMMDD/HHMM

Python script returning empty string in Bash script only when arguments supplied

If I have a Python script that simply prints the first argument it is given
#!/usr/bin/env python
import sys
try:
print sys.argv[1]
except IndexError:
print "No args"
I can run it find and get the expected output. Now if I write a Bash script that simply runs the python script and echos its output.
#!/bin/bash
test="Test"
echo `python test.py`
Then the bash script will successfully echo "No args". However if I change line 3 to
echo `python test.py $test`
Then I will simply get an empty string as the output of the Bash script and I'm not sure why. I even get the empty string when I change it to
echo `python test.py "test"`

Changing the backticks to $() seems to have worked

You should try in your bash script file:
python test.py $test
It will print your $test variable content.
I call Python script without backquotes and echo. I work a lot with Bash/Python scripts combinations, and this work well for me.

Error in check_call() subprocess, executing 'mv' unix command: "Syntax error: '(' unexpected"

I'm making a python script for Travis CI.
.travis.yml
...
script:
- support/travis-build.py
...
The python file travis-build.py is something like this:
#!/usr/bin/env python
from subprocess import check_call
...
check_call(r"mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder", shell=True)
...
When Travis building achieves that line, I'm getting an error:
/bin/sh: 1: Syntax error: "(" unexpected
I just tried a lot of different forms to write it, but I get the same result. Any idea?
Thanks in advance!
Edit
My current directory layout:
- my_project/final_folder/
- cmake-3.0.2-Darwin64-universal/
- fileA
- fileB
- fileC
I'm trying with this command to move all the current files fileA, fileB and fileC, excluding my_project and cmake-3.0.2-Darwin64-universal folders into ./my_project/final_folder. If I execute this command on Linux shell, I get my aim but not through check_call() command.
Note: I can't move the files one by one, because there are many others
I don't know which shell Travis are using by default because I don't specify it, I only know that if I write the command in my .travis.yml:
.travis.yml
...
script:
# Here is the previous Travis code
- mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder
...
It works. But If I use the script, it fails.
I found this command from the following issue:
How to use 'mv' command to move files except those in a specific directory?

You're using the bash feature extglob, to try to exclude the files that you're specifying. You'll need to enable it in order to have it exclude the two entries you're specifying.
The python subprocess module explicitly uses /bin/sh when you use shell=True, which doesn't enable the use of bash features like this by default (it's a compliance thing to make it more like original sh).
If you want to get bash to interpret the command; you have to pass it to bash explicitly, for example using:
subprocess.check_call(["bash", "-O", "extglob", "-c", "mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder"])
I would not choose to do the job in this manner, though.

Let me try again: in which shell do you expect your syntax !(...) to work? Is it bash? Is it ksh? I have never used it, and a quick search for a corresponding bash feature led nowhere. I suspect your syntax is just wrong, which is what the error message is telling you. In that case, your problem is entirely independent form python and the subprocess module.
If a special shell you have on your system supports this syntax, you need to make sure that Python is using the same shell when invoking your command. It tells you which shell it has been using: /bin/sh. This is usually just a link to the real shell executable. Does it point to the same shell you have tested your command in?
Edit: the SO solution you referenced contains the solution in the comments:
Tip: Note however that using this pattern relies on extglob. You can
enable it using shopt -s extglob (If you want extended globs to be
turned on by default you can add shopt -s extglob to .bashrc)
Just to demonstrate that different shells might deal with your syntax in different ways, first using bash:
$ !(uname)
-bash: !: event not found
And then, using /bin/dash:
$ !(uname)
Linux

The argument to a subprocess.something method must be a list of command line arguments. Use e.g. shlex.split() to make the string be split into correct command line arguments:
import shlex, subprocess
subprocess.check_call( shlex.split("mv !(...)") )
EDIT:
So, the goal is to move files/directories, with the exemption of some file(s)/directory(ies). By playing around with bash, I could get it to work like this:
mv `ls | grep -v -e '\(exclusion1\|exclusion2\)'` my_project
So in your situation that would be:
mv `ls | grep -v -e '\(myproject\|cmake-3.0.2-Darwin64-universal\)'` my_project
This could go into the subprocess.check_call(..., shell=True) and it should do what you expect it to do.

bash wrap a piped command with a python script

Is there a way to create a python script which wraps an entire bash command including the pipes.
For example, if I have the following simple script
import sys
print sys.argv
and call it like so (from bash or ipython), I get the expected outcome:
[pkerp#pendari trell]$ python test.py ls
['test.py', 'ls']
If I add a pipe, however, the output of the script gets redirected to the pipe sink:
[pkerp#pendari trell]$ python test.py ls > out.txt
And the > out.txt portion is not in sys.argv. I understand that the shell automatically process this output, but I'm curious if there's a way to force the shell to ignore it and pass it to the process being called.
The point of this is to create something like a wrapper for the shell. I'd like to run the commands regularly, but keep track of the strace output for each command (including the pipes). Ideally I'd like to keep all of the bash features, such as tab-completion and up and down arrows and history search, and then just pass the completed command through a python script which invokes a subprocess to handle it.
Is this possible, or would I have to write my own shell to do this?
Edit
It appears I'm asking the exact same thing as this question.

The only thing you can do is pass the entire shell command as a string, then let Python pass it back to a shell for execution.
$ python test.py "ls > out.txt"
Inside test.py, something like
subprocess.call("strace " + sys.argv[1], shell=True, executable="/bin/bash")
to ensure the entire string is passed to the shell (and bash, specifically).

Well, I don't quite see what you are trying to do. The general approach would be to give the desired output destination to the script using command line options: python test.py ls --output=out.txt. Incidentally, strace writes to stderr. You could capture everything using strace python test.py > out 2> err if you want to save everything...
Edit: If your script writes to stderr as well you could use strace -o strace_out python test.py > script_out 2> script_err
Edit2: Okay, I understand better what you want. My suggestion is this: Write a bash helper:
function process_and_evaluate()
{
strace -o /tmp/output/strace_output "$#"
/path/to/script.py /tmp/output/strace_output
}
Put this in a file like ~/helper.sh. Then open a bash, source it using . ~/helper.sh.
Now you can run it like this: process_and_evaluate ls -lA.
Edit3:
To capture output / error you could extend the macro like this:
function process_and_evaluate()
{
out=$1
err=$2
shift 2
strace -o /tmp/output/strace_output "$#" > "$out" 2> "$err"
/path/to/script.py /tmp/output/strace_output
}
You would have to use the (less obvious ) process_and_evaluate out.txt err.txt ls -lA.
This is the best that I can come up with...

At least in your simple example, you could just run the python script as an argument to echo, e.g.
$ echo $(python test.py ls) > test.txt
$ more test.txt
['test.py','ls']
Enclosing a command in parenthesis with a dollar sign first executes the contents then passes the output as an argument to echo.

How to call a shell script function/variable from python?

Is there any way to call a shell script and use the functions/variable defined in the script from python?
The script is unix_shell.sh
#!/bin/bash
function foo
{
...
}
Is it possible to call this function foo from python?
Solution:
For functions: Convert Shell functions to python functions
For shell local variables(non-exported), run this command in shell, just before calling python script:
export $(set | tr '\n' ' ')
For shell global variables(exported from shell), in python, you can:
import os
print os.environ["VAR1"]

Yes, in a similar way to how you would call it from another bash script:
import subprocess
subprocess.check_output(['bash', '-c', 'source unix_shell.sh && foo'])

This can be done with subprocess. (At least this was what I was trying to do when I searched for this)
Like so:
output = subprocess.check_output(['bash', '-c', 'source utility_functions.sh; get_new_value 5'])
where utility_functions.sh looks like this:
#!/bin/bash
function get_new_value
{
let "new_value=$1 * $1"
echo $new_value
}
Here's how it looks in action...
>>> import subprocess
>>> output = subprocess.check_output(['bash', '-c', 'source utility_functions.sh; get_new_value 5'])
>>> print(output)
b'25\n'

No, that's not possible. You can execute a shell script, pass parameters on the command line, and it could print data out, which you could parse from Python.
But that's not really calling the function. That's still executing bash with options and getting a string back on stdio.
That might do what you want. But it's probably not the right way to do it. Bash can not do that many things that Python can not. Implement the function in Python instead.

With the help of above answer and this answer, I come up with this:
import subprocess
command = 'bash -c "source ~/.fileContainingTheFunction && theFunction"'
stdout = subprocess.getoutput(command)
print(stdout)
I'm using Python 3.6.5 in Ubuntu 18.04 LTS.

I do not know to much about python, but if You use export -f foo after the shell script function definition, then if You start a sub bash, the function could be called. Without export, You need to run the shell script as . script.sh inside the sub bash started in python, but it will run everything in it and will define all the functions and all variables.

You could separate each function into their own bash file. Then use Python to pass the right parameters to each separate bash file.
This may be easier than just re-writing the bash functions in Python yourself.
You can then call these functions using
import subprocess
subprocess.call(['bash', 'function1.sh'])
subprocess.call(['bash', 'function2.sh'])
# etc. etc.
You can use subprocess to pass parameters too.