We Keep Coding

How to filter shell output to only number with decimal?

I have CI/CD Config which required python version to be set to default by pyenv. I want to python2 -V output showed up with only, example, 2.7.18. But, rather than showing 2.7.18, it showing full text Python 2.7.18 .
But, when I use it in python3 python -V, it showed the correct & current python3 version (3.9.0).
I use this code to try showing numbers only : $(python -V | grep -Eo '[0-9]\.[0-9]\.[10-19]').
And to set default with pyenv : pyenv global $(python3 -V | grep -Eo '[0-9]\.[0-9]\.[10-19]') $(python -V | grep -Eo '[0-9]\.[0-9]\.[10-19]')
So pyenv $(python3 version) $(python2 version)
Here is the image :
Image of wrong output
Thanks!

A simple way would be to just replace the string Python with the emtpy string, if it exists.
Here a quick one-liner
python -V 2>&1| sed -e "s/Python//g" | xargs
That would print the python version, redirects stderr to stdout, replaces "Python" with "". Xargs without parameters returns the trimmed input string.

Here are a few more ways to get the version number:
# Print 1 word per line, the select the last word:
python -V 2>&1 | xargs -n1 | tail -n1
# Print the last word:
python -V 2>&1 | perl -lane 'print $F[-1];'
# Print the first stretch of 1 or more { digits or periods }:
python -V 2>&1 | grep -Po '[\d.]+'

Running subprocess with spaces in options in python

I tried to search for an answer for a while, but I did not find anything so far for my specific case. I want to run command in python:
ssh -o ConnectTimeout=3 -o ProxyCommand="ssh -q -W %h:%p bastion.host.com" host.com "screen -dmS TEST /bin/bash --login -c 'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log'"
this is my code:
import subprocess
server_command = "screen -dmS TEST /bin/bash --login -c 'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log'"
command = ['ssh', '-o', 'ConnectTimeout=3', 'ProxyCommand="ssh -q -W %h:%p bastion.host.com"', 'host.com', server_command]
p = subprocess.Popen(command, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
stdout, stderr = p.communicate(input=None)
Everything was working (screen was spawned with script running) until I added option with spaces: ProxyCommand="ssh -q -W %h:%p bastion.host.com".
After that I get error:
>>> print(stderr)
b'ssh: Could not resolve hostname ProxyCommand="ssh -q -W %h:%p bastion.host.com": Name or service not known\r\n'
How can I please pass this option to my command?

Your SSH command contains invalid arguments: ProxyCommand is an option, so it needs to be preceded by -o, same as ConnectTimeout (and, as noted by Charles Duffy, the redundant quotes inside that option string need to be removed, since the command is not passed to the shell):
server_command = 'screen -dmS TEST /bin/bash --login -c \'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log\''
command = ['ssh', '-o', 'ConnectTimeout=3', '-o', 'ProxyCommand=ssh -q -W %h:%p bastion.host.com', 'host.com', server_command]
In general when your command line contains spaces and/or quotes and is passed to another command, it may be necessary to shell-quote it. The Python function shlex.quote automates this. In your case it’s not necessary because you (correctly) manually quoted the command you’re passing to screen inside server_command. Alternatively you could have written the following:
script_command = 'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log'
server_command = f'screen -dmS TEST /bin/bash --login -c {shlex.quote(script_command)}'
— Note the absence of manual quotes inside the shell command line. The advantage over manual quoting is that this will also work with nested levels of shell quoting, e.g. when nesting command invocations.

Python os.popen quotes within quotes error

I have the following command, but it does not work. Can anyone help what the issue.
cur_usage=os.popen("""df -k \tmp |tail -1 | awk '{{print $4"\n"$5}}'| grep '%'|tr -d '%'""").read()
print(cur_usage)

There are a couple of things you need to do here:
If you're on a Unix-like OS, you should change \tmp to /tmp
You need to either change \n to \\n or mark the string as a raw string.
One of either of the following should work for you:
curr_usage = os.popen("""df -k /tmp |tail -1 | awk '{{print $4"\\n"$5}}'| grep '%'|tr -d '%'""").read().strip()
or
curr_usage = os.popen(r"""df -k /tmp |tail -1 | awk '{{print $4"\n"$5}}'| grep '%'|tr -d '%'""").read().strip()

pass bash script args as named parameters to a command inside the script

I have a bash script that takes two parameters. Inside that script, I need to call ssh using a heredoc and call a method that expects the two arguments. For example:
ssh -o "IdentitiesOnly=yes" -t -i $key -l user localhost << 'ENDSSH'
/my_python_app.py -u -t tar -p $1 -f $2
ENDSSH
key is set by my script, I know that part is good.
However, my_python_app prints out args and it doesn't show any arguments for -p and -f
I would call my script like
my_script /tmp filename
I use argparse in my python app, but I am also printing out sys.argv and it gives me:
['my_python_app.py', '-u', '-t', 'tar', '-p', '-f']
Note there are no values received for -p and -f. (-u is a flag, and that is set correctly).
How do I pass $1 and $2 to my_python_app as the -p and -f values?

Remove the quotes around the here-document delimiter (i.e. use << ENDSSH instead of << 'ENDSSH'). The quotes tell the shell not to expand variable references (and some other things) in the here-document, so $1 and $2 are passed through to the remote shell... which doesn't have any parameters so it replaces them with nothing.
BTW, removing the single-quotes may not fully work, since if either argument contains whitespace or shell metacharacters, the remote end will parse those in a way you probably don't intend. As long as neither argument can contain a single-quote, you can use this:
ssh -o "IdentitiesOnly=yes" -t -i $key -l user localhost << ENDSSH
/my_python_app.py -u -t tar -p '$1' -f '$2'
ENDSSH
If either might contain single-quotes, it gets a little more complicated.

The more paranoid way to do this would be:
# store these in an array to reduce the incidental complexity below
ssh_args=( -o "IdentitiesOnly=yes" -t -i "$key" -l user )
posixQuote() {
python -c 'import sys, pipes; sys.stdout.write(pipes.quote(sys.argv[1])+"\n")' "$#"
}
ssh "${ssh_args[#]}" localhost "bash -s $(posixQuote "$1") $(posixQuote "$2")" << 'ENDSSH'
/path/to/my_python_app.py -u -t tar -p "$1" -f "$2"
ENDSSH
If you know with certainty that the destination account's shell matches the local one (bash if the local shell is bash, ksh if the local shell is ksh), consider the following instead:
printf -v remoteCmd '%q ' /path/to/my_python_app.py -u -t tar -p "$1" -f "$2"
ssh "${ssh_args[#]}" localhost "$remoteCmd"

Bash/Python processes matching

I can use
pgrep -f 'keyword1 | keyword2'
to run a pgrep and return all processes that match either keyword.
How can I use & to do this instead? I just want processes that contain both keywords
The following patterns failed:
pgrep -f 'keyword1 & keyword2'
pgrep -f 'keyword2 && keyword2'
MAN pgrep(1)
OPTIONS
-f The pattern is normally only matched against the process
name. When -f is set, the full command line is used.
.
Side question:
Is there a built in Python library for running these commands? I couldnt seem to find one and everyone suggested using subprocess.Popen(), which is how I'm running the 'pgrep' command, however I'd prefer a pure Python solution if it's available

I'm not sure you can do that with pgrep you can however use awk:
ps ax -o pid,cmd | awk '{pid = $1; $1=""}/[k]eyword1/ && /keyword2/ {print pid}'
The reason i use [k]eyword1 is to avoid matching the awk process.
If PCRE is supported with pgrep something like this would work:
pgrep -f '(?=.*keyword1)(?=.*keyword2)'

You can use or with a wildcard reversing the pattern to get either keyword in any order.
pgrep -f 'keyword1.*keyword2|keyword2.*keyword1'

The typical way to do a grep 'and' is to grep multiple times. Since pgrep returns pids you have to filter the list of processes directly and then extract the PID:
ps ax -o pid,cmd | grep 'keyword1' | grep 'keyword2' | awk '{print $2}'