I want to run a (complex) Bash while loop from a Python3 script.
I know os.subprocess and os.subprocess.check_output works in this case, but I can't wrap my head around how to include the while inside a Python subprocess.
while read -r line
do
if [ "$(echo "$line" | cut -d : -f 7)" = "/bin/bash" ] && [ $(printf "$(echo "$line" | cut -d : -f 1)" | wc -c) -gt $mida ]
then
echo $line | cut -d : -f 1
fi
done < /etc/passwd
I've tried the following:
out=subprocess.check_output(""" while read -r line; do; if [ "$(echo "$line" | cut -d : -f 7)" = "/bin/bash" ] && [ $(printf "$(echo "$line" | cut -d : -f 1)" | wc -c) -gt $mida ]; then; echo $line | cut -d : -f 1; fi; done < /etc/passwd """, shell=True)
Just include it normally. Like it is. You are using """ quotes anyway.
out = subprocess.check_output("""
while read -r line
do
if [ "$(echo "$line" | cut -d : -f 7)" = "/bin/bash" ] && [ $(printf "$(echo "$line" | cut -d : -f 1)" | wc -c) -gt $mida ]
then
echo $line | cut -d : -f 1
fi
done < /etc/passwd
""", shell=True)
Notes:
you should export mida environment variable before using it. When $mida variable is not set will result in some [: something expected but not there messages.
printf "$(stuff)" | wc -c? Just stuff | wc -c.
Check your scripts with http://shellcheck.net
Read https://mywiki.wooledge.org/BashFAQ/001
Just split the line on : using IFS when reading instead of using cut
And that said, do not use shell - use python and write the logic in python.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I am new to bash and wanted to learn what this code is trying to do, if it is done poorly or with errors and how it can be improved.
COMMAND=$1
case $COMMAND in
"upgrade")
UPSCRIPT=`ls -t ./assets/upgrade | head -n1`
python ./assets/upgrade/$UPSCRIPT | tee >> biglog.txt
VERSION=$(echo $UPSCRIPT | awk -F. '{print $1}')
echo `date` $VERSION > ./version.txt
test -e ./artifcts && rm -rf ./artifacts
;;
"downgrade")
DOWNSCRIPT=`ls -t ./assets/downgrade | head -n1`
python ./assets/downgrade/$DOWNSCRIPT | tee >> biglog.txt
VERSION=$(echo $UPSCRIPT | awk -F. '{print $1}')
echo `date` $VERSION > ./version.txt
test -e ./artifcts && rm -rf ./artifacts
;;
*)
while read -r UPSCRIPT; do
python $UPSCRIPT | tee >> biglog.txt
VERSION=$(echo $UPSCRIPT | awk -F. '{print $1}')
echo `date` $VERSION > ./version.txt
test -e ./artifcts && rm -rf ./artifacts
done <<< $(find "./assets/update" -type f -name "*.py")
esac
Use lower case variable names. Upper case is recommended for environment and shell internal variables.
Use $() instead of `...`. It nests better.
use parameter expansion instead of running a command in a subshell, if possible. It's much faster.
Where the logic of the script was unclear, I left a comment in the code.
#! /bin/bash
command=$1
artifacts=./artifacts
case "$command" in
upgrade)
upscript=$(ls -t ./assets/upgrade | head -n1)
python ./assets/upgrade/"$upscript" | tee >> biglog.txt
version=${upscript%.*}
echo $(date) "$version" > ./version.txt
test -e "$artifacts" && rm -rf "$artifacts" # artifacts or artifcts?
;;
downgrade)
downscript=$(ls -t ./assets/downgrade | head -n1)
python ./assets/downgrade/"$downscript" | tee >> biglog.txt
version=${downscript%.*} # upscript or downscript?
echo $(date) "$version" > ./version.txt
test -e "$artifacts" && rm -rf "$artifacts"
;;
*)
while read -r upscript; do
python "$upscript" | tee >> biglog.txt
version=${upscript%.*}
echo $(date) "$version" > ./version.txt
test -e "$artifacts" && rm -rf "$artifacts"
done <<< $(find "./assets/update" -type f -name '*.py')
esac
I would probably also extract the common logic from upgrade and downgrade to a function to avoid repetition.
Parsing the output of ls or find is suspicious, as file names can contain weird characters. I'd need to understand more what you're trying to do to fix that.
I have the following shell script. It checks whether the python script is running or not, and if not, it will start the python script.
When I run it from command line like ./crontab.sh it works even from other places like MaskRCNN/crontab.sh. But when I put it into the crontab only the echos are working.
#!/bin/bash
out=$(ps aux | grep 'python train/train.py' | rev | cut -d ' ' -f 1| rev | wc -l)
if [ $out -eq "2" ];then
echo "2 processes" >> /tmp/testing.txt
else
echo "1 process" >> /tmp/testing.txt;
cd /hdd1/Alex/testMaskRCNN_human_bodyparts/MaskRCNN_body;
CUDA_VISIBLE_DEVICES=0 /usr/bin/python train/train.py
fi
The crontab looks like this:
* * * * * /hdd1/Alex/testMaskRCNN_human_bodyparts/MaskRCNN_body/crontab.sh
I hope there are enough details.
Thank you
try change train/train.py to full path...
I managed to debug it by adding the &>> operator.
`(/usr/bin/python /hdd1/Alex/testMaskRCNN_human_bodyparts/MaskRCNN_body/train/train.py &>> /tmp/testing.txt)`
And I found out that it missed an export:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64/
Final script:
#!/bin/bash
out=$(ps aux | grep '/usr/bin/python /hdd1/Alex/testMaskRCNN_human_bodyparts/MaskRCNN_body/train/train.py' | rev | cut -d ' ' -f 1 | rev | wc -l)
if [ $out -eq "2" ];then
echo "2 processes" >> /tmp/testing.txt
else
echo "1 process" >> /tmp/testing.txt
export CUDA_VISIBLE_DEVICES=0
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64/
`(/usr/bin/python /hdd1/Alex/testMaskRCNN_human_bodyparts/MaskRCNN_body/train/train.py &>> /tmp/testing.txt)`
fi
I'm trying to write a bash function named myrun, such that doing
myrun script.py
with a Python file:
#MYRUN:nohup python -u script.py &
import time
print 'Hello world'
time.sleep(2)
print 'Once again'
will run the script with the command specified in the first line of the file, just after #MYRUN:.
What should I insert in .bashrc to allow this? Here is what I have now:
myrun () {
[[ "$1" = "" ]] && echo "usage: myrun python_script.py" && return 0
<something with awk here or something else?>
}
A minimalist version:
$ function myrun {
[[ "$1" = "" ]] && echo "usage: myrun python_script.py" && return
local cmd=$(head -n 1 < "$1" | sed s'/# *MYRUN://')
$cmd
}
$ myrun script.py
appending output to nohup.out
$ cat nohup.out
Hello world
Once again
$
(It's not clear to me whether you're better off using eval "$cmd" or simply $cmd in the last line of the function, but if you want to include the "&" in the MYCMD directive, then $cmd is simpler.)
With some basic checking:
function myrun {
[[ "$1" = "" ]] && echo "usage: myrun python_script.py" && return
local cmd=$(head -n 1 <"$1")
if [[ $cmd =~ ^#MYRUN: ]] ; then cmd=${cmd#'#MYRUN:'}
else echo "myrun: #MYRUN: header not found" >&2 ; false; return ; fi
if [[ -z $cmd ]] ; then echo "myrun: no command specified" >&2 ; false; return; fi
$cmd # or eval "$cmd" if you prefer
}
This is unrelated to Bash. Unfortunately, the shebang line cannot portably contain more than a single argument or option group.
If your goal is to specify options to Python, the simplest thing is probably a simple sh wrapper:
#!/bin/sh
nohup python -u <<'____HERE' &
.... Your Python script here ...
____HERE
I have written a bash script that consists of multiple Unix commands and Python scripts. The goal is to make a pipeline for detecting long non coding RNA from a certain input. Ultimately I would like to turn this into an 'app' and host it on some bioinformatics website. One problem I am facing is using getopt tools in bash. I couldn't find a good tutorial that I understand clearly. In addition any other comments related to the code is appreciated.
#!/bin/bash
if [ "$1" == "-h" ]
then
echo "Usage: sh $0 cuffcompare_output reference_genome blast_file"
exit
else
wget https://github.com/TransDecoder/TransDecoder/archive/2.0.1.tar.gz && tar xvf 2.0.1 && rm -r 2.0.1
makeblastdb -in $3 -dbtype nucl -out $3.blast.out
grep '"u"' $1 | \
gffread -w transcripts_u.fa -g $2 - && \
python2.7 get_gene_length_filter.py transcripts_u.fa transcripts_u_filter.fa && \
TransDecoder-2.0.1/TransDecoder.LongOrfs -t transcripts_u_filter.fa
sed 's/ .*//' transcripts_u_filter.fa | grep ">" | sed 's/>//' > transcripts_u_filter.fa.genes
cd transcripts_u_filter.fa.transdecoder_dir
sed 's/|.*//' longest_orfs.cds | grep ">" | sed 's/>//' | uniq > longest_orfs.cds.genes
grep -v -f longest_orfs.cds.genes ../transcripts_u_filter.fa.genes > longest_orfs.cds.genes.not.genes
sed 's/^/>/' longest_orfs.cds.genes.not.genes > temp && mv temp longest_orfs.cds.genes.not.genes
python ../extract_sequences.py longest_orfs.cds.genes.not.genes ../transcripts_u_filter.fa longest_orfs.cds.genes.not.genes.fa
blastn -query longest_orfs.cds.genes.not.genes.fa -db ../$3.blast.out -out longest_orfs.cds.genes.not.genes.fa.blast.out -outfmt 6
python ../filter_sequences.py longest_orfs.cds.genes.not.genes.fa.blast.out longest_orfs.cds.genes.not.genes.fa.blast.out.filtered
grep -v -f longest_orfs.cds.genes.not.genes.fa.blast.out.filtered longest_orfs.cds.genes.not.genes.fa > lincRNA_final.fa
fi
Here is how I run it:
sh test.sh cuffcompare_out_annot_no_annot.combined.gtf /mydata/db/Brapa_sequence_v1.2.fa TE_RNA_transcripts.fa
If you wanted the call to be :
test -c cuffcompare_output -r reference_genome -b blast_file
You would have something like :
#!/bin/bash
while getopts ":b:c:hr:" opt; do
case $opt in
b)
blastfile=$OPTARG
;;
c)
comparefilefile=$OPTARG
;;
h)
echo "USAGE : test -c cuffcompare_output -r reference_genome -b blast_file"
;;
r)
referencegenome=$OPTARG
;;
\?)
echo "Invalid option: -$OPTARG" >&2
exit 1
;;
:)
echo "Option -$OPTARG requires an argument." >&2
exit 1
;;
esac
done
In the string ":b:c:hr:",
- the first ":" tells getopts that we'll handle any errors,
- subsequent letters are the allowable flags. If the letter is followed by a ':', then getopts will expect that flag to take an argument, and supply that argument as $OPTARG
I have a simple python script i need to start and stop and i need to use a start.sh and stop.sh script to do it.
I have start.sh:
#!/bin/sh
script='/path/to/my/script.py'
echo 'starting $script with nohup'
nohup /usr/bin/python $script &
and stop.sh
#!/bin/sh
PID=$(ps aux | grep "/path/to/my/script.py" | awk '{print $2}')
echo "killing $PID"
kill -15 $PID
I'm mainly concerned with the stop.sh script. I think that's an appropriate way to find the pid but i wouldn't bet much on it. start.sh successfully starts it. when i run stop.sh, i can no longer find the process by "ps aux | grep 'myscript.py'" but the console outputs:
killing 25052
25058
./stop.sh: 5: kill: No such process
so it seems like it works AND gives an error of sorts with "No such process".
Is this actually an error? Am I approaching this in a sane way? Are there other things I should be paying attention to?
EDIT - I actually ended up with something like this:
start.sh
#!/bin/bash
ENVT=$1
COMPONENTS=$2
TARGETS=("/home/user/project/modules/script1.py" "/home/user/project/modules/script2.py")
for target in "${TARGETS[#]}"
do
PID=$(ps aux | grep -v grep | grep $target | awk '{print $2}')
echo $PID
if [[ -z "$PID" ]]
then
echo "starting $target with nohup for env't: $ENVT"
nohup python $target $ENVT $COMPONENTS &
fi
done
stop.sh
#!/bin/bash
ENVT=$1
TARGETS=("/home/user/project/modules/script1.py" "/home/user/project/modules/script2.py")
for target in "${TARGETS[#]}"
do
pkill -f $target
echo "killing process $target"
done
It is because ps aux |grep SOMETHING also finds the grep SOMETHING process, because SOMETHING matches. After the execution the grep is finished, so it cannot find it.
Add a line: ps aux | grep -v grep | grep YOURSCRIPT
Where -v means exclude. More in man grep.
The "correct" approach would probably be to have your script write its pid to a file in /var/run, and clear it out when you kill the script. If changing the script is not an option, have a look at start-stop-daemon.
If you want to continue with the grep-like approach, have a look at proctools. They're built in on most GNU/Linux machines and readily available on BSD including OS X:
pkill -f /path/to/my/script.py
init-type scripts are useful for this. This is very similar to one I use. You store the pid in a file, and when you want to check if it's running, look into the /proc filesystem.
#!/bin/bash
script_home=/path/to/my
script_name="$script_home/script.py"
pid_file="$script_home/script.pid"
# returns a boolean and optionally the pid
running() {
local status=false
if [[ -f $pid_file ]]; then
# check to see it corresponds to the running script
local pid=$(< "$pid_file")
local cmdline=/proc/$pid/cmdline
# you may need to adjust the regexp in the grep command
if [[ -f $cmdline ]] && grep -q "$script_name" $cmdline; then
status="true $pid"
fi
fi
echo $status
}
start() {
echo "starting $script_name"
nohup "$script_name" &
echo $! > "$pid_file"
}
stop() {
# `kill -0 pid` returns successfully if the pid is running, but does not
# actually kill it.
kill -0 $1 && kill $1
rm "$pid_file"
echo "stopped"
}
read running pid < <(running)
case $1 in
start)
if $running; then
echo "$script_name is already running with PID $pid"
else
start
fi
;;
stop)
stop $pid
;;
restart)
stop $pid
start
;;
status)
if $running; then
echo "$script_name is running with PID $pid"
else
echo "$script_name is not running"
fi
;;
*) echo "usage: $0 <start|stop|restart|status>"
exit
;;
esac
ps aux | grep "/path/to/my/script.py"
will return both the pid for the instance of script.py and also for this instance of grep. That'll probably be why you're getting a no such process: by the time you get around to killing the grep, it's already dead.
I don't have a unix box on at the moment, so i can't test this, but it should be fairly simple to get the idea.
start.sh:
if [ -e ./temp ]
then
pid=`cat temp`
echo "Process already exists; $pid"
else
script='/path/to/my/script.py'
echo 'starting $script with nohup'
nohup /usr/bin/python $script &
echo $! > temp
fi
stop.sh:
if [ -e ./temp ]
then
pid=`cat temp`
echo "killing $pid"
kill -15 $PID
rm temp
else
echo "Process not started"
fi
Try this out.