I'm working on a Django project where I've created a makefile task to reset my database setup (i.e. reset-db).
However, as I just want to syncdb the core application, I need to be able to toggle comment on the legacy app line. Commenting prior to the syncdb and uncommenting after (so other operation happen normally).
Default state
INSTALLED_APPS = (
…
'django_extensions',
'core',
'legacy' # #reset-db
)
Goal state
INSTALLED_APPS = (
…
'django_extensions',
'core',
# 'legacy' # #reset-db
)
Makefile reset-db
The task is currently
reset-db:
# cmd to comment line
DJANGO_SETTINGS_MODULE=${SETTINGS} sudo -u postgres -H dropdb evrpa \
&& sudo -u postgres -H createdb evrpa -O elopez; \
./manage.py syncdb --noinput --database=default;
# cmd to UNcomment line
So what's the sed or awk command to do that?
Just take a copy of the original file, modify it, use it, then restore it:
mv ${SETTINGS_PY} ${SETTINGS_PY}.tmp &&
sed '/#reset-db/s/^/#/' ${SETTINGS_PY}.tmp > ${SETTINGS_PY} &&
./manage.py syncdb --noinput --database=default &&
mv ${SETTINGS_PY}.tmp ${SETTINGS_PY}
That way you don't have to come up with a script to try to get the modified file back to it's original content since you have a copy of that original file to restore from.
I assume you have a good reason for not quoting your variables and so I have also left them unquoted.
I added && at the end of every line because you always want to test for the previous command succeeding before executing the next command. If that's not the right syntax to do that in your makefile, change it as appropriate.
Here is how I've done it
Commenting
awk '/#reset-db/{ $0="# " $0 } {print}' ${SETTINGS_PY} > ${SETTINGS_PY}.tmp
For lines matching the pattern /#reset-db/, I updated the line content by prefixing # to it using { $0="# " $0 }.
Then I print all lines with {print}.
Uncommenting
awk '/^# .*#reset-db/{ $0=gensub(/^#(.*)/, "\\1", "", $0) } {print}' ${SETTINGS_PY} > ${SETTINGS_PY}.tmp
For lines matching the pattern /^# .*#reset-db/, I updated the current line content by removing # using { $0=gensub(/^#(.*)/, "\\1", "", $0) }.
Then I print all lines with {print}.
Full task
Note: You need to escape the $ in makefile scripts with another $ (e.g. $0 → $$0).
reset-db:
DJANGO_SETTINGS_MODULE=${SETTINGS} sudo -u postgres -H dropdb evrpa \
&& sudo -u postgres -H createdb evrpa -O elopez; \
awk '/#reset-db/{ $$0="# " $$0 } {print}' ${SETTINGS_PY} > ${SETTINGS_PY}.tmp \
&& mv ${SETTINGS_PY}{.tmp,}
./manage.py syncdb --noinput --database=default;
awk '/^# .*#reset-db/{ $$0=gensub(/^#(.*)/, "\\1", "", $$0) } {print}' ${SETTINGS_PY} > ${SETTINGS_PY}.tmp \
&& mv ${SETTINGS_PY}{.tmp,}
Related
Is there a way to create a Makefile/script that will fail if a file from coverage.py library has values below a certain threshold? Say 80%.
For coverage.py you can use fail_under option
First, gather test coverage by using the coverage run command (here), then run the coverage report command
Option 1.
Run via coverage report command directly
coverage report --fail-under=80
Option 2.
Use configuration file by defining value in the report section of .coveragerc
[report]
fail_under = 80
Then run the coverage report command
coverage report
It will return non zero exit code if the coverage is below the value of fail_under
If you want something more fine-grained than an overall percentage threshold, you can try this coverage goals program I threw together: https://nedbatchelder.com/blog/202111/coverage_goals.html
I figured it out with the following. Feel free to copy if it fits your needs!
The Makefile goes as follows:
test: $(TEST_DIRECTORY)/*/*
rm -f $(TEST_OUTPUT_DIRECTORY)/coverage_tests.log
cd $(TEST_DIRECTORY)/inference && coverage run -m unittest discover
for file in $^ ; do \
TEST_FILE=$$(echo $${file} | grep -E -o test_.*) ; \
FILE=$$(echo $${TEST_FILE} | sed 's/test_//') ; \
if [[ ! -z $${FILE} ]] ; then \
cd $(TEST_DIRECTORY)/inference && coverage report --include $(INFERENCE_DIRECTORY)/$${FILE} > $(TEST_OUTPUT_DIRECTORY)/coverage_temp.log ; \
cd $(TEST_DIRECTORY) && ./coverage_check.sh $${FILE} < $(TEST_OUTPUT_DIRECTORY)/coverage_temp.log >> $(TEST_OUTPUT_DIRECTORY)/coverage_tests.log ; \
rm -f $(TEST_OUTPUT_DIRECTORY)/coverage_temp.log ; \
fi ; \
done
rm -f $(TEST_DIRECTORY)/inference/.coverage
and it utilizes a coverage check script which is here:
#!/bin/sh
while read line; do
for word in $line; do
if [[ "$word" == *"%"* ]]; then
COVERAGE=`echo $word | sed 's/%//g'`
if [ $COVERAGE -gt 80 ]
then
echo "$1: PASSED $word"
break 2
else
echo "$1: FAILED $word"
break 2
fi
fi
done
done
I have been following this tutorial to install opencv and python:
https://www.pyimagesearch.com/2015/06/22/install-opencv-3-0-and-python-2-7-on-ubuntu/#comment-441393
The only difference is that I am trying to install opencv 3.3.1 instead of 3.0.0
I'm running on a laptop with Ubuntu 14.04 an i7 and NVIDIA GTX950M
The problem is that when I execute the command ldconfig
$ sudo make install
$ sudo ldconfig
I get the following message:
/sbin/ldconfig.real: /usr/lib/nvidia-384/libEGL.so.1 is not a symbolic link
/sbin/ldconfig.real: /usr/lib32/nvidia-384/libEGL.so.1 is not a symbolic link
So I found a solution to the problem:
Source: https://askubuntu.com/questions/900285/libegl-so-1-is-not-a-symbolic-link #muru #Gerard Tromp
The following is an easy-to-use version of Noisy_Botnet's solution. It
facilitates repeating the process for any update.
Create a shell script, i.e., paste the code bellow a text file, and save it with the .sh extension.
Change the executing permissions of the file i.e., go to the location of the file in the terminal and execute the following command $sudo chmod 744 nameofthefieleyoucreated.sh
execute the following command $sudo ./nameofthefileyoucreated.sh
#! /bin/sh
#
# find the file in /usr/lib
LIBEGL=`find /usr/lib/nvidia* -name libEGL.so.\* | egrep "[0-9][0-9]*\.[0-9][0-9]*$"`
LIBEGL_LINK=`echo $LIBEGL | sed 's/[0-9][0-9]*\.[0-9][0-9]*$/1/'`
printf "\n\nThe following commands will be executed:\n+++++++++++++++++++++++++++++++++++++++\n"
printf "mv $LIBEGL_LINK ${LIBEGL_LINK}.orig\nln -s $LIBEGL $LIBEGL_LINK\n\n"
while true; do
read -p "Do you wish to perform these commands? " yn
case $yn in
[Yy]* ) mv $LIBEGL_LINK ${LIBEGL_LINK}.orig; ln -s $LIBEGL $LIBEGL_LINK ; break;;
[Nn]* ) break;;
* ) echo "Please answer yes or no.";;
esac
done
# find the file in /usr/lib32
LIBEGL=`find /usr/lib32/nvidia* -name libEGL.so.\* | egrep "[0-9][0-9]*\.[0-9][0-9]*$"`
LIBEGL_LINK=`echo $LIBEGL | sed 's/[0-9][0-9]*\.[0-9][0-9]*$/1/'`
printf "\n\nThe following commands will be executed:\n+++++++++++++++++++++++++++++++++++++++\n"
printf "mv $LIBEGL_LINK ${LIBEGL_LINK}.orig\nln -s $LIBEGL $LIBEGL_LINK\n\n"
while true; do
read -p "Do you wish to perform these commands? " yn
case $yn in
[Yy]* ) mv $LIBEGL_LINK ${LIBEGL_LINK}.orig; ln -s $LIBEGL $LIBEGL_LINK ; break;;
[Nn]* ) break;;
* ) echo "Please answer yes or no.";;
esac
done
Running this on osx...
cd ${BUILD_DIR}/mydir && for DIR in $(find ./ '.*[^_].py' | sed 's/\/\//\//g' | awk -F "/" '{print $2}' | sort |uniq | grep -v .py); do
if [ -f $i/requirements.txt ]; then
pip install -r $i/requirements.txt -t $i/
fi
cd ${DIR} && zip -r ${DIR}.zip * > /dev/null && mv ${DIR}.zip ../../ && cd ../
done
cd ../
error:
(env) ➜ sh package_lambdas.sh find: .*[^_].py: No such file or directory
why?
find takes as an argument a list of directories to search. You provided what appears to be regular expression. Because there is no directory named (literally) .*[^_].py, find returns an error.
Below I have revised your script to correct that mistake (if I understand your intention). Because I see so many ill-written shell scripts these days, I've taken the liberty of "traditionalizing" it. Please see if you don't also find it more readable.
Changes:
use #!/bin/sh, guaranteed to be on an Unix-like system. Faster than bash, unless (like OS X) it is bash.
use lower case for variable names to distinguish from system variables (and not hide them).
eschew braces for variables (${var}); they're not needed in the simple case
do not pipe output to /usr/bin/true; route it to dev/null if that's what you mean
rm -f by definition cannot fail; if you meant || true, it's superfluous
put then and do on separate lines, easier to read, and that's how the Bourne shell language was meant to be used
Let && and || serve as line-continuation, so you can see what's happening step by step
Other changes I would suggest:
Use a subshell when changing the working directory temporarily. When it terminates, the working directory is restored automatically (retained by the parent), saving you the cd .. step, and errors.
Use set -e to cause the script to terminate on error. For expected errors, use || true explicitly.
Change grep .py to grep '\.py$', just for good measure.
To avoid Tilting Matchstick Syndrome, use something other than / as a sed substitute delimiter, e.g., sed 's://:/:g'. But sed could be avoided altogether with awk -F '/+' '{print $2}'.
Revised version:
#! /bin/sh
src_dir=lambdas
build_dir=bin
mkdir -p $build_dir/lambdas
rm -rf $build_dir/*.zip
cp -r $src_dir/* $build_dir/lambdas
#
# The sed is a bit complicated to be osx / linux cross compatible :
# ( .//run.sh vs ./run.sh
#
cd $build_dir/lambdas &&
for L in $(find . -exec grep -l '.*[^_].py' {} + |
sed 's/\/\//\//g' |
awk -F "/" '{print $2}' |
sort |
uniq |
grep -v .py)
do
if [ -f $i/requirements.txt ]
then
echo "Installing requirements"
pip install -r $i/requirements.txt -t $i/
fi
cd $L &&
zip -r $L.zip * > /dev/null &&
mv $L.zip ../../ &&
cd ../
done
cd ../
The find(1) manpage says its args are [path ...] [expression], where "expression" consists of "primaries" and "operands" (-flags). '.*[^-].py' doesn't look like any expression, so it's being interpreted as a path, and it's reporting that there is no file named '.*[^-].py' in the working directory.
Perhaps you meant:
find ./ -regex '.*[^-].py'
With Seafile one is able to create a public upload link (e.g. https://cloud.seafile.com/u/d/98233edf89/) to upload files via Browser w/o authentication.
Seafile webapi does not support any upload w/o authentication token.
How can I use such kind of link from command line with curl or from python script?
needed 2 hours to find a solution with curl, it needs two steps:
make a get-request to the public uplink url with the repo-id as query parameter as follows:
curl 'https://cloud.seafile.com/ajax/u/d/98233edf89/upload/?r=f3e30b25-aad7-4e92-b6fd-4665760dd6f5' -H 'Accept: application/json' -H 'X-Requested-With: XMLHttpRequest'
The answer is (json) a id-link to use in next upload-post e.g.:
{"url": "https://cloud.seafile.com/seafhttp/upload-aj/c2b6d367-22e4-4819-a5fb-6a8f9d783680"}
Use this link to initiate the upload post:
curl 'https://cloud.seafile.com/seafhttp/upload-aj/c2b6d367-22e4-4819-a5fb-6a8f9d783680' -F file=#./tmp/index.html -F filename=index.html -F parent_dir="/my-repo-dir/"
The answer is json again, e.g.
[{"name": "index.html", "id": "0a0742facf24226a2901d258a1c95e369210bcf3", "size": 10521}]
done ;)
#!/usr/bin/env bash
# this script depend on jq,check it first
RED='\033[0;31m'
NC='\033[0m' # No Color
if ! command -v jq &> /dev/null
then
echo -e "${RED}jq could not be found${NC}, installed and restart plz!\n"
exit
fi
usage () { echo "Usage : $0 -u <username> -p <password> -h <seafile server host> -f <upload file path> -d <parent dir default value is /> -r <repo id> -t <print debug info switch off/on,default off>"; }
# parse args
while getopts "u:p:h:f:d:r:t:" opts; do
case ${opts} in
u) USER=${OPTARG} ;;
p) PASSWORD=${OPTARG} ;;
h) HOST=${OPTARG} ;;
f) FILE=${OPTARG} ;;
d) PARENT_DIR=${OPTARG} ;;
r) REPO=${OPTARG} ;;
t) DEBUG=${OPTARG} ;;
*) usage; exit;;
esac
done
# those args must be not null
if [ ! "$USER" ] || [ ! "$PASSWORD" ] || [ ! "$HOST" ] || [ ! "$FILE" ] || [ ! "$REPO" ]
then
usage
exit 1
fi
# optional args,set default value
[ -z "$DEBUG" ] && DEBUG=off
[ -z "$PARENT_DIR" ] && PARENT_DIR=/
# print vars key and value when DEBUG eq on
[[ "on" == "$DEBUG" ]] && echo -e "USER:${USER} PASSWORD:${PASSWORD} HOST:${HOST} FILE:${FILE} PARENT_DIR:${PARENT_DIR} REPO:${REPO} DEBUG:${DEBUG}"
# login and get token
TOKEN=$(curl -s --location --request POST "${HOST}/api2/auth-token/" --header 'Content-Type: application/x-www-form-urlencoded' --data-urlencode "username=${USER}" --data-urlencode "password=${PASSWORD}" | jq -r ".token")
[ -z "$TOKEN" ] && echo -e "${RED}login seafile faild${NC}, call your administrator plz!\n" && exit 1
# gen upload link
UPLOAD_LINK=$(curl -s --header "Authorization: Token ${TOKEN}" "${HOST}/api2/repos/${REPO}/upload-link/?p=${PARENT_DIR}" | jq -r ".")
[ -z "$UPLOAD_LINK" ] && echo -e "${RED}get upload link faild${NC}, call your administrator plz!\n" && exit 1
# upload file
UPLOAD_RESULT=$(curl -s --header "Authorization: Token ${TOKEN}" -F file="#${FILE}" -F filename=$(basename ${FILE}) -F parent_dir="${PARENT_DIR}" -F replace=1 "${UPLOAD_LINK}?ret-json=1")
[ -z "$UPLOAD_RESULT" ] && echo -e "${RED}faild to upload ${FILE}${NC}, call your administrator plz!\n" && exit 1
# print upload result
[[ "on" == "$DEBUG" ]] && echo -e "TOKEN:${TOKEN} UPLOAD_LINK:${UPLOAD_LINK} UPLOAD_RESULT:${UPLOAD_RESULT}"
save as seafile-upload.sh
# ubuntu
apt install -y jq
# centos
# yum install -y jq
chmod +x ./seafile-upload.sh
./seafile-upload.sh -u <username> -p <password> -h <seafile server host> -f <upload file path> -d <parent dir default value is /,must be start with /> -r <repo id> -t <print debug info switch off/on,default off>
my blog 060-轻量级基于curl的seafile上传脚本
official document web-api/v2.1/file-upload.md
I have written a bash script that consists of multiple Unix commands and Python scripts. The goal is to make a pipeline for detecting long non coding RNA from a certain input. Ultimately I would like to turn this into an 'app' and host it on some bioinformatics website. One problem I am facing is using getopt tools in bash. I couldn't find a good tutorial that I understand clearly. In addition any other comments related to the code is appreciated.
#!/bin/bash
if [ "$1" == "-h" ]
then
echo "Usage: sh $0 cuffcompare_output reference_genome blast_file"
exit
else
wget https://github.com/TransDecoder/TransDecoder/archive/2.0.1.tar.gz && tar xvf 2.0.1 && rm -r 2.0.1
makeblastdb -in $3 -dbtype nucl -out $3.blast.out
grep '"u"' $1 | \
gffread -w transcripts_u.fa -g $2 - && \
python2.7 get_gene_length_filter.py transcripts_u.fa transcripts_u_filter.fa && \
TransDecoder-2.0.1/TransDecoder.LongOrfs -t transcripts_u_filter.fa
sed 's/ .*//' transcripts_u_filter.fa | grep ">" | sed 's/>//' > transcripts_u_filter.fa.genes
cd transcripts_u_filter.fa.transdecoder_dir
sed 's/|.*//' longest_orfs.cds | grep ">" | sed 's/>//' | uniq > longest_orfs.cds.genes
grep -v -f longest_orfs.cds.genes ../transcripts_u_filter.fa.genes > longest_orfs.cds.genes.not.genes
sed 's/^/>/' longest_orfs.cds.genes.not.genes > temp && mv temp longest_orfs.cds.genes.not.genes
python ../extract_sequences.py longest_orfs.cds.genes.not.genes ../transcripts_u_filter.fa longest_orfs.cds.genes.not.genes.fa
blastn -query longest_orfs.cds.genes.not.genes.fa -db ../$3.blast.out -out longest_orfs.cds.genes.not.genes.fa.blast.out -outfmt 6
python ../filter_sequences.py longest_orfs.cds.genes.not.genes.fa.blast.out longest_orfs.cds.genes.not.genes.fa.blast.out.filtered
grep -v -f longest_orfs.cds.genes.not.genes.fa.blast.out.filtered longest_orfs.cds.genes.not.genes.fa > lincRNA_final.fa
fi
Here is how I run it:
sh test.sh cuffcompare_out_annot_no_annot.combined.gtf /mydata/db/Brapa_sequence_v1.2.fa TE_RNA_transcripts.fa
If you wanted the call to be :
test -c cuffcompare_output -r reference_genome -b blast_file
You would have something like :
#!/bin/bash
while getopts ":b:c:hr:" opt; do
case $opt in
b)
blastfile=$OPTARG
;;
c)
comparefilefile=$OPTARG
;;
h)
echo "USAGE : test -c cuffcompare_output -r reference_genome -b blast_file"
;;
r)
referencegenome=$OPTARG
;;
\?)
echo "Invalid option: -$OPTARG" >&2
exit 1
;;
:)
echo "Option -$OPTARG requires an argument." >&2
exit 1
;;
esac
done
In the string ":b:c:hr:",
- the first ":" tells getopts that we'll handle any errors,
- subsequent letters are the allowable flags. If the letter is followed by a ':', then getopts will expect that flag to take an argument, and supply that argument as $OPTARG