We Keep Coding

Docker : Add a valid entrypoint for multiple python script

Helli, I have to build a Docker image for the following bioinformatics tool: https://github.com/CAMI-challenge/CAMISIM. Their dockerfile works but takes a long time to build and I would like to build my own, slightly differently, to learn. I face issues: there are several python script that I should be able to choose to run, not only a main. If I add one script in particular as an ENTRYPOINT then the behavior isn't exactly what I shoud have.
The Dockerfile:
FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
USER root
#COPY ./install_docker.sh ./
#RUN chmod +x ./install_docker.sh && sh ./install_docker.sh
RUN apt-get update && \
apt install -y git python3-pip libxml-simple-perl libncursesw5 && \
git clone https://github.com/CAMI-challenge/CAMISIM.git && \
pip3 install numpy ete3 biom-format biopython matplotlib joblib scikit-learn
ENTRYPOINT ["python3"]
ENV PATH="/CAMISIM/:${PATH}"
This yields :
sudo docker run camisim:latest metagenomesimulation.py --help
python3: can't open file 'metagenomesimulation.py': [Errno 2] No such file or directory
Adding that script as an ENTRYPOINT after python3 allows me to use it with 2 drawbacks: I cannot use another script (I could build a second docker image but that would be a bad solution), and it outputs:
ERROR: 0
usage: python metagenomesimulation.py configuration_file_path
#######################################
# MetagenomeSimulationPipeline #
#######################################
Pipeline for the simulation of a metagenome
optional arguments:
-h, --help show this help message and exit
-silent, --silent Hide unimportant Progress Messages.
-debug, --debug_mode more information, also temporary data will not be deleted
-log LOGFILE, --logfile LOGFILE
output will also be written to this log file
optional config arguments:
-seed SEED seed for random number generators
-s {0,1,2}, --phase {0,1,2}
available options: 0,1,2. Default: 0
0 -> Full run,
1 -> Only Comunity creation,
2 -> Only Readsimulator
-id DATA_SET_ID, --data_set_id DATA_SET_ID
id of the dataset, part of prefix of read/contig sequence ids
-p MAX_PROCESSORS, --max_processors MAX_PROCESSORS
number of available processors
required:
config_file path to the configuration file
You can see there is an error that should'nt be there, it actually does not use the help flag. The original Dockerfile is:
FROM ubuntu:20.04
RUN apt update
RUN apt install -y python3 python3-pip perl libncursesw5
RUN perl -MCPAN -e 'install XML::Simple'
ADD requirements.txt /requirements.txt
RUN cat requirements.txt | xargs -n 1 pip install
ADD *.py /usr/local/bin/
ADD scripts /usr/local/bin/scripts
ADD tools /usr/local/bin/tools
ADD defaults /usr/local/bin/defaults
WORKDIR /usr/local/bin
ENTRYPOINT ["python3"]
It works but shows the error as above, so not so much. Said error is not present when using the tool outside of docker. Last time I made a Docker image I just pulled the git repo and added the main .sh script as an ENTRYPOINT and everything worked despite being more complex (see https://github.com/Louis-MG/Metadbgwas).
Why would I need ADD and moving everything ? I added the git folder to the path, why can't I find the scripts ? How is it different from the Metadbgwas image ?

In your first setup, you start in the image root directory / and run git clone to check out the repository into /CAMISIM. You never change the current directory, though, so when you try to run python3 metagenomesimulation.py --help it's looking in / and not /CAMISIM, hence the "not found" error.
You can fix this just by changing the current directory. At any point after you check out the repository, run
WORKDIR /CAMISIM
You should also delete the ENTRYPOINT line. For each of the scripts you could run as a top-level entry point, check two things:
Is it executable; if you ls -l metagenomesimulation.py are there x in the permission listing? If not, on the host system, run chmod +x metagenomesimulation.py and commit to source control. (Or you could RUN chmod ... in the Dockerfile if you really can't change the repository.)
Does it have a "shebang" line? The very first line of the script should be
#!/usr/bin/env python3
If both of these things are true, then you can just run ./metagenomesimulation.py without explicitly saying python3; since you add the directory to $PATH as well, you can probably run it without specifying the ./... file location.
(Probably deleting the ENTRYPOINT line on its own is enough, given that ENV PATH setting, but your script still might be confused by starting up in the wrong directory.)
The long "help" output just suggests to me that the script is expecting a configuration file name as a parameter and you haven't provided it, or else you've repeated the script name in both the entrypoint and command parts of the container command string.

In the end very little was recquired and the original Dockerfile was correct, the same error is displayed anyway, that is due to the script itself.
What was missing was a link to the interpreter, so I could remove the ENTRYPOINT and actually interpret the script instead of having python look for it in its own path. The Dockerfile:
FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
USER root
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN apt-get update && \
apt install -y git python3-pip libxml-simple-perl libncursesw5 && \
git clone https://github.com/CAMI-challenge/CAMISIM.git && \
pip3 install numpy ete3 biom-format biopython matplotlib joblib scikit-learn
ENV PATH="/CAMISIM:${PATH}"
Trying WORKDIR as suggested instead of the PATH yielded an error.

"bin/bash: python: command not found" returned when running docker image

Here is the docker run output:
hausey#ubuntu:~/niso2-jxj934$ docker run niso2-jxj934
Test version: 15:59, Mar 24th 2020
Question 1: Evaluation of expression.
Command failed: /bin/bash -c "python /bin/jxj934.py -question 1 -expr \"(ifleq (ifleq -1.11298616747 1.63619642199 (sub -1.11298616747 -1.11298616747) 1.7699684348) (add (exp -0.822479932786) 1.39992604386) (add -1.11298616747 (exp 0.385042309638)) 0.205973267133)\" -n 10 -x \"-0.168958230447 -0.131749160548 0.0971246476126 1.8706205565 -0.464122426299 2.35887369763 -0.375948313434 -0.613901105864 0.411326743135 -0.149276696072\"" Exit status: exited with code 127 stderr: /bin/bash: python: command not found
Here is the Dockerfile:
FROM pklehre/niso2020-lab2-msc
ADD jxj934.py /bin
CMD ["-username","jxj934", "-submission", "python /bin/jxj934.py"]
Here is check for python:
hausey#ubuntu:~/niso2-jxj934$ which python
/usr/bin/python
Is that related to the PATH of python?

Usually, it is related to the value of PATH but, specifically, that image only has python3. In other words, looking through the filesystem with
find / -name -type f "python*"
Look for regular files named "python*" in /
There were only python3 results.
...
/usr/bin/python3.8
/usr/bin/python3.7
...
A quick solution is to specify python3 in your CMD line (python3 /bin/jxj934.py). Another is to add a soft link (ln -s /usr/bin/python /usr/bin/python3.8). The best solution is to solve it using the package manager. Then again, that depends if you're in control of the Dockerfile + image.
When you queried which python, you did so on your local machine. The container runs in a different filesystem namespace than yours and with a completely different terminal. The container will behave differently than your machine and any such investigations will yield relevant results only when run within the container.
A little unrelated to your question but it might serve you.
docker run has a --entrypoint option that allows you to override the image's entrypoint. You can ask for bash and explore the container.
docker run --it --entrypoint=bash pklehre/niso2020-lab2-msc
Note that bash has to be in the $PATH.

how to parse volume in dockerfile for an application to be pushed on docker hub

FROM python:3
WORKDIR /Users/vaibmish/Documents/new/graph-report
RUN pip install graphreport==1.2.1
CMD [ cd /Users/vaibmish/Documents/new/graph-report/graphreport_metrics ]
CMD [ graphreport ]
THIS IS PART OF DCOKERFIILE
i wish to remove cd volumes from tha file and have a command like -v there so that whoever runs that can give his or her own volume path in same

The line
CMD [ cd /Users/vaibmish/Documents/new/graph-report/graphreport_metrics ]
is wrong. You achieve the same with WORKDIR:
WORKDIR /Users/vaibmish/Documents/new/graph-report/graphreport_metrics
WORKDIR creates the path if it doesn't exist and then changes the current directory to that path (same as mkdir -p /path/new && cd /path/new)
You can also declare the path as a volume and instruct who runs the container to provide their own path (docker run -v host_path:container_path ...)
VOLUME /Users/vaibmish/Documents/new/graph-report
A final note: It looks like these paths are from the host. Remember that the paths in the Dockerfile are not host paths. They are paths inside the container.

Typical practice here is to pick some fixed path inside the Docker container. It should be a different path from where your application is installed; it does not need to match any particular host path at all.
FROM python:3
RUN pip3 install graphreport==1.2.1
WORKDIR /data
CMD ["graphreport"]
docker build -t me/graphreport:1.2.1 .
docker run --rm \
-v /Users/vaibmish/Documents/new/graph-report:/data \
me/graphreport:1.2.1
(Remember that only the last CMD has an effect, and if it's not a well-formed JSON array, Docker will interpret it as a shell command. What you show in the question would run the test(1) command and not the program you're installing.)
If you're trying to install a single package from PyPI and just run it on local files, a Python virtual environment will be much easier to set up than anything based on Docker, and will essentially work as you expect:
python3 -m venv graphreport
. graphreport/bin/activate
pip3 install graphreport==1.2.1
cd /Users/vaibmish/Documents/new/graph-report
graphreport
deactivate # switch back to system Python/pip
All of the installed Python code is inside the graphreport virtual environment directory, and if you don't need this application again, you can just delete the directory tree.

Implement Git hook - prePush and preCommit

Could you please show me how to implement git hook?
Before committing, the hook should run a python script. Something like this:
cd c:\my_framework & run_tests.py --project Proxy-Tests\Aeries \
--client Aeries --suite <Commit_file_Name> --dryrun
If the dry run fails then commit should be stopped.

You need to tell us in what way the dry run will fail. Will there be an output .txt with errors? Will there be an error displayed on terminal?
In any case you must name the pre-commit script as pre-commit and save it in .git/hooks/ directory.
Since your dry run script seems to be in a different path than the pre-commit script, here's an example that finds and runs your script.
I assume from the backslash in your path that you are on a windows machine and I also assume that your dry-run script is contained in the same project where you have git installed and in a folder called tools (of course you can change this to your actual folder).
#!/bin/sh
#Path of your python script
FILE_PATH=tools/run_tests.py/
#Get relative path of the root directory of the project
rdir=`git rev-parse --git-dir`
rel_path="$(dirname "$rdir")"
#Cd to that path and run the file.
cd $rel_path/$FILE_PATH
echo "Running dryrun script..."
python run_tests.py
#From that point on you need to handle the dry run error/s.
#For demonstrating purproses I'll asume that an output.txt file that holds
#the result is produced.
#Extract the result from the output file
final_res="tac output | grep -m 1 . | grep 'error'"
echo -e "--------Dry run result---------\n"${final_res}
#If a warning and/or error exists abort the commit
eval "$final_res" | while read -r line; do
if [ $line != "0" ]; then
echo -e "Dry run failed.\nAborting commit..."
exit 1
fi
done
Now every time you fire git commit -m the pre-commit script will run the dry run file and abort the commit if any errors have occured, keeping your files in the stagin area.

I have implemented this in my hook. Here is the code snippet.
#!/bin/sh
#Path of your python script
RUN_TESTS="run_tests.py"
FRAMEWORK_DIR="/my-framework/"
CUR_DIR=`echo ${PWD##*/}`
`$`#Get full path of the root directory of the project under RUN_TESTS_PY_FILE
rDIR=`git rev-parse --git-dir --show-toplevel | head -2 | tail -1`
OneStepBack=/../
CD_FRAMEWORK_DIR="$rDIR$OneStepBack$FRAMEWORK_DIR"
#Find list of modified files - to be committed
LIST_OF_FILES=`git status --porcelain | awk -F" " '{print $2}' | grep ".txt" `
for FILE in $LIST_OF_FILES; do
cd $CD_FRAMEWORK_DIR
python $RUN_TESTS --dryrun --project $CUR_DIR/$FILE
OUT=$?
if [ $OUT -eq 0 ];then
continue
else
return 1
fi
done

Dockerfile - Can't use another name than 'app.py'

I've been searching a lot for the past days reagarding Dockerfile. I'm using cx_Oracle in python 2.7. Here's how my Dockerfile looks like:
FROM sbanal/python-oracle-xe12.1-latest
WORKDIR /code/app
COPY generate_distance.py /code/app/app.py
COPY generate_values.py /code/app/app2.py
To make it easier to explain, I've made a method to print out the name of the file. In generate_distance.py:
def test():
print "Generate distance"
test()
In generate_values.py:
def test():
print "Generate values"
test()
Then I'm running docker build with a tag:
docker build -t gen .
Sending build context to Docker daemon 13.82kB
Step 1/4 : FROM sbanal/python-oracle-xe12.1-latest
---> 723335924016
Step 2/4 : WORKDIR /code/app
---> Using cache
---> 9fde6fb3ac02
Step 3/4 : COPY generate_distance.py /code/app/app.py
---> 1dbf7ef85ee3
Removing intermediate container ae626dcef48c
Step 4/4 : COPY generate_values.py /code/app/app2.py
---> 7a54500b88a3
Removing intermediate container f496edfc237d
Successfully built 7a54500b88a3
Successfully tagged gen:latest
When running 'docker images', I can see the 'gen' image. But when I run the 'gen' image, only app.py is working:
>docker run -p 5500:5000 gen
>Generate distance
I can't see what mistake I've done. I also don't know why it has to be called app.py. If I use different file name during COPY in Dockerfile, I get 'No such file or directory' error. That is:
FROM sbanal/python-oracle-xe12.1-latest
WORKDIR /code/app
COPY generate_relation_distance.py /code/app/generate_relation_distance.py
COPY generate_ten_values.py /code/app/generate_ten_values.py
Build and run like the part over:
docker run -p 5500:5000 gen
python: can't open file 'app.py': [Errno 2] No such file or directory
Hope someone can help me :)

Your image is based on sbanal/python-oracle-xe12.1-latest (first line of your Dockerfile).
In this Dockerfile is a "CMD" defined, which specifies the first command of your container. Here, that is
CMD python app.py
(see last line of your base image).
The command will be executed as sh -c "python app.py".
This is why your Dockerfile starts on container creation python app.py
You need to override the "CMD" part in your Dockerfile, e.g.
CMD ["python", "app2.py"]
See the official docker docs to understand CMD.
You should only have one CMD in your Dockerfile containing the first command, which is automatically executed by the container.
If you want to start multiple services, you should consider, if this should really be packed into one image. Or you follow the official docs and consider using a supervisor or a script, which starts your desired services.

If anyone wondering how I solved the problem, I used bash script instead.
script.sh:
#Build the image
docker build -t image_name .
#Run image image_name in container
docker run -d -p 550:500 image_name tail -f /dev/null
#Get the container id
container_id="$(docker ps | grep $IMAGE_NAME | grep -Eo '^[^ ]+')"
#Run your program in container
docker exec -it container_id /bin/sh -c "python generate_distance.py"
docker exec -it container_id /bin/sh -c "python generate_values.py"
My goal was to store the output in text files and copy those from docker container to localhost.