Docker with Jupyter Notebook and Spark - python

I am trying to use docker to run Jupyter Notebook with spark , on PowerShell I am using the following command:
docker run --rm -h 0.0.0.0 -p 8888:8888 -e PYTHONHASHSEED=0 jupyter/all-spark-notebook
When I run that I get the following error:
Container must be run with group "root" to update passwd file
Any Ideas about how to fix it/ update group root password?

Related

container: volume dockerfile does not work

I m not able to launch new jupyter-notebook from my project.
Below my dockerfile.
FROM python:3.9.0
ARG WORK_DIR=/opt/dir1
RUN apt-get update && apt-get install cron -y && apt-get install -y default-jre
# Install python libraries
COPY requirements.txt /tmp/requirements.txt
RUN pip install --upgrade pip && pip install -r /tmp/requirements.txt
WORKDIR $WORK_DIR
EXPOSE 8888
# Copy etl code
# copy code on container under your workdir "/opt/dir1"
COPY . .
ENTRYPOINT ["sh", "-c"]
CMD ["jupyter-notebook --ip 0.0.0.0 --no-browser --allow-root]
VOLUME /home/data/dir1/
then in my terminal i did
#build
docker build -t my-python-app .
#run
docker run -it -p 8888:8888 my-python-app
#in container i did
jupyter notebook --ip 0.0.0.0 --no-browser --allow-root
I think that my VOLUME doesn't work because when i did modifications in file of container nothing happens in the host /home/data/dir1/.
Does anyone knows why and how to solve it?
You can use, docker run -it /bin/bash (image name)
and try to navigate to the folder you have set the volume to, and see if an error occurs and check permissions.
When using volumes, check on the host system you can access the folder. Afterwards check which user you are, Docker allows to parse your USER_ID and GROUP_ID to the container.
From there you can use the same user and group as you are on the host system. If you wanted to access the same folder on the host system, you can enter into permissions problems.
More information on this on the following webpage.
https://jtreminio.com/blog/running-docker-containers-as-current-host-user/
Maybe it's could be help you : try another way to start the volume inside container, for example add it within "run" command container creation. i've working with docker and i've never added volumes in this way (this doesn't mean you way is wrong)
Here two examples to working with volumens,I recommend you second link.
docker official docs , working with volumes example

bash: ./run: No such file or directory

Im new to docker.
I am starting the run command with a script called r, which has the following code
proxy="--build-arg http_proxy=http://wwwcache.open.ac.uk:80 --build-arg https_proxy=http://wwwcache.open.ac.uk:80"
if [ "$http_proxy" == "" ]; then
proxy=
fi
docker build $proxy -t bi-tbcnn docker
docker run -v $(pwd):/e -w /e --entrypoint bash --rm -it bi-tbcnn -c ./run
When I execute r I am getting the following error
bash: ./run: No such file or directory
but when I directly execute the ./run command on my terminal is ok
I use Docker Toolbox on windows
The project address is https://github.com/bdqnghi/bi-tbcnn
thanks
This is a known issue on docker for windows
https://blogs.msdn.microsoft.com/stevelasker/2016/09/22/running-scripts-in-a-docker-container-from-windows-cr-or-crlf/
it seems you're facing an issue with Carriage Return(CR) and Line Feeds(LF) characters, maybe your code editor is changing the newline format automatically
can you to try open a bash session on the container and execute the script manually?
docker run -v $(pwd):/e -w /e --entrypoint bash --rm -it bi-tbcnn
root#a83fcd779f8e:/e# ./run
Please paste the output here

Running Jupyter notebook in Docker

I want to run jupyter in docker container. I am not able to launch the jupyter notebook. When I copy paste the URL given in the terminal.. server cannot be reached. Will appreciate any ideas to try
You are forwarding port 8080 in the docker run call with -p 8080:8080. But you also need to forward port 8888 by adding -p 8888:8888. More specifically, you want to run:
docker -it -p 8080:8080 -p 8888:8888 jupyter/minimal-notebook
First thing is Jupyter nootbook runs on port 8888. If you want to access the notebook on a diff port on your host you should map it like this -p 80:8888.
If you don't mind using the defaults, you should use run the following command.
Run this command: docker run -p 8888:8888 jupyter/minimal-notebook. Then
replace the host name in the url given in terminal with localhost like this
http://localhost:8888/\?token\=<TOKEN>\&token\=<TOKEN>
This should work.
Note: If you map it to a diff port, you should replace it in the url you get in the terminal. Ex. http://localhost:80/\?token\=<TOKEN>\&token\=<TOKEN>

Dockerized web-service does not run in background even with detached flag

I'm trying to Dockerize a web service using Tangelo and python.
My project structure is as follows:
test.py
requirements.txt
Dockerfile
test.py
import ...
def run(query):
...
return response
requirements.txt
... # other packages, numpy, open-cv, etc
tangelo
Dockerfile
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y python python-pip git
EXPOSE 9220
ADD . /test
WORKDIR /test
RUN pip install -r requirements.txt
CMD "tangelo --port 9220"
I build this using
docker build -t "test" .
And run in detached mode using
docker run -p 9220:9220 -d "test"
But docker ps shows me that the docker stops almost as soon as it has started. I don't know what the problem is since I cannot inspect the logs.
I have tried a lot of things but I still can't figure this thing out.
Any ideas? If needed, I can provide more info.
EDIT:
When I build, step 8 says
Step 8/8 : ENTRYPOINT tangelo --port 9220
---> Running in 8b54841853ab
Removing intermediate container 8b54841853ab
So it means these are run in an intermediate container. Why is that and how can I prevent it?
TL;DR: Use:
CMD tangelo -np --port 9220
Instead of:
CMD "tangelo --port 9220"
Explanation:
You have two ways to debug the problem:
Inspect the logs of the container:
$ docker run -d test
28684015e519c0c8d644fccf98240d1465acabab6d16c19fd59c5f465b7f18af
$ sudo docker logs 28684015e519c
/bin/sh: 1: tangelo --port 9220: not found
Instead of running in detached mode, run in foreground with -i/--interactive (and optionally also -t/--tty):
$ docker run -ti test
/bin/sh: 1: tangelo --port 9220: not found
As you can see from above, the problem is that tangelo --port 9220 is being interpreted as a single argument. Split it by removing quotes:
CMD tangelo --port 9220 # this will use a shell
or use the "exec" form (preferred, given that you don't need any shell features):
CMD ["tangelo", "--port", "9220"] # this will execute tangelo directly
or even better use ENTRYPOINT + CMD:
ENTRYPOINT ["tangelo"]
CMD ["--port", "9220"] # this will execute tangelo directly
After this change, you'll still have a problem:
$ sudo docker run -ti test
...
[29/Apr/2018:02:43:39] TANGELO no such group 'nobody' to drop privileges to
Tangelo is complaining about the fact that there is no user and group named nobody inside the container. Again, there are two things you can do: add a RUN to create the nobody user and group, or run Tangelo with the -np/--no-drop-privileges option:
ENTRYPOINT ["tangelo"]
CMD ["--no-drop-privileges", "--port", "9220"]
It's fine if during the build you see intermediate containers: Docker creates them for each build step. The commands you specify in ENTRYPOINT or CMD are not executed during build, they're just recorded into the final image.

How to execute a local python script into a docker from another python script?

Let me clarify what I want to do.
I have a python script in my local machine that performs a lot of stuff and in certain point it have to call another python script that must be executed into a docker container. Such script have some input arguments and it returns some results.
So i want to figure out how to do that.
Example:
def function()
do stuff
.
.
.
do more stuff
''' call another local script that must be executed into a docker'''
result = execute_python_script_into_a_docker(python script arguments)
The docker has been launched in a terminal as:
docker run -it -p 8888:8888 my_docker
You can add your file inside docker container thanks to -v option.
docker run -it -v myFile.py:/myFile.py -p 8888:8888 my_docker
And execute your python inside your docker with :
py /myFile.py
or with the host:
docker run -it -v myFile.py:/myFile.py -p 8888:8888 my_docker py /myFile.py
And even if your docker is already running
docker exec -ti docker_name py /myFile.py
docker_name is available after a docker ps command.
Or you can specify name in the run command like:
docker run -it --name docker_name -v myFile.py:/myFile.py -p 8888:8888 my_docker
It's like:
-v absoluteHostPath:absoluteRemotePath
You can specify folder too in the same way:
-v myFolder:/customPath/myFolder
More details at docker documentation.
You can use docker's python SDK library. First you need to move your script there, I recommend you do it when you create the container or when you start it as Callmemath mentioned:
docker run -it -v myFile.py:/myFile.py -p 8888:8888 my_docker
Then to run the script using the library:
...
client = docker.client.from_env()
container = client.containers.get(CONTAINER_ID)
exit_code, output = container.exec_run("python your_script.py script_args")
...
you have to use docker exec -it image_name python /filename
Note: To use 'docker exec' you must run the container using docker run

Categories