I wrote a Python script that analyzes and filters stock data. I write the stock data into a mongodb. The result is a CSV file with the filtered values in it.
Is it now possible to create a docker container that contains Python & mongodb and copies the CSV from the container to the host?
I tried creating a Dockerfile with python only. But when it comes to adding the mongodb service and exporting the file to the host i am a little overstrained.
My goal is that at the end I'll have one Docker container that runs the python script & exports the file to the host.
Do you know any best practice? Or a good tutorial that covers my needs?
I would not recommend python and mongodb to be installed on the same docker container. Usually db and the app should be installed on separate containers using docker-compose. But still, if you want them to be on same containers, then you can do so by using an Ubuntu ( or anything else you are comfortable with) image installing mongodb and python on it and then running your scripts. I found this following git repo that contains one such Dockerfile.
Regarding copying CSVs from dockert to host machine, you can do so by using volumes, if you want to use docker-compose which I would totally recommend, or you can use docker cp command to get the data manually from docker to host.
Related
I'm new to using docker, and I was wondering how I could create a docker image for others to use to replicate the exact same setup I have for python when testing code. Currently what I've done is I created a docker image with the dockerfile below:
FROM python:3.7
COPY requirements.txt ./
RUN pip install -r requirements.txt
where the requirements text file contains the libraries and the versions I am using. I then ran this image on my local machine, and things looked good, so I pushed this image to a public repository on my docker hub account, so other people could pull it. This makes sense so far, but I'm still a little confused what to do next. If someone wants to test a python script on their local machine using my configuration what exactly would they do? I think they would pull my image from docker hub and then create a container from this image on their local machine. However, I tried this on my machine, testing a python file that runs and saves a file, and it's not really working how I anticipated. I can make a python file and put it in WORKDIR and have it run, but I think that's all it does. I'd really like to be able to navigate the container to find the file the python program created and then save it back on my local machine. Is there any way to do this, and am I going about this the wrong way?
you can execute bash in the container and check the files created inside of the container. Or you can share a volume between host and container. Such that you can check the files created in your host machine. Check this link, it might be helpful for you :-)
Volume
I would like to run docker-compose via python docker sdk.
However I couldn't find any reference on how to achieve this using these reference Python SDK? I could also use subprocess but I have some other difficulty while using that. see here docker compose subprocess
I am working on the same issue and was looking for answers, but nothing so far. The best shot I can give it is to simplify that docker-compose logic. For example, you have a YAML file with a network and services - create them separately using Python Docker SDK and connect containers to a network.
It gets cumbersome, but eventually you can get things working that way from Python.
I created a package to make this easy: python-on-whales
Install with
pip install python-on-whales
Then you can do
from python_on_whales import docker
docker.compose.up()
docker.compose.stop()
# and all the other commands.
You can find the source for the package in my GitHub repository: https://gabrieldemarmiesse.github.io/python-on-whales/
The stack I selected for my project are Python, R and MongoDB. However, I'd like to adopt Docker for this project but when I did my research on the internet, I pretty much found example for MySQL with PHP or Wordpress. So, I'm curious to know where I can find tutorials or example for using containers with Python, R, and MongoDB or any idea on how to put them together. What will the Dockerfile will be like? Especially, in my project, R used for data processing and data visualisation will be called from Python used for data collector as a sub-module for data cleaning as well.
Any help will be appreciated.
Option 1:
Split them in multiple docker images and run them all using docker-compose from a YAML that will set them all up easier.
There's probably already an image for each of those services that you can use and just add some code to them using docker volumes. Just look for them at Docker Hub.
Example of use with exiting Python Image are already in its description. It even shows how to create your own Docker Image using a Dockerfile which you need for each image.
Option 2:
You can build just one image using a less specific image (let's say debian/ubuntu), install all interpreters, libraries and other requirements inside and then create an ENTRYPOINT which will call a script that will run each service and keep open to avoid the container finalisation.
I have an application which is split into multiple Docker containers:
Container 1 contains a MongoDB database
Container 2 contains a python script which performs some operations on a daily basis and stores the results in the MongoDB of Container 1 using pymongo.
Previously to using Docker, i.e., having the MongoDB and the python app on the same machine, I could use mongoexport right after the daily scripts finished to backup my database. However, in the Docker setup, I can not use mongoexport in Container 2 because MongoDB is not installed, i.e., the command is unknown.
From my point of view, the only option would be using a cronjob in Container 1 running a script which performs the backup on a preset time.
However, I would prefer a solution in which Container 2 triggers the backup since the runtime of the daily scripts can vary considerably.
Thanks in advance for any suggestions!
You can download mongodb binaries on docker 2 form here. That way you can get rid of the command is unknown
You can export mongodb collection from mongodb running on container 1 using mongoexport or take dump using mongodump from container 2 using --host and --port options.
Note: mongoexport does not export indexes from collection.
I'm trying to "dockerize" my java web application and finally run the docker image on EC2.
My application is a WAR file and connects to a database. There is also a python script which the application calls via REST. The python side uses the tornado webserver
Question 1:
Should I have the following Docker containers?
Container for Application Server (Tomcat 7)
Container for HTTP Server (nginx of httpd)
Container for postgres db
Container for python script (this will have tornado web server and my python script).
Question 2:
What is the best way to build dockerfile? I will have to do trial and error for what commands need to be put into the dockerfile for each container. Should I have an ubuntu VM on which I do trial and error and once I nail down which commands I need then put them into the dockerfile for that container?
That list looks about right.
The advantage of splitting up your stack to separate containers is that you can (in many cases) use off-the-shelf official images, and only have to provide the right configuration to make them work together. In addition, you'd be able to upgrade the components (containers) separately.
Note that combining multiple services in a single container is not forbidden, but in Docker it's overall best practice to separate concerns, and have a
single container only be responsible for a single task/service.
To get all containers started with the right configuration, docker-compose is
a good choice; it enables you to create a single file (docker-compose.ymlhttps://docs.docker.com/compose/compose-file/) that
describes your project; which images to build for each container, how the containers relate to each-other, and pass configurations to them.
With docker-compose you can then start all containers by simply running
docker-compose up -d
You can use Docker Machine to create a Docker development environment on Mac or Windows. This is really good for trial and error. There is no need to for Ubuntu VM.
Docker container does one thing only. So your application would consist of multiple containers, one for each component. You've also clearly identified the different containers for your application. Here is how the workflow might look like:
Create a Dockerfile for Tomcat container, nginx, postgres, tornado
Deploy the application to Tomcat in Dockerfile or by mapping volumes
Create image for each of the container
Optionally push these images to Docker hub
If you plan to deploy these containers on multiple hosts then create an overlay network
Use Docker Compose to start these containers together. It would use the network created previously. Alternatively you can also use --x-networking for Docker Compose to create the network.