I have read this question and I understand that you need some kind of foundation to build your docker image. However, I still don't see the purpose of docker images like python.
Why do I need this:
FROM python:latest
when I can just do that:
FROM ubuntu
RUN apt install python3
Say I want to run a container where a python server is hosted using apache. What would be the difference between
Using the apache base image and installing python manually
Using the python base image and installing apache manually
Using the ubuntu base image and installing both manually
The difference is slim in the given example because in the end you will get the same thing but using slightly different commands.
Things change when you need to use either latest or specific version of the software. The required version may not be available in standard Ubuntu repositories or may come with a delay.
What you get from using python or apache2 as a base is the ability to choose the version you need with just one line of code as soon as it's published.
More significantly, there may be no need to combine python and apache. Docker containers are usually made to host a single process and it is more common to have a python backend in one container and a web-server as a proxy in another.
In this case you don't care about installing apache at all, you just mount its config into the container at runtime. Eliminating the web server you only need to focus on the application and its dependencies, so in the end you will have less code and easier time maintaining it.
Related
I have a python script that runs locally via a scheduled task each day. Most of the time, this is fine -- except when I'm on vacation and the computer it runs on needs to be manually restarted. Or when my internet/power is down.
I am interested in putting it on some kind of rented server time. I'm a totally newbie at this (having never had a production-type process like this). I was unable to find any tutorials that seemed to address this type of use case. How would I install my python environment and any config, data files, or programs that the script needs (e.g., it does some web scraping and uses headless chrome w/a defined user profile).
Given the nature of the program, is it possible to do or would I need to get a dedicated server whose environment can be better set up for my specific needs? The process runs for about 20 seconds a day.
setting up a whole dedicated server for 20s worth of work is really a suboptimal thing to do. I see a few options:
Get a cloud-based VM that gets spin up and down only to run your process. That's relatively easy to automate on Azure, GCP and AWS.
Dockerize the application, along with the whole environment and running it as an image on the cloud - e.g. on a service like Beanstalk (AWS) or App Service (Azure) - this is more complex, but should be cheaper as it consumes less resources
Get a dedicated VM (droplet?) on a service like Digital Ocean, Heroku or pythonanywhere.com - dependent upon the specifics of your script, it may be quite easy and cheap to set up. This is the easiest and most flexible solution for a newbie I think, but it really depends on your script - you might hit some limitations.
In terms of setting up your environment - there are multiple options, with the most often used being:
pyenv (my preferred option)
anaconda (quite easy to use)
virtualenv / venv
To efficiently recreate your environment, you'll need to come up with a list of dependencies (libraries your script uses).
A summary of the steps:
run $pip freeze > requirements.txt locally
manually edit the requirements.txt file by removing all packages that are not used by your script
create a new virtual environment via pyenv, anaconda or venv and activate it wherever you want to run the script
copy your script & requirements.txt to the new location
run $pip install -r requirements.txt to install the libraries
ensure the script works as expected in its new location
set up the cornjob
If the script only runs for 20 seconds and you are not worried about scalability, running it directly on a NAS or raspberry could be a solution for a private environment if you have the hardware on hand.
If you don’t have the necessary hardware available, you may want to have a look at PythonAnywhere which offers a free version.
https://help.pythonanywhere.com/pages/ScheduledTasks/
https://www.pythonanywhere.com/
However, in any professional environment I would opt for a tool like Apache Airflow. Your process of “it does some web scraping and uses headless chrome w/a defined user profile” describes an ETL workflow.
https://airflow.apache.org/
I am starting the Docker tutorial with the Python app and would like to know what is the meaning of "Python runtime" in this context:
In the past, if you were to start writing a Python app, your first order of business was to install a Python runtime onto your machine. But, that creates a situation where the environment on your machine has to be just so in order for your app to run as expected; ditto for the server that runs your app.
With Docker, you can just grab a portable Python runtime as an image,
no installation necessary. Then, your build can include the base
Python image right alongside your app code, ensuring that your app,
its dependencies, and the runtime, all travel together.
So I guess what it mean is the "Python runtime" is like the configuration of your local Python.
When it says "Python runtime," it just means the Python interpreter in a binary format and its local configuration of package dependencies. Your interpretation is correct.
I am currently trying to insert Docker in my Python development workflow of non-web applications.
What are the current best practices in Python development using Docker and an IDE?
I need the possibility to isolate my environments with Docker and debug my code.
On the web I found many articles about the use of Docker to deploy your code:
Production deployments: how to build Docker images ready to spin with your application already packaged inside
Development environments that mirror production: extension of the above, where you can use a container to fully QA the current status of a project before deploying to production while developing
I found a lot less about an actual development workflow, apart from some tips on how to use containers with shared volumes mapped to the directories on the host while developing web applications. This approach does not apply to non-web applications and it has some issues where a simple reload (with a LiveReload-like mechanism) is not enough so you need to restart your container(s).
The closest writing I could find is this "Eight Docker Development Patterns" blog post, but it does not consider an IDE (like PyCharm I am using now).
Maybe this question is the result of the 3-4 hours (and counting) spent configuring PyCharm to use a remote Python interpreter running in a Docker container. I expected a much better integration between the two.
Actually, I believe that using the Docker interpreter in PyCharm is the way to go. Which version of PyCharm do you have? If you have the 2016 version, it should be set up within seconds. You just have to make sure your docker machine is running and you must have your image built that you would like to use with your project. PyCharm will find the Docker machine in the "add remote interpreter" dialog automatically. Then select your image and you're all set up.
You can run your code as usual then, almost without any delay.
Here's what worked for me: https://www.jetbrains.com/help/pycharm/2016.1/configuring-remote-interpreters-via-docker.html
And make sure to update PyCharm, that solved some issues I had.
Hey everyone,
I am expanding my team and I have recently added an additional front end engineer on my site. I am currently using django to run my site but my site is using a lot of plugins, namely: django-celery, django-mailer, django-notification, and django-socialregistration.
Let me describe my situation:
He is using Mac OS X, and I have no experience in installing stuff on mac os X or configuration on that platform
I believe that getting my backend to run on his computer might be somewhat troublesome, i.e. I have to install a bunch of plugins (which are not available on pip or easy_install as they are the latest version) and I have also made heavy modification to django-socialregistration which I am currently using by symlinking to the modified code in my repos in my python path
I tried to look into solutions like pip and easy_install but I have not been able to get them to install code from github
I think the easiest way is to get my backend working on his computer and then he just commiting to the repos. Any ideas how I can make this easy?
Another, free option, is to use VirtualBox. I would recommend installing the same OS on it as your production server. Then, he's developing in the same environment as the live site, and can just check into the repo the same as you. Hey, you may want to do the same on your end--then both of your environments are the same and also the same as the live site.
Get him to set up a virtual machine on his Mac, using VMWare Fusion or Parallels, running the same operating system that you currently use for your back end. If he prefers developing using Mac tools he can do still that by sharing his local changes to the virtual machine via a shared directory.
An alternative, if that's possible, would be to set up a testing/development environment on a machine with an OS you're familiar with, then install something like Dropbox on his local machine where he can develop the frontend code, and install Dropbox on that other environment with the backend components. Dropbox would sync his local changes to that testing environment for him to run the code on.
That way, he would be able to use that environment to test his code, you wouldn't need to set up a backend on his machine (or keep it up to date) and you'd still be getting the same functionality.
Again, if that's an option.
I've made a small little "application" utilizing Django as a framework. This is an application that is not ment to be deployed to a server but run locally on a machine. Thus the runserver.py works just nice.
I, as an developer is comfortable with fireing up the terminal, running python manage.py runserver and using it.
But I have some Mac OS X and Windows friends wanting to use my application, and they dont have virtualenv, git or anything else.
Is there a way I can package this to be a standalone product? Of course it would depend on Python being installed on the system, but it is possbile to package the virtualenv — with django and everything, and just copy it to another system and make it work?
And maybe even run the runserver in some kind a deamon mode?
Use setuptools and easy_install.
Here's an introductory article.
Yes, you can package it. Django may not be the easiest to do this with, but the principles are the same for other frameworks. You need to make an installer that installs everything you need. And that installer needs to be different for different platforms. such as Windows, Ubuntu, OS X etc. That also means that the answer is significantly different for each platform, and only half of the answer is dependning on Django. :-(
This kinda sucks, but that's life, currently. There is no nice platform independent way to install software for end users.
I also haven't found the perfect solution for this yet.
My current approach is to provide a docker image because that's really easy to use for everyone. This includes an alpine base image because it's tiny and python + django and the app itself. You can also include a webserver like nginx and an app server like uwsgi or gunicorn and expose a port for it.
So in the end your user would just run the container and access the web app under http://localhost:9000/ or something like this.
This is really handy and also my preferred way of trying out some app I've found.
The "proper" way would be do build a package for every OS and distribution you are targeting and a simple zip bundle so people can also install the app manually.
To build the packages I suggest using fpm. It takes most of the pain of doing the packaging with their native tools away. The packages would then depend on a proper application server like uwsgi or gunicorn.
So in the end you could then install it like apt install your-package and it would depend on python-django, uwsgi etc.
For the location and where to put all the files in the package every distribution has their own way of doing it. I prefer putting everything under /usr/share/webapps/myapp/ and having the config under /etc/myapp/config.py or something like that.
For Windows and macOS there are solutions like PyInstaller. I haven't used it yet for a django app but it should do the job. You should include a app server like uwsgi in there too.
In generel you don't want to run the django dev server in a production environment. So keep that in mind when packaging.
I hope that helps a bit.
There are several ways of doing this. I think you are looking more for build tools (which includes packaging) rather than just a Python solution. Here are a couple that I've used in the past:
zc.buildout: Used to build and deploy Python modules and applications, but is also able to work with other languages with a little massaging. Easy to use (for a build tool).
make: The software build classic. Works with practically all languages but a little archaic and hard to learn for a first timer.
The new snap package manager for Linux should suit the task perfectly. It provides all the solutions that were quite a pain for Python apps so far (dependencies, interpreter etc) and at the same time avoiding the complexity of Docker.
These days Docker is probably a good answer
User needs to install Docker first, but it runs on Windows and OSX as well as Linux.
Your Dockerfile takes care of installing all the dependencies and then runs the devserver (or you could even run a proper webserver in the container)