I am new to the deployment process of python, I have been reading about things like Docker (I know a little bit about Docker) but I haven't really found any concrete guides to deploying python scripts. (I see a lot of examples for flask applications and django applications) My exact use case is a Kafka Consumer that needs to run 24/7. Would I just drop my project on an EC2 and run the docker command or the python script from the command line? if not how can I deploy my python script?
edit: I have tried writing a Dockerfile that just runs my script. Which means I would just drop my project on an EC2 and run the docker file? This sounds wrong to me, but im not entirely sure.
Related
I'm hosting a website on AWS. Its a web interface with a SQL database. The website will be used to:
1. View results of query from Database
2. Insert data into database
3. View the data and update it where needed.
The codes and connections works file when I run the application on localhost (Apache on my C drive). But we want to host it on AWS so that people around me can use it.
So, In AWS I uploaded the code on EC2 and installed apache on it, all the html links are working but the python file is simply displaying the code.
I'm guessing it has something to do with the shebang. Currently my code has the following shebang:
#!C:\Python27\python.exe
Can someone guide me if its the shebang or if there is something else i need to do.
I have installed boto, but not sure what to do next. The AWS website and most of the forums talk about using Elastic Beanstalk. I want to host a fully functioning Python webApp on AWS without using Elastic Beanstalk.
When apache displays code, that is a clear sign that Apache is not configured properly to execute python. You should look to see if mod_python is installed and configured correctly.
Also, #! is generally used with Linux not windows. If apache/mod_python is installed and configured correctly I can't imagine what code you'd have that would need #! since the .py extension would be enough.
IF your EC2 instance is indeed running Linux, and your code does indeed need #! try:
#!/bin/python
OR
#!/usr/local/bin/python
(Depends on where the python binary is, and those are the most common locations.)
If your EC2 instance is running Windows then "Unless you are using cygwin, windows has no shebang support"
Hi have you logged into your EC2 instance through the endpoint and then run your script, from the command line. I have some experience with EC2 running apache2 only my application was written in Java, having previously used python scripts I was able to run them by logging into my EC2 instance, you can do this from AWS management console. hope this helps you somewhat.
I currently have a handful of small Python scripts on my laptop that are set to run every 1-15 minutes, depending on the script in question. They perform various tasks for me like checking for new data on a certain API, manipulating it, and then posting it to another service, etc.
I have a NAS/personal server (unRAID) and was thinking about moving the scripts to there via Docker, but since I'm relatively new to Docker I wasn't sure about the best approach.
Would it be correct to take something like the Phusion Baseimage which includes Cron, package my scripts and crontab as dependencies to the image, and write the Dockerfile to initialize all of this? Or would it be a more canonical approach to modify the scripts so that they are threaded with recursive timers and just run each script individually in it's own official Python image?
No dude just install python on the docker container/image, move your scripts and run them as normal.
You may have to expose some port or add firewall exception but your container can be as native linux environment.
The thing is, I read this post stating best practices to set up a code to run at every specified interval over a period of time using the python library - APS Scheduler. Now, it obviously works perfectly fine if I do it on a test environment and run the application from the command prompt.
However, I come from a background where most my projects are university level and never ran in production but for this one, I would like to. I have access to AWS and can configure any kind of server on AWS and I am open to other options as well. It would be great if I could get a headstart on what to look if I have to run this application as a service from a server or a remote machine without having to constantly monitoring it and providing interrupts on command prompt.
I do not have any experience of running Python applications in production so any input would be appreciated. Also, I do not know how to execute this code in production (except for through aws cli) but that session expires once I close my CLI so that does not seem like the most appropriate way to do it so any help on that end would be appreciated too.
The Answer was very simple and does not make a lot of sense and might not be applicable to all.
Now, what I had was a python flask application so I configured the app in a virtual environment using eb-virt on the aws server and then created an executable wsgi script which I then ran as a service using mod_wsgi plugin for the apache http server and then I was able to run my app.
I am currently trying to insert Docker in my Python development workflow of non-web applications.
What are the current best practices in Python development using Docker and an IDE?
I need the possibility to isolate my environments with Docker and debug my code.
On the web I found many articles about the use of Docker to deploy your code:
Production deployments: how to build Docker images ready to spin with your application already packaged inside
Development environments that mirror production: extension of the above, where you can use a container to fully QA the current status of a project before deploying to production while developing
I found a lot less about an actual development workflow, apart from some tips on how to use containers with shared volumes mapped to the directories on the host while developing web applications. This approach does not apply to non-web applications and it has some issues where a simple reload (with a LiveReload-like mechanism) is not enough so you need to restart your container(s).
The closest writing I could find is this "Eight Docker Development Patterns" blog post, but it does not consider an IDE (like PyCharm I am using now).
Maybe this question is the result of the 3-4 hours (and counting) spent configuring PyCharm to use a remote Python interpreter running in a Docker container. I expected a much better integration between the two.
Actually, I believe that using the Docker interpreter in PyCharm is the way to go. Which version of PyCharm do you have? If you have the 2016 version, it should be set up within seconds. You just have to make sure your docker machine is running and you must have your image built that you would like to use with your project. PyCharm will find the Docker machine in the "add remote interpreter" dialog automatically. Then select your image and you're all set up.
You can run your code as usual then, almost without any delay.
Here's what worked for me: https://www.jetbrains.com/help/pycharm/2016.1/configuring-remote-interpreters-via-docker.html
And make sure to update PyCharm, that solved some issues I had.
I have experience coding PHP and Python but have until recently not used Python for much web development.
I have started a project using mod_python with Python Server Pages which does a very good job of what I need it to do. Amongst the advantages of this is performance; the web server does not spawn a new interpreter for each request.
The system will finally be deployed on a server where I am able to setup /etc/apache/conf.d correctly, etc.
However, during development and for automated regression testing I would like the ability to run the .psp scripts without having to serve using an Apache instance. For example, it is possible to run PHP scripts using the PHP cli, and I would like to run my .psp scripts from the shell and see the resulting HTTP/HTML on stdout.
I suppose this mode would need to support simulation of POST requests, etc.
Update
OK, after typing all that question I discovered the mod_python command line in the manual:
http://modpython.org/live/current/doc-html/commandline.html
I guess that would get me most of the way there, which is to be able to exercise my application as a local user without deploying to an Apache server.
I am leaving this question here in case anyone does a web search like I did.
The mod_python command line tool is one way to do this.
See http://modpython.org/live/current/doc-html/commandline.html
Essentially,
mod_python create /path/to/new/server_root \
--listen 8888 \
--pythonpath=/path/to/my/app \
--pythonhandler=mod_python.wsgi \
--pythonoption="mod_python.wsgi.application myapp.wsgi::application"
sets up a skeleton app.
Then you run it:
mod_python start /path/to/new/server_root/conf/httpd.conf
wget http://localhost/path/to/my/app:8888
I am still testing this...
Caveat This function only seems to be from 3.4.0