I've built the flask app that designed to run inside docker container. It will
accept POST HTTP Methods and return appropriate JSON response if the header key matched with the key that I put inside docker-compose environment.
...
environment:
- SECRET_KEY=fakekey123
...
The problem is: when it comes to testing. The app or the client fixture of
flask (pytest) of course can't find the docker-compose environment. Cause the app didn't start from docker-compose but from pytest.
secret_key = os.environ.get("SECRET_KEY")
# ^^ the key loaded to OS env by docker-compose
post_key = headers.get("X-Secret-Key")
...
if post_key == secret_key:
RETURN APPROPRIATE RESPONSE
.....
What is the (best/recommended) approach to this problem?
I find some plugins
one,
two,
three to do this. But I asked here if there is any more "simple"/"common" approach. Cause I also want to automate this test using CI/CD tools.
You most likely need to run py.test from inside of your container. If you are running locally, then there's going to be a conflict between what your host machine is seeing and what your container is seeing.
So option #1 would be to using docker exec:
$ docker exec -it $containerid py.test
Then option #2 would be to create a script or task in your setup.py so that you can run a simpler command like:
$ python setup.py test
My current solution is to mock the function that read the OS environment. OS ENV is loaded if the app started using docker. In order to make it easy for the test, I just mock that function.
def fake_secret_key(self):
return "ffakefake11"
def test_app(self, client):
app.secret_key = self.fake_secret_key
# ^^ real func ^^ fake func
Or another alternative is using pytest-env as #bufh suggested in comment.
Create pytest.ini file, then put:
[pytest]
env =
APP_KEY=ffakefake11
Related
Forgive my ignorance..
I'm trying to learn how to schedule python scripts with Google Cloud. After a bit of research, I've seen many people suggest Docker + Google Cloud Run + Cloud Scheduler. I've attempted to get a "hello world" example working, to no avail.
Code
hello.py
print("hello world")
Dockerfile
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.8-slim
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
WORKDIR /app
COPY . /app
# Creates a non-root user with an explicit UID and adds permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers
RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["python", "hello.py"]
Steps
Create a repo with Google Cloud Artifact Registry
gcloud artifacts repositories create test-repo --repository-format=docker \
--location=us-central1 --description="My test repo"
Build the image
docker image build --pull --file Dockerfile --tag 'testdocker:latest' .
Configure auth
gcloud auth configure-docker us-central1-docker.pkg.dev
Tag the image with a registry name
docker tag testdocker:latest \
us-central1-docker.pkg.dev/gormanalysis/test-repo/testdocker:latest
Push the image to Artifact Registry
docker push us-central1-docker.pkg.dev/gormanalysis/test-repo/testdocker:latest
Deploy to Google Cloud Run
Error
At this point, I get the error
The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable.
I've seen posts like this which say to add
app.run(port=int(os.environ.get("PORT", 8080)),host='0.0.0.0',debug=True)
but this looks like a flask thing, and my script doesn't use flask. I feel like i have a fundamental misunderstanding of how this is supposed to work. Any help would be appreciated it.
UPDATE
I've documented my problem and solution in much more detail here ยป
I had been trying to deploy my script as a Cloud Run Service. I should've tried deploying it as a Cloud Run Job. The difference is that cloud run services require your script to listen for a port. jobs do not.
Confusingly, you cannot deploy a cloud run job directly from Artifact Registry. You have to start from the cloud run dashboard.
Your Flask application should be something like below:
import os
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello_world():
return "Hello World!"
if __name__ == "__main__":
app.run(debug=True, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))
See this official documentation for step by step instruction: Deploy a Python service to Cloud Run
There is a plugin called: Cloud Code IDE plugin which makes the test and deployment easy. I am using it for VS code, once the initial setups and permissions are taken care, few clicks, you will be able to run locally, debug and deploy Cloud run services from your local instance.
I have a python script that will take a single argument. The script makes calls to 3rd party APIs, which obviously need to be server-side because of the API keys.
I want to make a website that makes an ajax call (or similar) to run the python script and give it some browser-side data.
I thought this would be a good project to use Docker for, but I can't find anything to indicate that this will work.
Is python not the right tool? Is Docker not the right tool?
Any help is greatly appreciated!
Edit (also in comments)
I've deployed a test image/service/cluster successfully on DigitalOcean, but I don't see how I can configure the endpoint to be RESTful
Edit (for source code)
To test the setup, I've been using a Docker-provided tutorial.
Dockerfile
#use an official Python runtime as a parent image
FROM python:3.6.2-slim
WORKDIR /app
ADD . /app
RUN pip install -r requirements.txt
EXPOSE 80
ENV NAME World
CMD ["python", "app.py"]
docker-compose.yml
version: "3"
services:
web:
# replace username/repo:tag with your name and image details
image: username/repo:latest
deploy:
replicas: 5
resources:
limits:
cpus: "0.1"
memory: 50M
restart_policy:
condition: on-failure
ports:
- "80:80"
networks:
- webnet
networks:
webnet:
app.py
from flask import Flask
from redis import Redis, RedisError
import os
import socket
# Connect to Redis
redis = Redis(host="redis", db=0, socket_connect_timeout=2,
socket_timeout=2)
app = Flask(__name__)
#app.route("/")
def hello():
try:
visits = redis.incr("counter")
except RedisError:
visits = "<i>cannot connect to Redis, counter disabled</i>"
html = "<h3>Hello {name}!</h3>" \
"<b>Hostname:</b> {hostname}<br/>" \
"<b>Visits:</b> {visits}"
return html.format(name=os.getenv("NAME", "world"), hostname=socket.gethostname(), visits=visits)
if __name__ == "__main__":
app.run(host='0.0.0.0', port=80)
It suddenly seems like I probably just need to change #app.route("/")...
Short answer, yes, its possible.
I recommend you first run your Python code outside of a container and try to connect to the droplet before putting in Docker, just to make sure it works.
Once you have the basic connection working, you can move into a container, and build out the REST API. (or GraphQL, if you're into that)
Containers aren't RESTful, the app they host can be. You don't have any REST endpoints yet, looks like, though.
Anyways, you've mapped the port
ports:
- "80:80"
And the app in running there
if __name__ == "__main__":
app.run(host='0.0.0.0', port=80)
Assuming you have opened your firewall to allow port 80 on DigitalOcean, that's all you need
Just fire up your http://digital-ocean.address
Is python not the right tool?
Use whatever you're comfortable with. For example, if you just want make HTTP requests, straight shell commands to do cURL commands and echo HTML responses would work.
Is Docker not the right tool?
It's not required. Even if you have keys, placing them within a container just adds a layer of redirection, not obfuscation
I'm investigating how to develop a decent web app with Python. Since I don't want some high-order structures to get in my way, my choice fell on the lightweight Flask framework. Time will tell if this was the right choice.
So, now I've set up an Apache server with mod_wsgi, and my test site is running fine. However, I'd like to speed up the development routine by making the site automatically reload upon any changes in py or template files I make. I see that any changes in site's .wsgi file causes reloading (even without WSGIScriptReloading On in the apache config file), but I still have to prod it manually (ie, insert extra linebreak, save). Is there some way how to cause reload when I edit some of the app's py files? Or, I am expected to use IDE that refreshes the .wsgi file for me?
Run the flask run CLI command with debug mode enabled, which will automatically enable the reloader. As of Flask 2.2, you can pass --app and --debug options on the command line.
$ flask --app main.py --debug run
--app can also be set to module:app or module:create_app instead of module.py. See the docs for a full explanation.
More options are available with:
$ flask run --help
Prior to Flask 2.2, you needed to set the FLASK_APP and FLASK_ENV=development environment variables.
$ export FLASK_APP=main.py
$ export FLASK_ENV=development
$ flask run
It is still possible to set FLASK_APP and FLASK_DEBUG=1 in Flask 2.2.
If you are talking about test/dev environments, then just use the debug option. It will auto-reload the flask app when a code change happens.
app.run(debug=True)
Or, from the shell:
$ export FLASK_DEBUG=1
$ flask run
http://flask.palletsprojects.com/quickstart/#debug-mode
In test/development environments
The werkzeug debugger already has an 'auto reload' function available that can be enabled by doing one of the following:
app.run(debug=True)
or
app.debug = True
You can also use a separate configuration file to manage all your setup if you need be. For example I use 'settings.py' with a 'DEBUG = True' option. Importing this file is easy too;
app.config.from_object('application.settings')
However this is not suitable for a production environment.
Production environment
Personally I chose Nginx + uWSGI over Apache + mod_wsgi for a few performance reasons but also the configuration options. The touch-reload option allows you to specify a file/folder that will cause the uWSGI application to reload your newly deployed flask app.
For example, your update script pulls your newest changes down and touches 'reload_me.txt' file. Your uWSGI ini script (which is kept up by Supervisord - obviously) has this line in it somewhere:
touch-reload = '/opt/virtual_environments/application/reload_me.txt'
I hope this helps!
If you're running using uwsgi look at the python auto reload option:
uwsgi --py-autoreload 1
Example uwsgi-dev-example.ini:
[uwsgi]
socket = 127.0.0.1:5000
master = true
virtualenv = /Users/xxxx/.virtualenvs/sites_env
chdir = /Users/xxx/site_root
module = site_module:register_debug_server()
callable = app
uid = myuser
chmod-socket = 660
log-date = true
workers = 1
py-autoreload = 1
site_root/__init__.py
def register_debug_server():
from werkzeug.debug import DebuggedApplication
app = Flask(__name__)
app.debug = True
app = DebuggedApplication(app, evalex=True)
return app
Then run:
uwsgi --ini uwsgi-dev-example.ini
Note: This example also enables the debugger.
I went this route to mimic production as close as possible with my nginx setup. Simply running the flask app with it's built in web server behind nginx it would result in a bad gateway error.
For Flask 1.0 until 2.2, the basic approach to hot re-loading is:
$ export FLASK_APP=my_application
$ export FLASK_ENV=development
$ flask run
you should use FLASK_ENV=development (not FLASK_DEBUG=1)
as a safety check, you can run flask run --debugger just to make sure it's turned on
the Flask CLI will now automatically read things like FLASK_APP and FLASK_ENV if you have an .env file in the project root and have python-dotenv installed
app.run(use_reloader=True)
we can use this, use_reloader so every time we reload the page our code changes will be updated.
I got a different idea:
First:
pip install python-dotenv
Install the python-dotenv module, which will read local preference for your project environment.
Second:
Add .flaskenv file in your project directory. Add following code:
FLASK_ENV=development
It's done!
With this config for your Flask project, when you run flask run and you will see this output in your terminal:
And when you edit your file, just save the change. You will see auto-reload is there for you:
With more explanation:
Of course you can manually hit export FLASK_ENV=development every time you need. But using different configuration file to handle the actual working environment seems like a better solution, so I strongly recommend this method I use.
Use this method:
app.run(debug=True)
It will auto-reload the flask app when a code change happens.
Sample code:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def index():
return "Hello World"
if __name__ == '__main__':
app.run(debug=True)
Well, if you want save time not reloading the webpage everytime when changes happen, then you can try the keyboard shortcut Ctrl + R to reload the page quickly.
From the terminal you can simply say
export FLASK_APP=app_name.py
export FLASK_ENV=development
flask run
or in your file
if __name__ == "__main__":
app.run(debug=True)
Enable the reloader in flask 2.2:
flask run --reload
Flask applications can optionally be executed in debug mode. In this mode, two very convenient modules of the development server called the reloader and the debugger are enabled by default.
When the reloader is enabled, Flask watches all the source code files of your project and automatically restarts the server when any of the files are modified.
By default, debug mode is disabled. To enable it, set a FLASK_DEBUG=1 environment variable before invoking flask run:
(venv) $ export FLASK_APP=hello.py for Windows use > set FLASK_APP=hello.py
(venv) $ export FLASK_DEBUG=1 for Windows use > set FLASK_DEBUG=1
(venv) $ flask run
* Serving Flask app "hello"
* Forcing debug mode on
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 273-181-528
Having a server running with the reloader enabled is extremely useful during development, because every time you modify and save a source file, the server automatically restarts and picks up the change.
To achieve this in PyCharm set 'Environment Variables' section to:
PYTHONUNBUFFERED=1;
FLASK_DEBUG=1
For Flask 'run / debug configurations'.
To help with fast automatic change in browser:
pip install livereload
from livereload import Server
if __name__ == '__main__':
server = Server(app.wsgi_app)
server.serve()
Next, Start your server again:
eg. your .py file is app.py
python app.py
I believe a better solution is to set the app configuration. For me, I built the tool and then pushed it to a development server where I had to set up a WSGI pipeline to manage the flask web app. I had some data being updated to a template and I wanted it to refresh every X minutes (WSGI deployment for the Flask site through APACHE2 on UBUNTU 18). In your app.py or whatever your main app is, add app.config.update dictionary below and mark TEMPLATES_AUTO_RELOAD=True, you will find that any templates that are automatically updated on the server will be reflected in the browser. There is some great documentation on the Flask site for configuration handling found here.
app = Flask(__name__)
app.config.update(
TEMPLATES_AUTO_RELOAD=True
)
Suppose i my web application running with following settings
LOG_DIR = "/var/log/main"
This variable defines where the log should go.
Now then i run my tests and i have test_settings like
from settings import *
LOG_DIR = "/var/log/test"
Now i want to know that does that mean while my test script is running then my main application logs will also go to test folder because i have chnaged the global variable.
For my integration testiing i need to change variables if am afraid that if that will affect the main application or not .Like my application depends upon
Shell ENV variables . I wanted to chnage that for my tests. but i am afraid that if that will chnage the main running application.
This is not for PROD but for other testing applications environment
Running a django process will not affect the settings of another process. Just make sure to explicitly pass the --settings flag to your manage.py script when you run your tests/dev server/ etc..
e.g.
python manage.py test --settings project/settings/test.py
python manage.py runserver --settings project/settings
I prefer having a set of shell scripts that get sourced when I run my test environment (you can configure that with a test runner, see the docs).
I'm investigating how to develop a decent web app with Python. Since I don't want some high-order structures to get in my way, my choice fell on the lightweight Flask framework. Time will tell if this was the right choice.
So, now I've set up an Apache server with mod_wsgi, and my test site is running fine. However, I'd like to speed up the development routine by making the site automatically reload upon any changes in py or template files I make. I see that any changes in site's .wsgi file causes reloading (even without WSGIScriptReloading On in the apache config file), but I still have to prod it manually (ie, insert extra linebreak, save). Is there some way how to cause reload when I edit some of the app's py files? Or, I am expected to use IDE that refreshes the .wsgi file for me?
Run the flask run CLI command with debug mode enabled, which will automatically enable the reloader. As of Flask 2.2, you can pass --app and --debug options on the command line.
$ flask --app main.py --debug run
--app can also be set to module:app or module:create_app instead of module.py. See the docs for a full explanation.
More options are available with:
$ flask run --help
Prior to Flask 2.2, you needed to set the FLASK_APP and FLASK_ENV=development environment variables.
$ export FLASK_APP=main.py
$ export FLASK_ENV=development
$ flask run
It is still possible to set FLASK_APP and FLASK_DEBUG=1 in Flask 2.2.
If you are talking about test/dev environments, then just use the debug option. It will auto-reload the flask app when a code change happens.
app.run(debug=True)
Or, from the shell:
$ export FLASK_DEBUG=1
$ flask run
http://flask.palletsprojects.com/quickstart/#debug-mode
In test/development environments
The werkzeug debugger already has an 'auto reload' function available that can be enabled by doing one of the following:
app.run(debug=True)
or
app.debug = True
You can also use a separate configuration file to manage all your setup if you need be. For example I use 'settings.py' with a 'DEBUG = True' option. Importing this file is easy too;
app.config.from_object('application.settings')
However this is not suitable for a production environment.
Production environment
Personally I chose Nginx + uWSGI over Apache + mod_wsgi for a few performance reasons but also the configuration options. The touch-reload option allows you to specify a file/folder that will cause the uWSGI application to reload your newly deployed flask app.
For example, your update script pulls your newest changes down and touches 'reload_me.txt' file. Your uWSGI ini script (which is kept up by Supervisord - obviously) has this line in it somewhere:
touch-reload = '/opt/virtual_environments/application/reload_me.txt'
I hope this helps!
If you're running using uwsgi look at the python auto reload option:
uwsgi --py-autoreload 1
Example uwsgi-dev-example.ini:
[uwsgi]
socket = 127.0.0.1:5000
master = true
virtualenv = /Users/xxxx/.virtualenvs/sites_env
chdir = /Users/xxx/site_root
module = site_module:register_debug_server()
callable = app
uid = myuser
chmod-socket = 660
log-date = true
workers = 1
py-autoreload = 1
site_root/__init__.py
def register_debug_server():
from werkzeug.debug import DebuggedApplication
app = Flask(__name__)
app.debug = True
app = DebuggedApplication(app, evalex=True)
return app
Then run:
uwsgi --ini uwsgi-dev-example.ini
Note: This example also enables the debugger.
I went this route to mimic production as close as possible with my nginx setup. Simply running the flask app with it's built in web server behind nginx it would result in a bad gateway error.
For Flask 1.0 until 2.2, the basic approach to hot re-loading is:
$ export FLASK_APP=my_application
$ export FLASK_ENV=development
$ flask run
you should use FLASK_ENV=development (not FLASK_DEBUG=1)
as a safety check, you can run flask run --debugger just to make sure it's turned on
the Flask CLI will now automatically read things like FLASK_APP and FLASK_ENV if you have an .env file in the project root and have python-dotenv installed
app.run(use_reloader=True)
we can use this, use_reloader so every time we reload the page our code changes will be updated.
I got a different idea:
First:
pip install python-dotenv
Install the python-dotenv module, which will read local preference for your project environment.
Second:
Add .flaskenv file in your project directory. Add following code:
FLASK_ENV=development
It's done!
With this config for your Flask project, when you run flask run and you will see this output in your terminal:
And when you edit your file, just save the change. You will see auto-reload is there for you:
With more explanation:
Of course you can manually hit export FLASK_ENV=development every time you need. But using different configuration file to handle the actual working environment seems like a better solution, so I strongly recommend this method I use.
Use this method:
app.run(debug=True)
It will auto-reload the flask app when a code change happens.
Sample code:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def index():
return "Hello World"
if __name__ == '__main__':
app.run(debug=True)
Well, if you want save time not reloading the webpage everytime when changes happen, then you can try the keyboard shortcut Ctrl + R to reload the page quickly.
From the terminal you can simply say
export FLASK_APP=app_name.py
export FLASK_ENV=development
flask run
or in your file
if __name__ == "__main__":
app.run(debug=True)
Enable the reloader in flask 2.2:
flask run --reload
Flask applications can optionally be executed in debug mode. In this mode, two very convenient modules of the development server called the reloader and the debugger are enabled by default.
When the reloader is enabled, Flask watches all the source code files of your project and automatically restarts the server when any of the files are modified.
By default, debug mode is disabled. To enable it, set a FLASK_DEBUG=1 environment variable before invoking flask run:
(venv) $ export FLASK_APP=hello.py for Windows use > set FLASK_APP=hello.py
(venv) $ export FLASK_DEBUG=1 for Windows use > set FLASK_DEBUG=1
(venv) $ flask run
* Serving Flask app "hello"
* Forcing debug mode on
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 273-181-528
Having a server running with the reloader enabled is extremely useful during development, because every time you modify and save a source file, the server automatically restarts and picks up the change.
To achieve this in PyCharm set 'Environment Variables' section to:
PYTHONUNBUFFERED=1;
FLASK_DEBUG=1
For Flask 'run / debug configurations'.
To help with fast automatic change in browser:
pip install livereload
from livereload import Server
if __name__ == '__main__':
server = Server(app.wsgi_app)
server.serve()
Next, Start your server again:
eg. your .py file is app.py
python app.py
I believe a better solution is to set the app configuration. For me, I built the tool and then pushed it to a development server where I had to set up a WSGI pipeline to manage the flask web app. I had some data being updated to a template and I wanted it to refresh every X minutes (WSGI deployment for the Flask site through APACHE2 on UBUNTU 18). In your app.py or whatever your main app is, add app.config.update dictionary below and mark TEMPLATES_AUTO_RELOAD=True, you will find that any templates that are automatically updated on the server will be reflected in the browser. There is some great documentation on the Flask site for configuration handling found here.
app = Flask(__name__)
app.config.update(
TEMPLATES_AUTO_RELOAD=True
)