I am working on a flask backend and one of its features require regular calls to an external API and store some results in the database. I used APScheduler to do so.
Not having full access to the upstream server (docker container within google cloud platforms), we managed to serve the backend using apache's mod_wsgi.
Running the backend from my debugger on my PC, the scheduled tasks seem to work as my database is populated.
But the server does not seem to run those tasks at all at when I query the database which should be populated, the tables are empty.
My usage of APScheduler is as follows in the __init__.py :
from apscheduler.schedulers.background import BackgroundScheduler
# Some unrelated code here
scheduler = BackgroundScheduler()
import module as service
scheduler.add_job(service.job1, 'interval', minutes=10)
scheduler.add_job(service.job2, 'interval', minutes=5)
scheduler.add_job(service.job3, 'interval', minutes=1)
scheduler.start()
I'm asking if there are additional steps that I need to do in order for those tasks to be run on the upstream server.
Related
I'm deploying a sample fast api app to the cloud with google standard app engine model. The app is served with gunicorn this way:
gunicorn main:app --workers 4 --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:80
This command spawns 4 worker proccesses of my app.
I've read that in fast api you can either create sync or async endpoints. If an endpoint is async all requests run on a single thread with the event loop. If the endpoint is sync, it runs the function on another thread to prevent it from blocking the server.
I have sync blocking endpoints, so fastapi should run them on threads, but also i have gunicorn spawning worker proccesess.
Given that python only executes one thread at a time, but also the standard app engine is also limited CPU wise on multiple proccessing, i'm confused on the best configuration for a fastapi application on the cloud.
Should i let gunicorn or fastapi handle the concurrency?
The number of workers you specify should match the instance class of your App Engine app; and since you're using 4 workers in your app, it has an equivalence of 4 instance classes. Here's an example that shows an App Engine deployment that uses 4 gunicorn workers for serving apps: entrypoint: gunicorn -b :8080 -w 4 main:app. The examples I've provided was stated in the entrypoint best practices.
Just a note, the gunicorn uses sync workers by default so that worker class is compatible with all web applications, but each worker can only handle one request at a time.
Lastly if using Google App Engine Flex, kindly check the recommended gunicorn configurations for further guide in your app.
I'm trying to use django in combination with celery.
Therefore I came across autodiscover_tasks() and I'm not fully sure on how to use them. The celery workers get tasks added by other applications (in this case a node backend).
So far I used this to start the worker:
celery worker -Q extraction --hostname=extraction_worker
which works fine.
Now I'm not sure what the general idea of the django-celery integration is. Should workers still be started from external (e.g. with the command above), or should they be managed and started from the django application?
My celery.py looks like:
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'main.settings')
app = Celery('app')
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
then I have 2 apps containing a tasks.py file with:
#shared_task
def extraction(total):
return 'Task executed'
how can I now register django to register the worker for those tasks?
You just start worker process as documented, you don't need to register anything else
In a production environment you’ll want to run the worker in the
background as a daemon - see Daemonization - but for testing and
development it is useful to be able to start a worker instance by
using the celery worker manage command, much as you’d use Django’s
manage.py runserver:
celery -A proj worker -l info
For a complete listing of the command-line options available, use the
help command:
celery help
celery worker collects/registers task when it runs and also consumes tasks which it found out
Using the great answer to "How to configure celery-redis in django project on microsoft azure?", I can configure Celery to use Azure Redis Cache using the non-ssl port, 6379, using the following Python code:
from celery import Celery
# This one works
url = 'redis://:<access key>#<my server>.redis.cache.windows.net:6379/0'
# I want to use a url that specifies ssl like one of the following:
# url = 'redis://:<my key>=#<my server>.redis.cache.windows.net:6380/0'
# url = 'redis://:<my key>#<my server>.redis.cache.windows.net:6380/0?ssl=True'
app = Celery('tasks', broker=url)
#app.task
def add(x, y):
return x + y
However, I would like to have celery use ssl and communicate on port 3380 using ssl to the Azure Redis Cache. If I change the port to 6380, I get an "Error while reading from socket" after a few minutes of waiting after running the following command:
celery -A tasks worker --loglevel=INFO -Q "celery" -Ofair
Does anyone know how to configure this, on the Celery or Azure side, so that I can have celery communicate on the default 3380 port on Azure Redis Cache using ssl?
I am using the latest version of Celery (4.0.2)
Note that code like the following works with no problem when connecting directly from a Linux client (on Azure) using port 3380 and ssl using Python's redis library:
import redis
redis.StrictRedis(host='<my host>.redis.cache.windows.net', port=6380, db=0, password='<my key>', ssl=True)
It's already possible using rediss:// instead redis://.
url = 'rediss://:<access key>#<my server>.redis.cache.windows.net:6380/0'
For the broker, you should be able to set the broker_use_ssl configuration option.
For the backend, the option redis_backend_use_ssl was made available in the 4.1.0 release.
The ability to enable SSL via the URL isn't available yet: https://github.com/celery/celery/issues/2833
Also, note that official support for Windows was dropped in 4.0. However, you might be able to get it working by following the instructions at https://github.com/celery/celery/issues/4082
I have a flask application using an uwsgi instance. This application runs some threads in background when a cron command starts. Is there a method for updating my template files without restarting the uwsgi service ?
Currently I'm waiting for my threads to stop and then reloading the uwsgi service.
Enabling TEMPLATES_AUTO_RELOAD works nicely:
app = Flask(__name__)
app.config['TEMPLATES_AUTO_RELOAD'] = True
Whether to check for modifications of the template source and reload
it automatically. By default the value is None which means that Flask
checks original file only in debug mode.
Source: http://flask.pocoo.org/docs/0.12/config/
I'm building a Flask web app and would like to run some background services when the app starts up.
For example, in the web app you have the ability to add service accounts, routers ip's, server ip's and such. I'd like to have some background services running network scans, wmi calls and some other tasks to constantly update the database with relevant information.
In Rails I've used initializers in config/initializers to start some daemons:
# Start all the daemons to update the DB
system('script/daemon restart vs_updater.rb')
system('script/daemon restart c_fw_updater.rb')
system('script/daemon restart sans_updater.rb')
system('script/daemon restart sf_updater.rb')
Is there such an equivalent for Flask? Or should I just build separate scripts and run them in a different manner?
you can add commands to the __init__.py file in the root directory and use subprocess to run the script
from subprocess import call
call("script/daemon restart vs_updater.py")
fab
http://www.fabfile.org/
subprocess
https://docs.python.org/2/library/subprocess.html