We Keep Coding

Using PyTorch with Celery

I'm trying to run a PyTorch model in a Django app. As it is not recommended to execute the models (or any long-running task) in the views, I decided to run it in a Celery task. My model is quite big and it takes about 12 seconds to load and about 3 seconds to infer. That's why I decided that I couldn't afford to load it at every request. So I tried to load it at settings and save it there for the app to use it. So my final scheme is:
When the Django app starts, in the settings the PyTorch model is loaded and it's accessible from the app.
When views.py receives a request, it delays a celery task
The celery task uses the settings.model to infer the result
The problem here is that the celery task throws the following error when trying to use the model
[2020-08-29 09:03:04,015: ERROR/ForkPoolWorker-1] Task app.tasks.task[458934d4-ea03-4bc9-8dcd-77e4c3a9caec] raised unexpected: RuntimeError("Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method")
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
return self.run(*args, **kwargs)
/*...*/
File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/torch/cuda/__init__.py", line 191, in _lazy_init
"Cannot re-initialize CUDA in forked subprocess. " + msg)
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Here's the code in my settings.py loading the model:
if sys.argv and sys.argv[0].endswith('celery') and 'worker' in sys.argv: #In order to load only for the celery worker
import torch
torch.cuda.init()
torch.backends.cudnn.benchmark = True
load_model_file()
And the task code
#task
def getResult(name):
print("Executing on GPU:", torch.cuda.is_available())
if os.path.isfile(name):
try:
outpath = model_inference(name)
os.remove(name)
return outpath
except OSError as e:
print("Error", name, "doesn't exist")
return ""
The print in the task shows "Executing on GPU: true"
I've tried setting torch.multiprocessing.set_start_method('spawn') in the settings.py before and after the torch.cuda.init() but it gives the same error.

Setting this method works as long as you're also using Process from the same library.
from torch.multiprocessing import Pool, Process
Celery uses "regular" multiprocessing library, thus this error.
If I were you I'd try either:
run it single threaded to see if that helps
run it with eventlet to see if that helps
read this

A quick fix is to make things single-threaded. To do that set the worker pool type of celery to solo while starting the celery worker
celery -A your_proj worker -P solo -l info

This is due to the fact that the Celery worker itself is using forking. This appears to be a currently known issue with Celery >=4.0
You used to be able to configure celery to spawn, rather than fork, but that feature (CELERYD_FORCE_EXECV) was removed in 4.0.
There is no inbuilt options to get around this. Some custom monkeypatching to do this is probably be possible, but YMMV
Some potentially viable options might be:
Use celery <4.0 with CELERYD_FORCE_EXECV enabled.
Launch celery workers on Windows (where forking is not possible anyhow)

ValueError: Unknown type <class 'redis.client.StrictPipeline'>

I develop locally on win10, which is a problem for the usage of the RQ task queue, which only works on linux systems because it requires the ability to fork processes. I'm trying to extend the flask-base project https://github.com/hack4impact/flask-base/tree/master/app which can use RQ. I came across https://github.com/michaelbrooks/rq-win . I love the idea of this repo (If I can get it working it will really simplify my life, since I develop on win 10 -64):
After installing this library
I can queue a job in my views by running something like:
#login_required
#main.route('/selected')
def selected():
messages = 'abcde'
j = get_queue().enqueue(render_png, messages, result_ttl=5000)
return j.get_id()
This returns a job_code correctly.
I changed the code in manage.py to:
from rq_win import WindowsWorker
#manager.command
def run_worker():
"""Initializes a slim rq task queue."""
listen = ['default']
REDIS_URL = 'redis://localhost:6379'
conn = Redis.from_url(REDIS_URL)
with Connection(conn):
# worker = Worker(map(Queue, listen))
worker = WindowsWorker(map(Queue, listen))
worker.work()
When I try to run it with:
$ python -u manage.py run_worker
09:40:44
09:40:44 *** Listening on ?[32mdefault?[39;49;00m...
09:40:58 ?[32mdefault?[39;49;00m: ?[34mapp.main.views.render_png('{"abcde"}')?[39;49;00m (8c1b6186-39a5-4daf-9c45-f60e4241cd1f)
...\lib\site-packages\rq\job.py:161: DeprecationWarning: job.status is deprecated. Use job.set_status() instead
DeprecationWarning
09:40:58 ?[31mValueError: Unknown type <class 'redis.client.StrictPipeline'>?[39;49;00m
Traceback (most recent call last):
File "...\lib\site-packages\rq_win\worker.py", line 87, in perform_job
queue.enqueue_dependents(job, pipeline=pipeline)
File "...\lib\site-packages\rq\queue.py", line 322, in enqueue_dependents
for job_id in pipe.smembers(dependents_key)]
File "...\lib\site-packages\rq\queue.py", line 322, in <listcomp>
for job_id in pipe.smembers(dependents_key)]
File "...\lib\site-packages\rq\compat\__init__.py", line 62, in as_text
raise ValueError('Unknown type %r' % type(v))
ValueError: Unknown type <class 'redis.client.StrictPipeline'>
So in summary, I think the jobs are being queued correctly within redis. However when the worker process tries to grab a job off of the queue to process, This error occurs. How can I fix this?

So after some digging, it looks like the root of the error is here, where job_id being sent to the as_text function is, somehow, a StrictPipeline object. However, I have been unable to replicate the error locally; can you post more of your code? Also, I would try re-installing the redis, rq, and rq-win modules, and possibly try importing rq.compat

While using jenkins API, getting a failure on reconfig_job

I am using jenkins rest API to recurse through jobs and then reconfigure this one. All methods work except one. He's is my code :
def get_server_instance():
jenkins_url = 'xxxx'
#server = Jenkins(jenkins_url, username = '', password = '')
# Connect to instance - username and password are optional
server = jenkins.Jenkins(jenkins_url, username = '', password = '')
return server
def get_job_details():
# Refer Example #1 for definition of function 'get_server_instance'
server = get_server_instance()
for job in server.get_jobs_list():
if job == "GithubMigration":
configuration = server.get_job(job).get_config().encode('utf-8')
#server.reconfig_job(job, configuration)
if server.has_job("GithubMigration"):
server.reconfig_job('GithubMigration', config_xml)
It gets my configuration.xml, find the job as well but fails on server.reconfig_job('GithubMigration', config_xml) with the error , AttributeError: 'Jenkins' object has no attribute 'reconfig_job'
when obviously this functions exists in the jenkins rest API and yes I'm importing jenkins, from jenkinsapi.jenkins import Jenkins .
Edit 1 - I uninstalled Jenkinsapi and have only python-jenkins module and now it fails even before saying
AttributeError: 'module' object has no attribute 'Jenkins' for line : AttributeError: 'module' object has no attribute 'Jenkins'
Any ideas?
Edit 2 :
I tries solely python-jenkins API and tried their own example as you see here http://python-jenkins.readthedocs.org/en/latest/example.html
import jenkins
j = jenkins.Jenkins('http://your_url_here', 'username', 'password')
j.get_jobs()
j.create_job('empty', jenkins.EMPTY_CONFIG_XML)
j.disable_job('empty')
j.copy_job('empty', 'empty_copy')
j.enable_job('empty_copy')
j.reconfig_job('empty_copy', jenkins.RECONFIG_XML)
Even this fails at jenkins.Jenkins with attribute error at Jenkins - No module.
I am pretty sure the API is broken.

Your script is probably importing wrong module. You can check it as follows:
import jenkins
print jenkins.__file__
If printed path is other than installation path of jenkins module (eg. C:\Python27_32\lib\site-packages\jenkins\__init__.pyc), then you should check pythonpath:
import sys
print sys.path
Common problem is existence of python script with same name as imported module in current directory, which is at the first place in search path ''.
For more info on import order see module search path

Following #Chemik answer, I realized that the script I wrote was named jenkins.py and it was conflicting with python-jenkins import.
The library isn't broken. Check your script name.

had to add another solution, while running the same command
server = jenkins.Jenkins(jenkins_url, username = '', password = '')
i got the error:
'jenkins' has no attribute 'Jenkins'
my mistake was when installing the package, i installed package "jenkins" and the package i was needed is "python-jenkins".
docs can be found:
python-jenkins docs
so what i had to do is just
pip install python-jenkins

Refactored models, now celery won't start

I have a django app called "uar" and I refactored the uar/models.py so that there are base classes in uar_common/models.py - for instance instead of
class ReviewPerson(models.Model):
I have
class ReviewPerson(uar_common.models.CommonPerson):
I added uar_common to the INSTALLED_APPS in the settings.py, and my app works fine. But my celery tasks, which worked fine in the past, don't work any more. I start it with python manage.py celery worker --loglevel=debug as always, but now it dies immediately with a stack trace that ends with:
File "/home/ptomblin/src/cart/uar/models.py", line 8, in <module>
class ReviewPerson(uar_common.models.CommonPerson):
AttributeError: 'module' object has no attribute 'models'
If I just run python manage.py shell, I can do
from uar.models import ReviewPerson
rp = ReviewPerson.objects.get(review=2, last_name='Tomblin', first_name='Paul')
and it has no problem importing the model and find the correct record.
So why can't djcelery find the new base class?

I added the following to CELERY_IMPORTS
'uar_common.models',
'uar_history.models',
And now it's working.

Easy application logging/debugging with nginx, uwsgi, flask?

I'm not looking to turn on the dangerous debugging console, but my application is getting a 500 error and doesn't seem to be writing any output for me to investigate more deeply.
I saw this exchange on the mailing list, which led me to this page on logging errors.
However, I still find this very confusing and have a couple of questions:
(1) In which file should the stuff below go?
ADMINS = ['yourname#example.com']
if not app.debug:
import logging
from logging.handlers import SMTPHandler
mail_handler = SMTPHandler('127.0.0.1',
'server-error#example.com',
ADMINS, 'YourApplication Failed')
mail_handler.setLevel(logging.ERROR)
app.logger.addHandler(mail_handler)
...assuming the "getting bigger" file pattern for larger applications? __init__.py? config.py? run.py?
(2) I am overwhelmed by options there, and can't tell which I should use. Which loggers should I turn on, with what settings, to replicate the local python server debug I get to stdout when I run run.py? I find that default, local output stream very useful, more so than the interactive debugger in the page. Does anyone have a pattern they could share on setting up something replicating this with an nginx deployment, outputting to a log?
(3) Is there anything I need to change, not at the flask level, but in nginx, say in my /etc/nginx/sites-available/appname file, to enable logging?
UPDATE
Specifically, I'm looking for information like I get when python runs locally as to why, say, a package isn't working, or where my syntax error might be, or what variable doesn't exist:
$ python run.py
Traceback (most recent call last):
File "run.py", line 1, in <module>
from myappname import app
File "/home/me/myappname/myappname/__init__.py", line 27, in <module>
file_handler.setLevel(logging.debug)
File "/usr/lib/python2.7/logging/__init__.py", line 710, in setLevel
self.level = _checkLevel(level)
File "/usr/lib/python2.7/logging/__init__.py", line 190, in _checkLevel
raise TypeError("Level not an integer or a valid string: %r" % level)
When I run flask on a server, I never see this. I just get a uWSGI error in the browser, and have no idea which code was problematic. I would just like something like the above to be written to a file.
I notice also that setting the following logging didn't really write much to file, even when I turn the log way up to the DEBUG level:
from logging import FileHandler
file_handler = FileHandler('mylog.log')
file_handler.setLevel(logging.DEBUG)
app.logger.addHandler(file_handler)
mylog.log is blank, even when my application errors out.
I'll also add that I've tried to set debug = True in the following ways, in __init__.py:
app = Flask(__name__)
app.debug = True
app.config['DEBUG'] = True
from werkzeug.debug import DebuggedApplication
app.wsgi_app = DebuggedApplication(app.wsgi_app, True)
app.config.from_object('config')
app.config.update(DEBUG=True)
app.config['DEBUG'] = True
if __name__ == '__main__':
app.run(debug=True)
While in my config.py file, I have...
debug = True
Debug = True
DEBUG = True
Yet, no debugging happens, and without logging or debugging, this is rather hard to track down. Errors simply terminate the application with the un-useful browser message:
uWSGI Error
Python application not found

Set config['PROPAGATE_EXCEPTIONS'] to True when running app in production and you want tracebacks to be logged into log files. (I haven't tried with SMTP handler, though..)

The part where you create handlers, add to loggers etc. should be in the if __name__ == '__main__' clause, i.e. your main entry point. I assume that would be run.py.
I'm not sure I can answer this - it depends on what you want. I'd advise looking at the logging tutorial to see the various options available.
I don't believe you need to change anything at the nginx level.
Update: You might want to have an exception clause that covers uncaught exceptions, e.g.
if __name__ == '__main__':
try:
app.run(debug=True)
except Exception:
app.logger.exception('Failed')
which should write the traceback of any exception which occurred in app.run() to the log.

I know that this is a VERY old post, but I ran into the issue now, and it took me a bit to find the solution. Flask sends errors to the server. I was running Gunicorn with an upstart script on Ubuntu 14.04 LTS, and the place where I found the error logs was as follows:
/var/log/upstart/myapp.log
http://docs.gunicorn.org/en/stable/deploy.html#upstart
Just in case some other poor soul ends up in this situation.