We Keep Coding

Celery - ValueError: Port could not be cast to integer value as

I'm trying to run simple Celery example: celery-example-local-filesystem.
Here is a task module:
#tasks.py
from celery import Celery
app = Celery('tasks', broker='pyamqp://guest#localhost//')
app.config_from_object('celeryconfig')
#app.task
def add(x, y):
return x + y
Here is a config:
#celeryconfig.py
"""Celery configuration using local filesystem only."""
from pathlib import Path
# paths for file backend, create folders
_root = Path(__file__).parent.resolve().joinpath('data')
#_root = Path('c:\\temp').parent.resolve().joinpath('data')
_backend_folder = _root.joinpath('results')
_backend_folder.mkdir(exist_ok=True, parents=True)
_folders = {
'data_folder_in': _root.joinpath('in'),
'data_folder_out': _root.joinpath('in'), # has to be the same as 'data_folder_in'
'processed_folder': _root.joinpath('processed')
}
for fn in _folders.values():
fn.mkdir(exist_ok=True)
# celery config
result_backend = 'file://{}'.format(str(_backend_folder))
broker_url = 'filesystem://'
broker_transport_options = {k: str(f) for k, f in _folders.items()}
task_serializer = 'json'
persist_results = True
result_serializer = 'json'
accept_content = ['json']
imports = ('tasks',)
and here is a main module:
#main.py
from celery import Celery, signature
app = Celery('tasks')
app.config_from_object('celeryconfig')
add = signature('tasks.add')
print('1 + 1 = {}'.format(add.delay(1, 1).get(timeout=3.)))
And here, when i try to run celery on windows, get an error:
$ celery -A tasks worker --loglevel=INFO
[2021-04-03 18:16:35,578: CRITICAL/MainProcess] Unrecoverable error: ValueError("Port could not be cast to integer value as '\\\\Users\\\\marci\\\\code\\\\django\\\\cellery_test\\\\data\\\\results'")
Traceback (most recent call last):
File "c:\python39\lib\site-packages\celery\worker\worker.py", line 203, in start
self.blueprint.start(self)
File "c:\python39\lib\site-packages\celery\bootsteps.py", line 112, in start
self.on_start()
File "c:\python39\lib\site-packages\celery\apps\worker.py", line 136, in on_start
self.emit_banner()
File "c:\python39\lib\site-packages\celery\apps\worker.py", line 170, in emit_banner
' \n', self.startup_info(artlines=not use_image))),
File "c:\python39\lib\site-packages\celery\apps\worker.py", line 232, in startup_info
results=self.app.backend.as_uri(),
File "c:\python39\lib\site-packages\celery\backends\base.py", line 143, in as_uri
url = maybe_sanitize_url(self.url or '')
File "c:\python39\lib\site-packages\kombu\utils\url.py", line 118, in maybe_sanitize_url
return sanitize_url(url, mask)
File "c:\python39\lib\site-packages\kombu\utils\url.py", line 111, in sanitize_url
return as_url(*_parse_url(url), sanitize=True, mask=mask)
File "c:\python39\lib\site-packages\kombu\utils\url.py", line 76, in url_to_parts
parts.port,
File "c:\python39\lib\urllib\parse.py", line 175, in port
raise ValueError(message) from None
ValueError: Port could not be cast to integer value as '\\Users\\marci\\code\\django\\cellery_test\\data\\results'
It looks like some issue with path decoding. Does anyone faced with this issue? I will be grateful for your help!

The problem for me was the usage of special characters(/, ?, #, # and :) in the url I passed to urllib.parse.
Once I removed them from the path, it worked beautifully.

In my case, I created a .env file in my project folder root with the follow variables:
REDIS_URL=redis://127.0.0.1:6379/
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_DB=0
Then, make sure do you have a redis.py file at your project, the same folder where are the wsgi.py file with this:
import redis as r
from .settings import REDIS_HOST, REDIS_PORT, REDIS_DB
redis = r.Redis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB)
On the celery.py file, at the same folder where are the redis.py and wsgi.py files, put a similar code how this:
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'your_project.settings')
app = Celery('your_project')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
In the same folder yet, make sure do you have create a __init__.py file with:
from __future__ import absolute_import
from .celery import app as celery_app
And in the settings.py, I add the code bellow:
import os
from dotenv import load_dotenv
...
load_dotenv()
...
REDIS_URL = os.getenv('REDIS_URL')
REDIS_HOST = os.getenv('REDIS_HOST')
REDIS_PORT = os.getenv('REDIS_PORT')
REDIS_DB = os.getenv('REDIS_DB')
BROKER_URL = f'redis://{REDIS_HOST}:{REDIS_PORT}'
CELERY_RESULT_BACKEND = BROKER_URL
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
After this, do you can deploy your project with this tutorial for example: https://realpython.com/asynchronous-tasks-with-django-and-celery/ or this other tutorial: https://www.botreetechnologies.com/blog/implementing-celery-using-django-for-background-task-processing/
I hope help you.

I would first check whether the result backend URL is correct (check the https://en.wikipedia.org/wiki/File_URI_scheme for more information). I had no idea filesystem:// is a possible broker alternative. Even if it is, it is most likely highly experimental, so I recommend not using it until it reaches maturity (which I sincerely doubt will ever happen, as I really do not see the point - Celery is supposed to be a DISTRIBUTED system, so filesystem as broker makes no sense to me). Please use brokers listed here: https://docs.celeryproject.org/en/stable/getting-started/brokers/index.html or do not use Celery at all if you have hard requirement to use something that is not on that list.

Flask Error with wsgi_handler

I am trying to use WSGI on Windows Server to run a simple flask app. I keep running into the following error:
Error occurred while reading WSGI handler: Traceback (most recent call
last): File "c:\inetpub\wwwroot\test_site\wfastcgi.py", line 711, in
main env, handler = read_wsgi_handler(response.physical_path) File
"c:\inetpub\wwwroot\test_site\wfastcgi.py", line 568, in
read_wsgi_handler return env, get_wsgi_handler(handler_name) File
"c:\inetpub\wwwroot\test_site\wfastcgi.py", line 551, in
get_wsgi_handler raise ValueError('"%s" could not be imported' %
handler_name) ValueError: "app.app" could not be imported StdOut:
StdErr
For my site I configured a handler to call the FastCGIModule from Microsoft Web Platform installer
My app file looks as such:
from flask import Flask, request, jsonify
from analyzers import analyzer
import write_log
app = Flask(__name__)
#app.route("/")
def test():
return "Test load"
#app.route('/analyze', methods=['POST'])
def parse():
text = request.json['text']
name = request.json['name']
model = request.json['model']
try:
convert_flag = request.json['convert_flag']
except KeyError:
convert_flag = False
results= analyzer(text, name, model, convert_dose=convert_flag)
write_log.write_log(text, name, model, results)
return jsonify(results)
if __name__ == "__main__":
app.run()
If I comment out the custom import of my analyzer script and my write_log script along with the POST method things will run, so I know I must be messing something up there.
Does anybody have any suggestions?
Thanks in advance.
Paul

I had the same issue and the problem was with a third-party library. What's causing your problem is certainly something different, but here's something I did to identify my issue and may help you as well:
Open wfastcgi.py
Locate the method get_wsgi_handler (probably on line 519)
There's a try/except inside a while module_name statement
Add raise to the end of the except block and save the file, like this:
except ImportError:
...
raise
Access your website URL again and check your logs, they now should be more detailed about what caused the ImportError and will point you in the right direction to fix the issue

PySpark+Flask+CherryPy - AttributeError: 'module' object has no attribute 'tree'

I'm trying to test how to integrate Flask with Spark model according to this tutorial https://www.codementor.io/spark/tutorial/building-a-web-service-with-apache-spark-flask-example-app-part2#/ . Here CherryPy is used for wsgi. Trouble is that when we are launching app via spark-submit, it shows such stack trace:
Traceback (most recent call last):
File "/home/roman/dev/python/flask-spark/cherrypy.py", line 43, in <module>
run_server(app)
File "/home/roman/dev/python/flask-spark/cherrypy.py", line 21, in run_server
cherrypy.tree.graft(app_logged, '/')
AttributeError: 'module' object has no attribute 'tree'
I have no idea where the trouble is. I think that it because of new/old version or something like that, but I'm not sure. I have used also python 3 instead of python 2, but it didn't help. Here is wsgi config:
import time, sys, cherrypy, os
from paste.translogger import TransLogger
from webapp import create_app
from pyspark import SparkContext, SparkConf
def init_spark_context():
# load spark context
conf = SparkConf().setAppName("movie_recommendation-server")
# IMPORTANT: pass aditional Python modules to each worker
sc = SparkContext(conf=conf, pyFiles=['test.py', 'webapp.py'])
return sc
def run_server(app):
# Enable WSGI access logging via Paste
app_logged = TransLogger(app)
# Mount the WSGI callable object (app) on the root directory
cherrypy.tree.graft(app_logged, '/')
# Set the configuration of the web server
cherrypy.config.update({
'engine.autoreload.on': True,
'log.screen': True,
'server.socket_port': 5432,
'server.socket_host': '0.0.0.0'
})
# Start the CherryPy WSGI web server
cherrypy.engine.start()
cherrypy.engine.block()
if __name__ == "__main__":
# Init spark context and load libraries
sc = init_spark_context()
dataset_path = os.path.join('datasets', 'ml-latest-small')
app = create_app(sc, dataset_path)
# start web server
run_server(app)

Traceback you've provided clearly shows that your app is trying to use a local module called cherrypy (/home/roman/dev/python/flask-spark/cherrypy.py) not the actual cherrypy library (which should be something like /path/to/your/python/lib/python-version/siteX.Y/cherrypy).
To solve this problem you can simply rename the local module to avoid conflicts.

KeyError with CherryPy WSGIServer serving static files

I'm trying to use CherryPy's WSGI server to serve static files, like in Using Flask with CherryPy to serve static files. Option 2 of the accepted answer there looks exactly like what I'd like to do, but I'm getting a KeyError when I try to use the static directory handler.
What I've tried:
>>>> import cherrypy
>>>> from cherrypy import wsgiserver
>>>> import os
>>>> static_handler = cherrypy.tools.staticdir.handler(section='/', dir=os.path.abspath('server_files')
>>>> d = wsgiserver.WSGIPathInfoDispatcher({'/': static_handler})
>>>> server = wsgiserver.CherryPyWSGIServer(('localhost', 12345), d)
>>>> server.start()
Then, when I try to access the server I'm getting a 500 response and the following error in the console:
KeyError('tools',)
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1353, in communicate
req.respond()
File "/Library/Python/2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 868, in respond
self.server.gateway(self).respond()
File "/Library/Python/2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 2267, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "/Library/Python/2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 2477, in __call__
return app(environ, start_response)
File "/Library/Python/2.7/site-packages/cherrypy/_cptools.py", line 175, in handle_func
handled = self.callable(*args, **self._merged_args(kwargs))
File "/Library/Python/2.7/site-packages/cherrypy/_cptools.py", line 102, in _merged_args
tm = cherrypy.serving.request.toolmaps[self.namespace]
KeyError: 'tools'
This is displayed twice for each time I try to hit anything that the server should be able to display. When I hooked up a Flask app to the server the Flask app worked as expected, but the static file serving still gave the same error.
What do I need to do to get the staticdir.handler to work?

I've tried various ways of getting this to work and up until today was also hitting the KeyError you have been seeing (among other issues).
I finally managed to get CherryPy to serve static alongside a Django app by adapting the code from this gist (included below).
import os
import cherrypy
from cherrypy import wsgiserver
from my_wsgi_app import wsgi
PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), 'public'))
class Root(object):
pass
def make_static_config(static_dir_name):
"""
All custom static configurations are set here, since most are common, it
makes sense to generate them just once.
"""
static_path = os.path.join('/', static_dir_name)
path = os.path.join(PATH, static_dir_name)
configuration = {static_path: {
'tools.staticdir.on': True,
'tools.staticdir.dir': path}
}
print configuration
return cherrypy.tree.mount(Root(), '/', config=configuration)
# Assuming your app has media on diferent paths, like 'c', 'i' and 'j'
application = wsgiserver.WSGIPathInfoDispatcher({
'/': wsgi.application,
'/c': make_static_config('c'),
'/j': make_static_config('j'),
'/i': make_static_config('i')})
server = wsgiserver.CherryPyWSGIServer(('0.0.0.0', 8070), application,
server_name='www.cherrypy.example')
try:
server.start()
except KeyboardInterrupt:
print "Terminating server..."
server.stop()
Hopefully wrapping a Flask app will be fairly similar.
The key for me was using the cherrypy.tree.mount on a dummy class, rather than trying to use the staticdir.handler directly.
For the curious - I used the code in the gist to customise a version of django-cherrypy's runcpserver management command, although in hindsight it would probably have been easier to create a new command from scratch.
Good luck (and thanks to Alfredo Deza)!

Using django.test.client to test app http requests, getting errors

We're beginning to write unit tests for our API (created with the Django Rest Framework). We decided to start off simple and use the built in unittest and django.test.client classes. I've got the stub of a test written and it runs just fine:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import
import unittest
# from django.test.client import Client
class TestGrowth(unittest.TestCase):
def setUp(self):
# self.client = Client()
pass
def test_numbers_3_4(self):
self.assertEqual(4, 4, True)
def test_strings_a_3(self):
self.assertEqual('andy', 'andy', True)
def suite():
suite = unittest.TestSuite()
# Add test cases to suite
suite.addTests(unittest.makeSuite(TestGrowth))
return suite
if __name__ == '__main__':
# Run test suite
unittest.TextTestRunner(verbosity=2).run(suite())
However as soon as I uncomment the line reading from django.test.client import Client I get an error:
Traceback (most recent call last):
File "./AccountGrowth.py", line 8, in <module>
from django.test.client import Client
File "/home/vagrant/.virtualenvs/emmasocial/lib/python2.7/site-packages/django/test/__init__.py", line 5, in <module>
from django.test.client import Client, RequestFactory
File "/home/vagrant/.virtualenvs/emmasocial/lib/python2.7/site-packages/django/test/client.py", line 21, in <module>
from django.db import close_connection
File "/home/vagrant/.virtualenvs/emmasocial/lib/python2.7/site-packages/django/db/__init__.py", line 11, in <module>
if settings.DATABASES and DEFAULT_DB_ALIAS not in settings.DATABASES:
File "/home/vagrant/.virtualenvs/emmasocial/lib/python2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
self._setup(name)
File "/home/vagrant/.virtualenvs/emmasocial/lib/python2.7/site-packages/django/conf/__init__.py", line 46, in _setup
% (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting DATABASES, but settings are not configured.
You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
We are defining DATABASES in settings.py as follows:
if "DATABASE_URL" in os.environ:
# Parse database configuration from $DATABASE_URL
DATABASES = {
'default': dj_database_url.config()
}
else:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
'NAME': 'emmasocial', # Or path to database file if using sqlite3.
'USER': 'root', # Not used with sqlite3.
'PASSWORD': '', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
}
I'm running the tests from within the vagrant shell inside <app>/api/tests with the following command python ./AccountGrowth.py.
Can anyone shed some light on why this is happening?

You're not running the tests via Django's test runner, which sets all that up for you. You don't need that if __name__ == '__main__' block or its explicit call to run(), and you should run your tests via ./manage.py test.