We Keep Coding

profiling a python module from the command line

I have a python project where I execute the app as a module using the -m flag. So something like:
python -m apps.validate -i input.mp4
Now, I want to profile it using the command line. So the inline examples suggest invoking cProfile itself a module. However, I cannot do something like:
python -m cProfile apps.validate -i input.mp4
However, this results in the error "No such file or directory". I cannot just go to the apps directory and launch validate.py due to relative imports.
Is there a way to profile a module on the command line?

Instead of running cProfile in shell, maybe you can use cProfile in your python script by adding some code in apps.validate or creating a new script and import apps.validate like this. Maybe some typo below :)
import cProfile
import sys
def run_validate(args):
# run your apps.validate code with shell arguments args here
pass
if __name__ == '__main__':
pr = cProfile.Profile()
pr.enable()
run_validate(*sys.argv)
pr.disable()
pr.print_stats()
then just run the original script: python -m apps.validate -i input.mp4

Cannot start Celery Worker (Kombu.asynchronous.timer)

I followed the first steps with Celery (Django) and trying to run a heavy process in the background. I have RabbitMQ server installed. However, when I try,
celery -A my_app worker -l info it throws the following error
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "c:\anaconda3\lib\site-packages\celery\concurrency\prefork.py", line
18, in <module>
from celery.concurrency.base import BasePool
File "c:\anaconda3\lib\site-packages\celery\concurrency\base.py", line 15,
in <module>
from celery.utils import timer2
File "c:\anaconda3\lib\site-packages\celery\utils\timer2.py", line 16, in
<module>
from kombu.asynchronous.timer import Entry
ModuleNotFoundError: No module named 'kombu.asynchronous.timer'
I've searched a lot, but can't seem to get it working. Any help will be highly appreciated. Thank you!

I had this issue with the default Celery installation from pip (3.1.26Post2). As mentionned above, I installed instead version 3.1.25, but Celery was still not working. Thus I explicitly installed the latest version:
pip install Celery==4.3
and everything is working now!

I landed here after I tried to install django-celery while reading celery 4.4 documentation, this package forces celery version to 3.1.26.post2, so I had to:
pip uninstall django-celery
pip uninstall celery && pip install celery # Uninstall 3.1 and install latest
As documentation clearly says:
Django is supported out of the box now so this document only contains a basic way to integrate Celery and Django. You’ll use the same API as non-Django users so you’re recommended to read the First Steps with Celery tutorial first and come back to this tutorial.

TL;DR: remove the kombu directory from the root of your virtualenv (if it exists). It may only fail on Windows.
It seems to be a quirk. I found the same error and I checked out what was happening.
The wheel package that pip downloads looks fine (kombu.asynchronous.timer exists in it). The release for the last version (currently 4.2.0) also is fine. What was strange is what I found in my virtualenv installation.
I found a kombu directory at my virtualenv root which has the content of the library but it also has an "async" directory, alongside an "asynchronous" one. These directories aren't from the 4.2.0 release, as async has the timer.py file but asynchronous doesn't.
From where did it come? It appears that from the wheel's data directory.
So, the solution: I removed the kombu directory from the root of my virtualenv and celery worked.

I have the same problem, but solved it when reinstall celery with version 3.1.25
pip uninstall celery && pip install celery==3.1.25
Maybe because windows is not officially supported by celery 4， https://github.com/celery/celery/issues/3551

I tested celery on the same python version you have and it is okay. and also https://github.com/celery/kombu/blob/master/kombu/asynchronous/timer.py shows that renaming things randomly is not going to help you. Maybe you should try pip uninstall kombu && pip --no-cache-dir install -U kombu to perform a fresh install for kombu. I guess there must be something wrong with your installation. so if the kombu reinstall thing didn't work, try installing the whole thing again.

I just started with Celery.
I followed instructions and installed Celery v 4.2.0
when I was trying to run the command :
celery -A mysite worker -l info
I got the error :
ModuleNotFoundError: No module named 'kombu.asynchronous.timer
I removed Celery installation : pip uninstall celery
Afterwards installed Celery 3.1.25 as 'chuhy' recommended
but..It had some other issues, so I immediately un-installed 3.1.25 , and reinstalled celery v4.2.0 .
After this scenario the error didn't pop again.

I have faced similar type of issue this is because of older version of celery. Uninstall celery (pip uninstall celery) and install again (pip install Celery==4.3) and kaboom it will work.

So I am running windows 10, python3.9.x, using aws sqs as the broker, just finished updating some files:
settings.py
###
### For celery tasks!
###
from kombu.utils.url import safequote
import urllib.parse
AWS_ACCESS_KEY_ID = 'my aws_access_key_id for a user'
AWS_SECRET_ACCESS_KEY = 'my aws_secret_access_key for a user'
BROKER_URL = 'sqs://%s:%s#' % (urllib.parse.quote(AWS_ACCESS_KEY_ID, safe=''), urllib.parse.quote(AWS_SECRET_ACCESS_KEY, safe=''))
BROKER_TRANSPORT = 'sqs'
BROKER_TRANSPORT_OPTIONS = {
'canves-celery-queue': {
'access_key_id': safequote(AWS_ACCESS_KEY_ID),
'secret_access_key': safequote(AWS_SECRET_ACCESS_KEY),
'region': 'us-east-1'
}
}
CELERY_DEFAULT_QUEUE = 'celery<-project-queue>'
CELERY_QUEUES = {
CELERY_DEFAULT_QUEUE: {
'exchange': CELERY_DEFAULT_QUEUE,
'binding_key': CELERY_DEFAULT_QUEUE,
}
}
###
### End celery tasks
###
celery_tasks.py (referred to in the tutorial as celery.py - renamed because apparently that caused some other programmers some errors):
from __future__ import absolute_import
import os
from celery import Celery
# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<project>.settings')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app = Celery('<project>', include=['<project>.tasks'])
app.config_from_object('django.conf:settings')
# Load task modules from all registered Django apps.
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print("debug_task was fired!")
print(f'Request: {self.request!r}')
if __name__ == '__main__':
app.start()
tasks.py (this is in the same directory as the settings.py - also referrenced in celery_tasks.py)
from __future__ import absolute_import
from .celery_tasks import app
import time
#app.task(ignore_result=True)
def sleep(x, y):
print("Sleeping for: " + str(x + y))
time.sleep(x + y)
print("Slept for: " + str(x + y))
When I went to run the worker (make sure you are in the same directory as manage.py), it threw this error:
from kombu.async.timer import Entry, Timer as Schedule, to_timestamp, logger
to fix it, I ran as per gogaz's answer
pip uninstall django-celery
pip uninstall celery && pip install celery
which pushed me to the latest version of celery, 4.3... celery 4+ isn't supported on windows as per this SO question (Celery raises ValueError: not enough values to unpack), which conveniently has this answer (posted by Samuel Chen):
for celery 4.2+, python3, windows 10
pip install gevent
celery -A <project> worker -l info -P gevent
for celery 4.1+, python3, windows 10
pip install eventlet
celery -A <project> worker -l info -P eventlet
The only other error I get is from django's debugger being on, which apparently causes memory leaks...
The problem (for me at least) is that I can't use the Prefork pool, which means that I can't use app.control.revoke() to terminate tasks.
---EDIT---
Also worth mentioning that after this answer was posted, I switched to a linux box. Unknown to me, due to a lack of experience, there are different modes you can run background tasks in. I don't remember all the names, but if you type into google "celery multithreading vs gevent", it will likely come back with some other modes you can run celery in, their purposes and which ones are supported for each platform. Windows couldn't run the mode that I thought made the most sense for my problem (I believe it was multithreading), and that was a real issue. However linux can run all of them... so I switched back to linux, just for celery. I had some problems with DJango in a redhat environment, so I had to fix those issues as well :|

I work on Windows so I had a bit of problem with this.
But my solution was to create new conda env with python 3.6.8 ( as i have understand celery may work on python 3.7 but have lot of problems).
Then proceed to install latest versions of celery(4.3.0) and Django(2.2.3) and after that everything worked fine.

Intellij/Pycharm can't debug Python modules

I use PyCharm/IntelliJ community editions from a wile to write and debug Python scripts, but now I'm trying to debug a Python module, and PyCharm does a wrong command line instruction parsing, causing an execution error, or maybe I'm making a bad configuration.
This is my run/debug configuration:
And this is executed when I run the module (no problems here):
/usr/bin/python3.4 -m histraw
But when I debug, this is the output in the IntelliJ console:
/usr/bin/python3.4 -m /opt/apps/pycharm/helpers/pydev/pydevd.py --multiproc --client 127.0.0.1 --port 57851 --file histraw
/usr/bin/python3.4: Error while finding spec for '/opt/apps/pycharm/helpers/pydev/pydevd.py' (<class 'ImportError'>: No module named '/opt/apps/pycharm/helpers/pydev/pydevd')
Process finished with exit code 1
As you can see, the parameters are wrong parsed, and after -m option a IntelliJ debug script is passed before the module name.
I also tried just put -m histraw in the Script field, but doesn't work, that field is only to put Python script paths, not modules.
Any ideas?

There is another way to make it work.You can write a python script to run your module.Then just configure PyCharm to run this script.
import sys
import os
import runpy
path = os.path.dirname(sys.modules[__name__].__file__)
path = os.path.join(path, '..')
sys.path.insert(0, path)
runpy.run_module('<your module name>', run_name="__main__",alter_sys=True)
Then the debugger works.

In PyCharm 2019.1 (professional), I'm able to select run as module option under configurations, as below

I found it easiest to create a bootstrap file (debuglaunch.py) with the following contents.
from {package} import {file with __main__}
if __name__ == '__main__':
{file with __main__}.main()
For example, to launch locustio in the pycharm debugger, I created debuglaunch.py like this:
from locust import main
if __name__ == '__main__':
main.main()
And configured pycharm as follows.
NOTE: I found I was not able to break into the debugger unless I added a breakpoint on main.main() . That may be specific to locustio, however.

The problem is already fixed since PyCharm 4.5.2. See corresponding issue in PyCharm tracker:
https://youtrack.jetbrains.com/issue/PY-15230

Error "No such file or directory" when running Django ./manage.py

In my django project, the command ./manage.py [command] results in this error message:
: No such file or directory
The command python manage.py [command] works well. I tried with syncdb and runserver.
I tried chmod a+x manage.py, but the problem persists.
My manage.py:
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
I use django 1.4.1 in a virtualenv.
How can I fix this to use manage.py [command]?

Likely, the reason is because your line endings in the manage.py file are \n instead of \r\n. As a result the #! hash-bang is misinterpreted.
This happens to me when I use a Windows based text-editor for my linux connection.

The #! hash-bang line doesn't point to your virtualenv python; replace the first line with:
#!/path/to/virtualenv/bin/python

In my django project, the command ./manage.py [command] results in
this error message:
: No such file or directory
The command python manage.py [command] works well
If specifying the interpreter makes it work, then it is the first line that must be wrong:
#!/usr/bin/env python
Try:
#!/usr/bin/python
(or wherever the interpreter is. Find it with: which python).

In my case, I was erroneously changing the sys.path in my manage.py.

In my case on Windows 7, every else it seems to be, but I've accidentally added an import in a views.py file:
from Scripts.pilprint import description
My software doesn't need this import, maybe with some wrong short-kut, my Eclipse wrote it for me, but removed this line, the problem disappear.
I suppose that description contain some painful character or have a wrong encoding for Windows.

Tornado 'Hello World' Error

I am trying to run the following HelloWorld Script at Command Line
import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
from tornado.options import define, options
define("port", default=8888, help="run on the given port", type=int)
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
def main():
tornado.options.parse_command_line()
application = tornado.web.Application([
(r"/", MainHandler),
])
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(options.port)
tornado.ioloop.IOLoop.instance().start()
if __name__ == "__main__":
main()
and I am getting the following error
File "helloworld.py", line 17, in ?
import tornado.httpserver
File "/home/username/public_html/tornado-1.2.1/tornado/httpserver.py", line 28, in ?
from tornado import ioloop
File "/home/username/public_html/tornado-1.2.1/tornado/ioloop.py", line 184
action if action is not None else signal.SIG_DFL)
^
SyntaxError: invalid syntax
Brand New to Python, can someone explain what the problem being pointed out is? P.S. helloworld.py is in the /home/username/public_html/tornado-1.2.1/ directory, and there is a tornado subdirectory in the same directory.
Edit: (Ignore this edit now)
The command i am running is
python helloworld.py
The result of python -V is
Python 2.4.3
Unfortunately Tornado doesn't work with versions before 2.5 so this might be the problem. However, I have installed Python 2.6.6 How do I ensure that it is running with the correct version of Python and not the older one?
EDIT II
Now I have set Python to 2.6.6
and running
python helloworld.py
doesn't produce any output. The program just freezes at the command line.
Any thoughts here?

As you've found out yourself, the problem is that python 2.4 does not support the conditional expression operator.
How you can switch to another Python version depends on your system. On debian and Ubuntu, you can edit /usr/share/python/debian_defaults. On all Linux systems, you can remove /usr/bin/python and link to the version you'd like:
sudo mv /usr/bin/python /usr/bin/python.dist
sudo ln -s /usr/bin/python2.5 /usr/bin/python
Alternatively, you can modify the PATH environment variable to contain a directory with the desired python binary before /usr/bin (this is probably the way to go on Windows). You can make this permanent by editing ~/.profile (at every login) or ~/.bashrc (in interactive, bash shells).

To get Python 2.6 as default make sure you've mapped python to /usr/bin/python2.6 in your .bash_rc.
If you're trying to fix this, you'll need to go through and swap out the conditional operator:
if seconds is not None:
signal.signal(signal.SIGALRM,
action if action is not None else signal.SIG_DFL)
This syntax (action if action is not None else signal.SIG_DFL) is only available in Python 2.>=5
The alt? Not as nice but workable:
if seconds is not None:
if action is not None:
tmpaction = action
else
tmpaction = signal.SIG_DFL
signal.signal(signal.SIGALRM,tmpaction)
I HIGHLY RECOMMEND THAT YOU SIMPLY UPGRADE TO THE LATEST VERSION OF PYTHON. THERE IS NO GUARANTEE THAT YOU WON'T FIND OTHER ISSUES. (Unless, of course, you want the learning experience).