I started with fresh install of readthedocs.org:
http://read-the-docs.readthedocs.org/en/latest/install.html
Next I have added SocialApp GitHub in admin panel and then connected my superuser to that GitHub account.
Then I went to github and I forked readthedocs repository.
https://github.com/Drachenfels/Test-Fork
Next I clicked import projects. Task never concludes but when I refresh page, repos are there.
I picked forked repository Test-Fork and I clicked build.
Task never finishes, when I refresh or start another one, they are stuck in state "Triggered". There is no error, nothing.
What is more I am default configuration of readthedocs.
I have running in the background following processes:
./manager.py runserver 9000
./manage.py celerdybeat --verbosity=3
./manage.py celeryd -E
./manage.py celercycam
redis-server
Do I miss anything at this point?
It looks like for me that despite having celery active and running tasks are never initiated nor killed, nor errored.
Problem was not with celery, tasks were running eagerly (what I was suspecting but not really sure), so as soon as they triggered they were executed.
Problem was that task responsible for building documentation (update_docs) was failing silently. Thus state 'Triggering' never concluded and build was never initiated. It happens that this error was my own fault I run django server on different port than it's in default settings. Exception was thrown, it was never logged, state of task was never updated, readthedocs was left in the limbo. I hope it will help some lost souls out there.
Related
I'm running a little python project to collect data.
It's being triggered by a scheduled GitHub Action script (every midnight ).
As part of expanding the project i've added the pycaret library to the project.
so currently installing the requirements for the project is taking about 15 minutes, plust running the python project is another 10 minutes. But the interesting part is that now the action/job is being killed with :
/home/runner/work/_temp/bad86621-8542-4ea5-ae93-6f59b7ee2463.sh: line 1: 4382 Killed
python main.py
Error: Process completed with exit code 137.
Now i'vew tried looking up the reason for the process being killed by i have found nothing in GitHub Actions
i'm running the job on
ubuntu-latest machine in GitHub actions.
i've set the job timeout to 60 minutes , so i don't think that is the issue.
Error 137 indicates that the container (runner/build agent) that builds your project received SIGKILL and terminated.
It can be initiated manually or by the host machine when the runner exceeds its allocated memory limit.
In your case, since it is initiated by Github itself, then it is generally due to being out of memory.
(P.S. This is a very late answer, but it may help some folks)
I'm running a local django development server together with virtualenv and it's been a couple of days that it behaves in a weird way. Sometimes I don't see any log in the console sometimes I see them.
A couple of times I've tried to quit the process and restart it and I've got the port already taken error, so I inspected the running process and there was still an instance of django running.
Other SO answers said that this is due to the autoreload feature, well so why sometimes I have no problem at all and sometimes I do?
Anyway For curiosity I ps aux| grep python and the result is always TWO running process, one from python and one from my activated "virtualenv" python:
/Users/me/.virtualenvs/myvirtualenv/bin/python manage.py runserver
python manage.py runserver
Is this supposed to be normal?
I've solved the mistery: Django was trying to send emails but it could not because of improper configuration, so it was hanging there forever trying to send those emails.
Most probably (I'm not sure here) Django calls an OS function or a subprocess to do so. The point is that the main process was forking itself and giving the job to a subprocess or thread, or whatever, I'm not expert in this.
It turns out that when your python is forked and you kill the father, the children can apparently keep on living after it.
Correct me if I'm wrong.
In my Heroku Django app, for the user account registration flow, there is a typical request activate account that happens when the user receives an email with a special "activate" URL. In the app, this activation should happen only once. The only identifier on that URL is an activation token. The token is used to "activate" the account, and also to identify the user (retrieve his username from the DB). Once this token is used, it is deleted.
For that reason, activation must happen only once. However, for really odd reasons, when the request is sent to my Heroku app, it triggers the activate function twice. I am quite sure this is not a programming mistake in the app, because on local development the activation is not called twice, and neither on staging environment (which is on Heroku too, almost identical to production settings). It only happens in production.
Here is what I see in heroku logs:
http://pastebin.com/QeuP74fa
The first quarter of this log is interesting. Notice that at some point activation succeeded and attempted to redirect the user to the next correct page: GET Request to /iro/dashboard. But then after that the activation request happens, hence the crash that you see related to a "NoneType" object has no attribute...
My Procfile looks like this
web: newrelic-admin run-program gunicorn --bind=0.0.0.0:$PORT --workers=1 --log-level=debug iroquote.wsgi:application
worker: python manage.py rqworker high default low
I had 2 web dynos running, and 1 worker dyno, when I found the bug. I have tried to scale down to 1 web dyno and 1 worker, same bug. Then 0 dynos at all, and restarted 1 web dyno, still 0 worker, same bug.
Might something related to the Heroku router calling the dyno twice, or might not.
Help?
Silently, it stopped happening. This was likely a routing problem in Heroku that affected my app.
While running the status command I get the following error:-
Am using rabbitmq as the messaging broker
I am following this blog
sudo /etc/init.d/celeryd status
Error: No nodes replied within time constraint
How can I Debug this error?
I have also checked this question. The answer there did not help.
django/celery - celery status: Error: No nodes replied within time constraint
Edit:-
After checking the logs of celery beat I found the following error
celerybeat raised exception <class 'gdbm.error'>: error(13, 'Permission denied')
Perhaps this is caused by celery not having write permissions for the celerybeat-schedule file. The docs you linked to show celery configured to use /var/run/celerybeat-schedule as the celery beat schedule file.
Does your process have write permissions to that file? If that directory is owned by root (as it should be) and your process is running as anything other than the root user, that could cause the permission denied errors.
Check that your permissions are correct and then try deleting that file then restarting everything.
Use the following command to find the problem :
C_FAKEFORK=1 sh -x /etc/init.d/celeryd start
This usually happens because there are problems in your source project(permission issues, syntax error etc.)
As mentioned in celery docs:-
If the worker starts with “OK” but exits almost immediately afterwards
and there is nothing in the log file, then there is probably an error
but as the daemons standard outputs are already closed you’ll not be
able to see them anywhere. For this situation you can use the
C_FAKEFORK environment variable to skip the daemonization step
Good Luck
Source: Celery Docs
I'm having the same problem.
Restarting rabbitmq fixed it:
sudo systemctl restart rabbitmq-server
and the strange thing is that I needed to wait at least 100 seconds.
For me, I think there is a disk problem.
When my new versions of my django application are deployed to heroku, the workers are forced to be restarted. I have some long running tasks which should perform some cleanup prior to being killed.
I have tried registering a worker_shutdown hook which doesn't every seem to get called.
I have also tried the answer in Notify celery task of worker shutdown but i am unclear of how to abort a given task from within this context as calling celery.task.control.active() throws an exception (celery is no longer running).
Thanks for any help.
If you control the deployment maybe you can run a script that does a Control.broadcast to a custom command that you can register beforehand and only after receiving the required replies (you'd have to implement that logic) you'd continue the deployment (or raise a TimeoutException)?
Also, celery already has a predefined command for shutdown which I'm guessing you could overload in your instance or Subclass of worker. Commands have the advantage of being passed a Panel instance which allows you access to the consumer. That should expose a lot of control right there...