I'm running a little python project to collect data.
It's being triggered by a scheduled GitHub Action script (every midnight ).
As part of expanding the project i've added the pycaret library to the project.
so currently installing the requirements for the project is taking about 15 minutes, plust running the python project is another 10 minutes. But the interesting part is that now the action/job is being killed with :
/home/runner/work/_temp/bad86621-8542-4ea5-ae93-6f59b7ee2463.sh: line 1: 4382 Killed
python main.py
Error: Process completed with exit code 137.
Now i'vew tried looking up the reason for the process being killed by i have found nothing in GitHub Actions
i'm running the job on
ubuntu-latest machine in GitHub actions.
i've set the job timeout to 60 minutes , so i don't think that is the issue.
Error 137 indicates that the container (runner/build agent) that builds your project received SIGKILL and terminated.
It can be initiated manually or by the host machine when the runner exceeds its allocated memory limit.
In your case, since it is initiated by Github itself, then it is generally due to being out of memory.
(P.S. This is a very late answer, but it may help some folks)
Related
Using docker (docker-compose) on macOS. When running the Docker containers and attaching Visual Studio Code (VSCode) to the active app container it can make the hyperkit process go crazy :( the macBook fans have to go at full speed to try to keep the temperature down.
When using VSCode on python files I noticed that actions, such as done by pylint, that result in scanning/parsing your file will increase the hyperkit CPU usage to the max and the macBook fans go on full speed :(. Hyperkit CPU usage goes down again when the action of pylint is finished.
When using VSCode to debug my Django Python app the hyperkit CPU usage goes to the max again. When actively debugging the hyperkit goes wild but it does settle down again afterwards.
I'm currently switching "bind mounts" to "volume mounts" I think I see some improvements but haven't done enough testing to say anything conclusive. I've only switched my source code to using "volume mount" instead of "bind mount" and will do the same for my static files and database and see if that results in improvements.
You can check out this stackoverflow post on Docker volumes for some more info on the subject.
Here is some post that I found regarding this issue:
https://code.visualstudio.com/docs/remote/containers?origin_team=TJ8BCJSSG
https://github.com/docker/for-mac/issues/1759
Any other ideas on how to keep the hyperkit process under control❓
[update 27 March] Docker debug mode was set to TRUE I've changed this to FALSE but I have not seen any significant improvements.
[update 27 March] Using "delegated" option for my source code (app) folder and first impressions are positive. I'm seeing significant performance improvements we'll have to see if it lasts 😀
FYI Docker docu on delegated: the container’s view is authoritative (permit delays before updates on the container appear in the host)
[update 27 March] I've also reduced the number of CPU cores Docker desktop can use (settings->advanced). Hopefully this prevents the CPU from getting too hot.
I "solved" this issue by using http://docker-sync.io to create volumes that I can mount without raising my CPU usage at all. I am currently running 8 containers (6 Python and 2 node) with file watchers on and my CPU is at 25% usage.
For some reason, Django just started taking a really long time to load.
When running python manage.py runserverit takes about 30 seconds for the server to run. But everything looks normal.
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
March 12, 2020 - 19:59:26
Django version 3.0.3, using settings 'navio.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
And after, every page takes about 22 seconds to load.
Any clue about what is happening? The only thing I did before this happened was creating and then deleting a file called templatetags.py.
I had the same problem, restarting the computer fixed it.
As an option you can install django-debug-toolbar to check CPU/DB time and find a long running query, if that's a problem.
Another option is to profile django app. Here are some articles you may want to read.
If this is production app, you can check for Newrelic for a trial period.
Possible reasons are:
long running queries
requests to external servises, that respond with noticible delay (or fail after timeout)
insufficient system resourses
many other things
Something to ask is are doing any processing of your assets?
I was very stupidly running npx instead of npm when in debug mode to install a freshly minified version. npx runs a npm package without installing it, which means everytime before it runs, it downloads the package, run the commands, then cleans up after itself... all in about 3seconds per command.
I created a a very simple test that launches and close a software I was testing using Python Nose test platform to track down a bug in the start up sequence of the software I was working on.
The test was set up so that it would launch and close about 1,500 times in a singling execution.
A few hours later, I discovered that the test was not able to launch to the software around after 300 iterations. It was timing out while waiting for the process to start. As soon as I logged back in, the test started launching the process without any problem and all the tests started passing as well.
This is quite puzzling to me. I have never seen this behavior. This never happened on Windows also.
I am wondering if there is a sort of power saving state that Mac was waiting for currently running process to finish and prohibits new process from starting.
I would really appreciate if anybody can shed light on this confusion.
I was running Python 2.7.x on High Sierra.
I am not aware of any state where the system flat out denies new processes while old ones are still running.
However, I can easily imagine a situation in which a process may hang because of some unexpected dependency on e.g. the window server. For example, I once noticed that rsvg-convert, a command-line SVG-to-image converter, running in an SSH session, had different fonts available to it depending on whether I was also simultaneously logged in on the console. This behavior went away when I recompiled the SVG libraries to exclude all references to macOS specific libraries...
I started with fresh install of readthedocs.org:
http://read-the-docs.readthedocs.org/en/latest/install.html
Next I have added SocialApp GitHub in admin panel and then connected my superuser to that GitHub account.
Then I went to github and I forked readthedocs repository.
https://github.com/Drachenfels/Test-Fork
Next I clicked import projects. Task never concludes but when I refresh page, repos are there.
I picked forked repository Test-Fork and I clicked build.
Task never finishes, when I refresh or start another one, they are stuck in state "Triggered". There is no error, nothing.
What is more I am default configuration of readthedocs.
I have running in the background following processes:
./manager.py runserver 9000
./manage.py celerdybeat --verbosity=3
./manage.py celeryd -E
./manage.py celercycam
redis-server
Do I miss anything at this point?
It looks like for me that despite having celery active and running tasks are never initiated nor killed, nor errored.
Problem was not with celery, tasks were running eagerly (what I was suspecting but not really sure), so as soon as they triggered they were executed.
Problem was that task responsible for building documentation (update_docs) was failing silently. Thus state 'Triggering' never concluded and build was never initiated. It happens that this error was my own fault I run django server on different port than it's in default settings. Exception was thrown, it was never logged, state of task was never updated, readthedocs was left in the limbo. I hope it will help some lost souls out there.
I am currently trying to run a long running python script on Ubuntu 12.04. The machine is running on a Digital Ocean droplet. It has no visible memory leaks (top shows constant memory). After running without incident (there are no uncaught exceptions and the used memory does not increase) for about 12 hours, the script gets killed.
The only messages present in syslog relating to the script are
Sep 11 06:35:06 localhost kernel: [13729692.901711] select 19116 (python), adj 0, size 62408, to kill
Sep 11 06:35:06 localhost kernel: [13729692.901713] send sigkill to 19116 (python), adj 0, size 62408
I've encountered similar problems before (with other scripts) in Ubuntu 12.04 but the logs then contained the additional information that the scripts were killed by oom-killer.
Those scripts, as well as this one, occupy a maximum of 30% of available memory.
Since i can't find any problems with the actual code, could this be an OS problem? If so, how do i go about fixing it?
Your process was indeed killed by the oom-killer. The log message “select … to kill“ hints to that.
Probably your script didn’t do anything wrong, but it was selected to be killed because it used the most memory.
You have to provide more free memory, by adding more (virtual) RAM if you can, by moving other services from this machine to a different one, or by trying to optimize memory usage in your script.
See e.g. Debug out-of-memory with /var/log/messages for debugging hints. You could try to spare your script from being killed: How to set OOM killer adjustments for daemons permanently? But often killing some process at random may leave the whole machine in an unstable state. In the end you will have to sort out the memory requirements and then make sure enough memory for peak loads is available.