I am currently in the lookout for performance monitoring tools, tools which produce an output of metrics of specific OS processes. I want something which is lightweight (not to effect the performance of the system it is currently monitoring), hence I wrote a simple bash script which uses the ps command to retrieve CPU% and Memory%, writes them to a file, and sleeps for a specified couple of seconds, and repeats until it is terminated.
The question I have is whether this is a correct approach because as found on the documentation of ps (here):
Since ps cannot run faster than the system and is run as any other
scheduled process, the information it displays can never be exact.
I know of other ways such as using ps_util in python and retrieve the information that way. However, is this (and any other tool) faster or more reliable? If so, can you recommend a tool?
Or is the ps command safe enough?
ps is primarily a snapshot tool that gives you an instant view of at that given moment. You cannot get instant CPU percentage from ps. The ps page gives out this information regarding CPU:
CPU usage is currently expressed as the percentage of time spent
running during the entire lifetime of a process. This is not ideal,
and it does not conform to the standards that ps otherwise conforms
to. CPU usage is unlikely to add up to exactly 100%.
You can use the tool top which is a monitoring tool that keeps polling for numbers over time. I'd recommend using top as it for cases where you want repetitive updates.
For a more detailed explanation on differences between top and ps, you can read the answers on this question: https://unix.stackexchange.com/questions/58539/top-and-ps-not-showing-the-same-cpu-result
Related
I'd like to profile the CPU time and CPU consumption of a python application on an embedded device. The ideal would be to have something similar to resource_memory in that you can directly enter a command in the prompt shell. I could read a lot about psutil and some other libs but the truth is I am running out of time and can't spend too much time learning how to make it work or add it to my application's code. So this isn't the ideal, it's overly complicated for what I need I believe. Also I don't want to have a line-by-line analysis, I just want the overall values of CPU consumption an use time during one run.
Is there something that matches my needs that someone could recommend to me ? Thank you!
The situation: I have a website that allows people to execute arbitrary code in a different language (specifically, an esolang I created), using a Python interpreter on a shared-hosting server. I run this code in a separate process which is given a time limit of 60 seconds.
The problem: You can do stuff like (Python equivalent) 10**(10**10), which rapidly consumes far more memory than I have allotted to me. It also, apparently, locks up Apache - or it takes too long to respond - so I have to restart it.
I have seen this question, but the given answer uses Perl, which I do not know at all, hence I'd like an answer in Python. The OS is Linux too, though.
Specifically, I want the following characteristics:
Runs automatically
Force-kills any process that exceeds some memory limit like 1MB or 100MB
Kills any process spawned by my code that is more than 24 hours old
I use this piece of code (in a Django view) to create the process and run it (proxy_prgm is a Manager so I can retrieve data from the program that's interpreting the esolang code):
prgmT[uid] = multiprocessing.Process(
target = proxy_prgm.runCatch,
args = (steps,),
name="program run")
prgmT[uid].start()
prgmT[uid].join(60) #time limit of 1 minute
if prgmT[uid].is_alive():
prgmT[uid].terminate()
proxy_prgm.stop()
If you need more details, don't hesitate to tell me what to edit in (or ask me questions).
Another approach that might work; using resource.setrlimit() (more details in this other StackOverflow answer). It seems that by doing so you can set a memory limit on a process and it's subprocesses; you'll have to figure out how to handle if the limit is hit though. I don't have personal experience using it, but hopefully doing so would stop Apache from locking up on you.
I have a python script, that calls HandBrakeCli as a subprocess on Linux. During normal operations, HandBrakeCli takes almost 100% of available CPU resources. But sometimes it seems to starve or whatever - it seems not to be active, but does not return for a long time. This seems to be due to a broken DVD (Handbrake is a dvd ripping tool) - it seems that Handbrake is not handling this so well. It will finish after a long while (hours), but bevor it does nothing. Whereas in normal operation it will use lots of cpu cycles.
So my idea to solve this was to have a watch dog that checks if there is a HandBrakeCli process, and if so monitors if it does use a good part of the cpu. If that is not the case for maybe 5 minutes, it will kill that process, so that the parent script can continue its operations.
It does not seem too hard to program this, but I have a feeling it could involve some tedium. Also it seems not unlikely that this problem has been solved before. Is there a solution for this, possibly in python? If there is a standalone program or daemone that does the job on Linux I would not mind if its not python, as long as it is open source.
Well perhaps the most obvious answer for something to monitor and control child process in python would have to be supervisord. It's very mature, and has a lot of features and documentation.
It is written in Python, and has a plugin API, as well as a service API and a web dashboard.
Your problem description is a little bit vague though, re 'starvation'. It does sound as if your job is blocking on some resource, but you might want to diagnose this a little more precisely before reaching for a complicated software scaffolding to prop it up.
i.e. perhaps it's blocked on I/O (network or disk bandwidth), or it's stalled reading from a pipeline that's failed. You might want to look into process control (e.g. the renice command ) to manage it's CPU allocation a little better etc.
I am using python programs to nearly everything:
deploy scripts
nagios routines
website backend (web2py)
The reason why I am doing this is because I can reuse the code to provide different kind of services.
Since a while ago I have noticed that those scripts are putting a high CPU load on my servers. I have taken several steps to mitigate this:
late initialization, using cached_property (see here and here), so that only those objects needed are indeed initialized (including import of the related modules)
turning some of my scripts into http services (with a simple web.py implementation, wrapping-up my classes). The services are then triggered (by nagios for example), with simple curl calls.
This has reduced the load dramatically, going from over 20 CPU load to well under 1. It seems python startup is very resource intensive, for complex programs with lots of inter-dependencies.
I would like to know what other strategies are people here implementing to improve the performance of python software.
An easy one-off improvement is to use PyPy instead of the standard CPython for long-lived scripts and daemons (for short-lived scripts it's unlikely to help and may actually have longer startup times). Other than that, it sounds like you've already hit upon one of the biggest improvements for short-lived system scripts, which is to avoid the overhead of starting the Python interpreter for frequently-invoked scripts.
For example, if you invoke one script from another and they're both in Python you should definitely consider importing the other script as a module and calling its functions directly, as opposed to using subprocess or similar.
I appreciate that it's not always possible to do this, since some use-cases rely on external scripts being invoked - Nagios checks, for example, are going to be tricky to keep resident at all times. Your approach of making the actual check script a simple HTTP request seems reasonable enough, but the approach I took was to use passive checks and run an external service to periodically update the status. This allows the service generating check results to be resident as a daemon rather than requiring Nagios to invoke a script for each check.
Also, watch your system to see whether the slowness really is CPU overload or IO issues. You can use utilities like vmstat to watch your IO usage. If you're IO bound then optimising your code won't necessarily help a lot. In this case, if you're doing something like processing lots of text files (e.g. log files) then you can store them gzipped and access them directly using Python's gzip module. This increases CPU load but reduces IO load because you only need transfer the compressed data from disk. You can also write output files directly in gzipped format using the same approach.
I'm afraid I'm not particularly familiar with web2py specifically, but you can investigate whether it's easy to put a caching layer in front if the freshness of the data isn't totally critical. Try and make sure both your server and clients use conditional requests correctly, which will reduce request processing time. If they're using a back-end database, you could investigate whether something like memcached will help. These measures are only likely to give you real benefit if you're experiencing a reasonably high volume of requests or if each request is expensive to handle.
I should also add that generally reducing system load in other ways can occasionally give surprising benefits. I used to have a relatively small server running Apache and I found moving to nginx helped a surprising amount - I believe it was partly more efficient request handling, but primarily it freed up some memory that the filesystem cache could then use to further boost IO-bound operations.
Finally, if overhead is still a problem then carefully profile your most expensive scripts and optimise the hotspots. This could be improving your Python code, or it could mean pushing code out to C extensions if that's an option for you. I've had some great performance by pushing data-path code out into C extensions for large-scale log processing and similar tasks (talking about hundreds of GB of logs at a time). However, this is a heavy-duty and time-consuming approach and should be reserved for the few places where you really need the speed boost. It also depends whether you have someone available who's familiar enough with C to do it.
I'm writing a web application (http://www.checkio.org/) which allows users to write python code. As one feedback metric among many, I'd like to enable profiling while running checks on this code. This is to allow users to get a very rough idea of the relative efficiency of various solutions.
I need the profile to be (reasonably) deterministic. I don't want other load on the web server to give a bad efficiency reading. Also, I'm worried that some profilers won't give a good measurement because these short scripts run so quickly. The timeit module shows a function being run thousands of time, but I'd like to not waste server reasources on this small features if possible.
It's not clear which (if any) of the standard profilers meet this need. Ideally the profiler would give units of "interpreter bytecode ticks" which would increment one per bytecode instruction. This would be a very rough measure, but meets the requirements of determinism and high-precision.
Which profiling system should I use?
Python's standard profiler module provides deterministic profiling.
I also suggest giving a try to yappi. (http://code.google.com/p/yappi/) In v0.62, it supports CPU time profiling and you can stop the profiler at any time you want...