Am I doing something wrong or does web.py leak memory?
import web
class Index:
def GET(self): return 'hello web.py'
app = web.application(('/*', 'Index'), globals())
app.run()
Run the above file. Watch how much memory the task uses. Go to localhost:8080 in your browser. Close the browser (to keep the page from getting cached), then open the page again, and see how the memory usage rises. It goes up every time you close the browser and re-visit the page.
Running python 2.6 on Win XP.
After running your code and sending it thousands of requests (via another Python process using urllib2), I find that it grows by about 200k over the course of the first few hundred requests and then stops growing. That doesn't seem unreasonable, and it needn't indicate a memory leak. Remember that Python uses automatic memory management via a combination of reference counting and garbage collection, so there's no guarantee that every bit of memory it uses is reusable the instant it's no longer in use; and it may request memory from the OS and then not return it even though it isn't needed any more.
So I think the answer is: You aren't doing anything wrong, but web.py doesn't leak memory.
Related
I have a Django REST Framework service. Once in a while, the service gets a large request and when that happens, it eats a large chunk of OS memory (a K8 pod in my case). After the request is processed and garbage collection is done (sometimes forced), the objects are released but the amount of memory occupied by Python never goes down until the OS starts running low on memory.
I have read this, this and this. And it looks like I cannot force Python to give that memory back to the OS. However, is there a way to limit the amount of memory Python has access to in the first place? This way, I can ensure that Python doesn't eat up memory on its own and other processes have plenty to play around with too. Kinda similar to how you can set the memory a JVM can use.
I am trying to figure out how a while loop determines how much memory to use.
At the basic level:
while True:
pass
If I did a similar thing in PHP, it would grind my localhost to a crawl. But in this case I would expect it to use next to nothing.
So for an infinite loop in python, should I expect that it would use up tons of memory, or does it scale according to what is being done, and where?
Your infinite loop is a no-op (it doesn't do anything), so it won't increase the memory use beyond what is being used by the rest of your program. To answer your question, you need to post the code that you suspect is causing memory problems.
In PHP however, the same loop will "hang" because the web server is expecting a response to send back to the client. Since no response is being received, the web browser will simply "freeze". Depending on how the web server is configured, it may choose to end the process an issue a timeout error.
You could do the same if you used Python and a web framework and put an infinite loop in one of your methods that returns a response to the client.
If you ran the equivalent PHP code from the shell, it will have the same effect as if it was written in Python (or any other language). That is, your console will block until you kill the process.
I'm asking because I want to create a program that runs infinitely,
but I'm not sure how to determine it's footprint or how much it will
take from system resources.
A program that runs indefinitely (I think that's what you mean) - it generally has two cases:
Its waiting to do some work on a trigger (like a web server runs indefinitely, but its just sitting there until someone visits your website)
Its doing a process that is taking a long time.
For #2, you need to determine the resource use by figuring out what is the work being done.
If its building a large list of items to do some calculations/sorting, then memory use will grow as the list grows.
If its processing a bunch of files, and during this process, it generates a lot of output stored on disk - then disk usage will grow, and then shrink when the process is done.
If its a rendering engine, then memory use and CPU use will increase, along with disk use as the memory is swapped out during rendering. However, such a system will not tax the disk too much.
The bottom line is, you can't get an answer to this unless you explain the process being run.
We develop the application that use Tornado(2.4) and tornadio2 for transport. We have a problem with memory leaks, and tried to find what is wrong with pympler, but it haven't catched any leaks. But the problem is. Pmap shows us that memory is used (look through the added with the link screenshot http://prntscr.com/16wv6k).
There are more then 90% of memory is used by one anon process.
Whith every user coming in our application reserve some memory, but with user out memory is still reserved and doesnt free. We can't understand what is the problem.
The question is - what should we do to remove this leaks? We have to rebot server every hour just for 500 user online. It's bad((
Maybe this (already closed) bug ?
https://github.com/facebook/tornado/commit/bff07405549a6eb173a4cfc9bbc3fc7c6da5cdd7
Do you know about an efficient way to log memory usage of a django app per request ?
I have an apache/mod_wsgi/django stack, which runs usually well, but sometimes one process ends up eating a huge lot of memory. The servers ends up being short on mem, swapping a lot, and services are dramatically slowed down.
This situation is quite hard to fix because I don't know which request is to be blamed for this behavior, I can't reproduce it.
I'd like to have something deployed in production which logs the memory usage of the process before and after each request, with minimal overhead.
Before I start reinventing the wheel, do the community of my fellow djangoists know any existing solution to address this problem ?
Advices, middleware, snippet or maybe apache log configuration appreciated.
What (I think) I don't need is:
a set of dev-stage profiling/debugging tools, I already know some and I'd use them if I knew what to profile/debug, it looks a little bit too much to be forever monitoring services running in production. On top of that, what is usually displayed by those tol is a mem usage report of the code shred to pieces, It would really be helpful to just pinpoint the faulty request.
generic advices on how to optimize mem usage of a django app, well it's always good to read, but the idea here is rather «how to efficiently track down requests which need to be optimized».
My closest search results:
Django / WSGI - How to profile partial request? My profiling tools are per-request but app runs out of memory before then
Django memory usage going up with every request
Average php memory usage per request?
A Django middleware for tracking memory usage and generating a usable result immediately, needs to hook both process request and process response. In other words, look at difference between start and finish of request and log a warning if exceeds some threshold.
A complete middleware example is:
import os
import psutil
import sys
THRESHOLD = 2*1024*1024
class MemoryUsageMiddleware(object):
def process_request(self, request):
request._mem = psutil.Process(os.getpid()).memory_info()
def process_response(self, request, response):
mem = psutil.Process(os.getpid()).memory_info()
diff = mem.rss - request._mem.rss
if diff > THRESHOLD:
print >> sys.stderr, 'MEMORY USAGE %r' % ((diff, request.path),)
return response
This requires the 'psutil' module to be installed for doing memory calculation.
Is brute force and can lead to false positives in a multithread system. Because of lazy loading, you will also see it trigger on first few requests against new process as stuff loads up.
This may not fully cover your question, but I recommend trying nginx+uwsgi instead of apache2+mod_wsgi. In my tests it turned out to be much more stable (mod_wsgi choked at some point completely), much faster and uses a lot less memory (it may just fix all your issues altogether).
About tracking memory usage, you can create a simple middleware:
class SaveMemoryUsageMiddleware(object):
def process_response(self, request, response):
# track memory usage here and append to file or db
return response
and add it to your middlewares.
For memory tracking code I recommend checking out:
Total memory used by Python process?
However, it would probably be better if you could avoid doing this on production. Just for dev and tests to track down real problem.
How can I profile my python / django application which is crashing on a single request after 100 seconds of hogging more memory?
All I see in top is that the wsgi process is consuming memory slowly until it crashes.
The only profiling techniques I know run on a full request/response cycle but I'm not able to finish a request. What then?
I might even run the dev server and try to kill it mid-request and see where the stack is.
A bit fiddly and will have some overhead, but you could use sys.setprofile() to provide a function to be called on entry and exit from functions and dump progress of calls out to a log file yourself, potentially with check of memory in use at the same time.
http://docs.python.org/dev/library/sys.html#sys.setprofile
Also perhaps check out heapy as a way of getting console type access into your live process to dump out memory/object usage.