The script that I've written to add tasks to the my Celery queue is leaking memory (to the point where the kernel kills the process after 20 minutes). In this script, I'm just executing the same 300 tasks repeatedly, every 60 seconds (inside a while True:).
The parameters passed to the task, makeGroupRequest(), are dictionaries containing strings, and according to hpy and objgraph, dicts and strings are also what's growing uncontrollably in memory. I've included the outputs of hpy below on successive iterations of the loop.
I've spent days on this, and I can't understand why memory would grow uncontrollably, considering nothing is re-used between loops. If I skip the retrieval of tasks, the memory doesn't appear to leak (so it's really the .get() call that is leaking memory). How can I determine what's going on and how to stop the growth?
Here is an outline of the code that's executing. I'm using the rpc:// backend.
while True:
# preparation is done here to set set up the arguments for the tasks (processedChains)
chains = []
for processedChain in processedChains:
# shorthanding
supportingData = processedChain["supportingDataAndCheckedGroups"]
# init the first element, which includes the supportingData and the first group
argsList = [(supportingData, processedChain["groups"][0])]
# add in the rest of the groups
argsList.extend([(groupInChain,) for groupInChain in processedChain["groups"][1:]])
# actually create the chain
chain = celery.chain(*[makeGroupRequest.signature(params, options={'queue':queue}) for params in argsList])
# add this to the list of chains
chains.append(chain)
groupSignature = celery.group(*chains).apply_async()
# this line appears to cause a large increase in memory each cycle
results = groupSignature.get(timeout = 2 * acceptableLoopTime)
time.sleep(60)
Here is the output of hpy on sucessive runs:
Loop 2:
Partition of a set of 366560 objects. Total size = 57136824 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 27065 7 17665112 31 17665112 31 dict (no owner)
1 122390 33 11966720 21 29631832 52 unicode
2 89133 24 8291952 15 37923784 66 str
3 45448 12 3802968 7 41726752 73 tuple
4 548 0 1631072 3 43357824 76 dict of module
5 11195 3 1432960 3 44790784 78 types.CodeType
6 9224 3 1343296 2 46134080 81 list
7 11123 3 1334760 2 47468840 83 function
8 1414 0 1274552 2 48743392 85 type
9 1414 0 1240336 2 49983728 87 dict of type
Loop 3:
Index Count % Size % Cumulative % Kind (class / dict of class)
0 44754 9 29240496 37 29240496 37 dict (no owner)
1 224883 44 20946280 26 50186776 63 unicode
2 89104 18 8290248 10 58477024 74 str
3 45455 9 3803288 5 62280312 79 tuple
4 14955 3 2149784 3 64430096 81 list
5 548 0 1631072 2 66061168 83 dict of module
6 11195 2 1432960 2 67494128 85 types.CodeType
7 11122 2 1334640 2 68828768 87 function
8 1402 0 1263704 2 70092472 88 type
9 1402 0 1236976 2 71329448 90 dict of type
Turns out this is a bug in Celery. Switching to the memcache backend completely resolves the memory leak. Hopefully the issue will be resolved in a subsequent version.
Related
I have a pretty big multithreading Python project that apparently has a memory leak somewhere. A DoctorThread shows me these (shortened) results:
Partition of a set of 418 objects. Total size = 96792 bytes.
Index Count % Size % Cumulative % Referrers by Kind (class / dict of class)
0 43 10 22792 24 22792 24 guppy.etc.Glue.Interface
1 66 16 18480 19 41272 43 dict of guppy.etc.Glue.Owner
2 25 6 18344 19 59616 62 dict of guppy.etc.Glue.Share
3 8 2 8384 9 68000 70 guppy.etc.Glue.Share
4 86 21 6696 7 74696 77 dict (no owner)
5 22 5 6160 6 80856 84 guppy.etc.Glue.Owner
6 37 9 2608 3 83464 86 dict (no owner), dict of guppy.etc.Glue.Interface
7 28 7 2464 3 85928 89 guppy.heapy.heapyc.HeapView
8 11 3 1840 2 87768 91 <Nothing>
9 2 0 1112 1 88880 92 __builtin__.cell
<24 more rows. Type e.g. '_.more' to view.>
Partition of a set of 23178 objects. Total size = 1604224 bytes.
Index Count % Size % Cumulative % Referrers by Kind (class / dict of class)
0 11135 48 801440 50 801440 50 list
1 11153 48 602408 38 1403848 88 tuple
[...]
<95 more rows. Type e.g. '_.more' to view.>
Partition of a set of 45140 objects. Total size = 2987568 bytes.
Index Count % Size % Cumulative % Referrers by Kind (class / dict of class)
0 22114 49 1591936 53 1591936 53 list
1 22133 49 1195328 40 2787264 93 tuple
[...]
<95 more rows. Type e.g. '_.more' to view.>
Partition of a set of 66115 objects. Total size = 4337720 bytes.
Index Count % Size % Cumulative % Referrers by Kind (class / dict of class)
0 32524 49 2341216 54 2341216 54 list
1 32513 49 1755848 40 4097064 94 tuple
[...]
<104 more rows. Type e.g. '_.more' to view.>
Partition of a set of 88355 objects. Total size = 5739128 bytes.
Index Count % Size % Cumulative % Referrers by Kind (class / dict of class)
0 43644 49 3141856 55 3141856 55 list
1 43633 49 2356328 41 5498184 96 tuple
[...]
<104 more rows. Type e.g. '_.more' to view.>
Partition of a set of 110380 objects. Total size = 7097992 bytes.
Index Count % Size % Cumulative % Referrers by Kind (class / dict of class)
0 54734 50 3940576 56 3940576 56 list
1 54753 50 2956808 42 6897384 97 tuple
[...]
<97 more rows. Type e.g. '_.more' to view.>
As you can see, the number of list and tuple referrers increases gradually. And it never stops increasing. These two entries are the only ones that increase constantly.
The DoctorThread class looks like this:
class DoctorThread(threading.Thread):
def __init__(self):
super(DoctorThread, self).__init__()
self.daemon = True
self.hp = guppy.hpy()
def run(self):
time.sleep(5)
logging.info("Doctor Thread started - taking heap snapshots")
before_heap = self.hp.heap()
while not PippinNetwork.is_shutdown():
gc.collect()
leftover = self.hp.heap() - before_heap
print(leftover.byrcs)
time.sleep(2.0)
Memory consumption is equally increasing. How can I find the culprit of this leak?
Update: solved.
In hindsight it was a rampant list.append((object, object)). The guppy results could have been interpreted like that.
In case of memory leak in big source codes you can use 'pympler' in python to track for the reference in future work.
from pympler import muppy
from pympler import summary
all_objects = muppy.get_objects()
sum1 = summary.summarize(all_objects)
summary.print_(sum1)
you can filter objects as well. Only include those where you have doubt then you can see which object or variable is causing memory leak then see the logic behind that.
I am suggesting this method cause in this way you don't have to cut down the logic (cause your project would have the huge code)
In case of clarification, you can ask in comment.
Working with large datasets means worrying about memory usage. Is there a bulit-in function, neat hack or widely available package to determine the memory usage of a given type?
In the current case I am wondering how many bytes of memory a single pandas.timedelta Object requires, in order to determine how many of them I can reasonably load into local memory. A general method to determine the memory requirements of any type would be preferable, though.
this can be done by using python memory profiler
>>> from guppy import hpy; h=hpy()
>>> h.heap()
Partition of a set of 48477 objects. Total size = 3265516 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 25773 53 1612820 49 1612820 49 str
1 11699 24 483960 15 2096780 64 tuple
2 174 0 241584 7 2338364 72 dict of module
3 3478 7 222592 7 2560956 78 types.CodeType
4 3296 7 184576 6 2745532 84 function
5 401 1 175112 5 2920644 89 dict of class
6 108 0 81888 3 3002532 92 dict (no owner)
7 114 0 79632 2 3082164 94 dict of type
8 117 0 51336 2 3133500 96 type
9 667 1 24012 1 3157512 97 __builtin__.wrapper_descriptor
<76 more rows. Type e.g. '_.more' to view.>
>>> h.iso(1,[],{})
Partition of a set of 3 objects. Total size = 176 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 33 136 77 136 77 dict (no owner)
1 1 33 28 16 164 93 list
2 1 33 12 7 176 100 int
>>> x=[]
>>> h.iso(x).sp
0: h.Root.i0_modules['__main__'].__dict__['x']
>>>
tl/dr: how do I get the current memory usage of my python program using Guppy? Is there a simple command?
I'm trying to track memory usage in a python program using guppy. This is my first usage of guppy, so I'm not very sure of how it behaves. What I want is
to be able to plot the total usage as "time" progresses in a simulation. This is a basic bit of code for what I can do:
from guppy import hpy
import networkx as nx
h = hpy()
L=[1,2,3]
h.heap()
> Partition of a set of 89849 objects. Total size = 12530016 bytes.
> Index Count % Size % Cumulative % Kind (class / dict of class)
> 0 40337 45 3638400 29 3638400 29 str
> 1 21681 24 1874216 15 5512616 44 tuple
> 2 1435 2 1262344 10 6774960 54 dict (no owner)
But I would like to just know what the current size is (the 12530016 bytes). So I'd like to be able to call something like h.total() to get the total size. I'd be shocked if this doesn't exist as a simple command, but so far, looking through the documentation I haven't found it. It's probably documented, just not where I'm looking.
x = h.heap()
x.size
returns the total size. For example:
from guppy import hpy
import networkx as nx
h = hpy()
num_nodes = 1000
num_edges = 5000
G = nx.gnm_random_graph(num_nodes, num_edges)
x = h.heap()
print(x.size)
prints
19820968
which is consistent with the Total size reported by
print(x)
# Partition of a set of 118369 objects. Total size = 19820904 bytes.
# Index Count % Size % Cumulative % Kind (class / dict of class)
# 0 51057 43 6905536 35 6905536 35 str
# 1 7726 7 3683536 19 10589072 53 dict (no owner)
# 2 28416 24 2523064 13 13112136 66 tuple
# 3 516 0 1641312 8 14753448 74 dict of module
# 4 7446 6 953088 5 15706536 79 types.CodeType
# 5 6950 6 834000 4 16540536 83 function
# 6 584 0 628160 3 17168696 87 dict of type
# 7 584 0 523144 3 17691840 89 type
# 8 169 0 461696 2 18153536 92 unicode
# 9 174 0 181584 1 18335120 93 dict of class
# <235 more rows. Type e.g. '_.more' to view.>
Is there a way yo find total memory used by a process and whole program in Python Kivy .
I.e. some way by which i can find out :
total memory used by the program?
objects active and using how much memory ?
Thanks!
Heapy is a memory profiler for Python. Use it like this:
>>> from guppy import hpy
>>> h = hpy()
>>> h.heap()
The output will be something like this:
Partition of a set of 1449133 objects. Total size = 102766644 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 985931 68 46300932 45 46300932 45 str
1 24681 2 22311624 22 68612556 67 dict of pkgcore.ebuild.ebuild_src.package
2 49391 3 21311864 21 89924420 88 dict (no owner)
3 115974 8 3776948 4 93701368 91 tuple
4 152181 11 3043616 3 96744984 94 long
5 36009 2 1584396 2 98329380 96 weakref.KeyedRef
6 11328 1 1540608 1 99869988 97 dict of pkgcore.ebuild.ebuild_src.ThrowAwayNameSpace
7 24702 2 889272 1 100759260 98 types.MethodType
8 11424 1 851840 1 101611100 99 list
9 24681 2 691068 1 102302168 100 pkgcore.ebuild.ebuild_src.package
<54 more rows. Type e.g. '_.more' to view.>
This is "basically a snapshot of what's reachable in ram". I didn't do much Kivy development so I never got around to profiling, but I think this should work.
See:
Getting started with Heapy
How to use guppy/heapy for tracking down memory usage
I've followed excellent post here how to setup heapy with Django: http://www.toofishes.net/blog/using-guppy-debug-django-memory-leaks/
I've commanded hp.setref() and now after awhile I get also data with hp.heap():
>>> hp.heap()
Partition of a set of 12075 objects. Total size = 1515496 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 4048 34 339656 22 339656 22 str
1 3112 26 269368 18 609024 40 tuple
2 171 1 169992 11 779016 51 dict (no owner)
3 1207 10 144440 10 923456 61 list
4 49 0 102040 7 1025496 68 dict of module
5 591 5 66984 4 1092480 72 unicode
6 498 4 59760 4 1152240 76 function
7 433 4 51960 3 1204200 79 types.CodeType
8 57 0 50480 3 1254680 83 type
9 36 0 31584 2 1286264 85 dict of class
But what now? What I should understand from this output? How I should start tracking down where those 'str' and 'tuple' objects belong?
With get_rp, I get following output:
>>> hp.heap().get_rp()
Reference Pattern by <[dict of] class>.
0: _ --- [-] 12000 (0xd1d340 | 0xd88b50 | 0xf63f00 | __builtin__.Struct | __...
1: a [-] 137 dict (no owner): 0x761c30*160, 0x7655d0*1491, 0x781640*9...
2: aa ---- [-] 45 dict of django.db.models.options.Options: 0xcf3110...
3: a3 [-] 45 django.db.models.options.Options: 0xcf3110, 0xf0bb10...
4: a4 ------ [-] 140 dict of django.db.models.related.RelatedObject: 0x10bec...
5: a5 [-] 140 django.db.models.related.RelatedObject: 0xf14450...
6: a6 -------- [-] 63 dict of django.db.models.fields.related.ForeignKey: 0x...
7: a7 [+] 63 django.db.models.fields.related.ForeignKey: 0xf0e690...
8: a5b ------- [-] 7 dict of django.db.models.fields.related.OneToOneField: ...
9: a5ba [+] 7 django.db.models.fields.related.OneToOneField: 0x15447...
Is the correct assumption now that it's Django that is leaking memory? But what are those dict's who don't have owner?
I don't have any experience with heapy, but in my experience, Django (and most other Python programs) don't leak memory, but they also don't clean up memory as pristinely as some would like.
Also, Django has settings that cause it to consume memory for diagnostic reasons. For example, setting DEBUG=True can cause it to hold on to all SQL queries, so the longer the process runs, the more memory it uses.
UPDATE: Your problem isn't in your Python code. Look at the summary heapy is giving you: the total size of memory represented there is 1.5Mb! When Python programs truly leak, the most common cause is a leaky C extension. Do you have any C extensions that you are running under your Django process?