The problem with Perl subs

Home : Perl : Subs

Note: This was written on 13-Dec-2002; it started out as a bug report, until I learned that the "bug" was a well-known design flaw in perl memory management. -- rgr, 8-Jan-05.]

Introduction

I can't claim to understand Perl memory management very thoroughly, but I have learned a few things (after much tearing of hair).

In particular, I was attempting to write a combinatorically hairy application using closures. I used lots of them -- tens to hundreds of thousands of them, in fact. After an initial attempt using a more conventional procedural approach, it occurred to me to recast the problem as a series of relatively simple processing steps that use closures to pass results on to the next step. The resulting code was substantially easier to understand and modify, but I was unable to do anything useful with it, because as soon as I tried it on anything large, it ran out of memory. Memory usage was roughly proportional to runtime, with different constant multipliers depending on application options. This is despite the fact that I have tried to be careful about limiting the amount of memory used at any one time; indeed, I spent days developing new controls over memory consumption. If the code worked the way I think I wrote it, the memory usage should fluctuate up and down, instead of rising steadily.

As it turns out, this was doomed from the start. There is a specific reason why closures are particularly bad, but first it is worth noting that even simple Perl programs have unpredictable memory usage patterns. Consider the following script:

#!/usr/bin/perl

sub test3 {
    my $n = shift;
    my $state = [];

    for (my $i = 0; $i < $n; $i++) {
	push(@$state, [1, 2, 3, 4, 5]);
    }
}

$inner = shift(@ARGV) || 1000;
$outer = shift(@ARGV) || 100;
for (my $i = 0; $i < $outer; $i++) {
    test3($inner);
}

[rogers@rgrjr memtest]$ PERL_DEBUG_MSTATS=2 perl ./test3.pl 100 100
Memory allocation statistics after compilation: (buckets -8(4)..8180(8192)
   10740 free:     0     0     0    31    26     4   3   5     3   1 0 0
   84948 used:     0     0     0   161   326   140   5   3     7  13 1 1
Total sbrk(): 106496/20:169. Odd ends: pad+heads+chain+tail: 0+8760+0+2048.
Memory allocation statistics after execution:   (buckets -8(4)..8180(8192)
   10640 free:     0     0     0    30   122     4   3   5     0   0 0 0
  112424 used:     0     0     0   162   326   140   5   3    12  24 1 1
Total sbrk(): 135168/27:176. Odd ends: pad+heads+chain+tail: 0+10056+0+2048.
[rogers@rgrjr memtest]$ PERL_DEBUG_MSTATS=2 perl ./test3.pl 500 100
Memory allocation statistics after compilation: (buckets -8(4)..8180(8192)
   10740 free:     0     0     0    31    26     4   3   5     3   1 0 0
   84948 used:     0     0     0   161   326   140   5   3     7  13 1 1
Total sbrk(): 106496/20:169. Odd ends: pad+heads+chain+tail: 0+8760+0+2048.
Memory allocation statistics after execution:   (buckets -8(4)..8180(8192)
   37740 free:     0     0     0    30   506     4   3   5     1   1 1 0
  221236 used:     0     0     0   162   326   140   5   3    31  68 1 1
Total sbrk(): 280576/48:197. Odd ends: pad+heads+chain+tail: 0+15456+0+6144.
[rogers@rgrjr memtest]$ PERL_DEBUG_MSTATS=2 perl ./test3.pl 1000 100
Memory allocation statistics after compilation: (buckets -8(4)..8180(8192)
   10740 free:     0     0     0    31    26     4   3   5     3   1 0 0
   84948 used:     0     0     0   161   326   140   5   3     7  13 1 1
Total sbrk(): 106496/20:169. Odd ends: pad+heads+chain+tail: 0+8760+0+2048.
Memory allocation statistics after execution:   (buckets -8(4)..8180(8192)
   72544 free:     0     0     0    30  1018     4   3   5     1   1 1 1
  355468 used:     0     0     0   162   326   140   5   3    55 122 1 1
Total sbrk(): 458752/63:212. Odd ends: pad+heads+chain+tail: 0+22548+4096+4096.
[rogers@rgrjr memtest]$ 

Interestingly, the totals hardly change at all if I leave the size of the inner loop fixed and increase the size of the outer loop:

[rogers@rgrjr memtest]$ PERL_DEBUG_MSTATS=2 perl ./test3.pl 100 500
Memory allocation statistics after compilation: (buckets -8(4)..8180(8192)
   10740 free:     0     0     0    31    26     4   3   5     3   1 0 0
   84948 used:     0     0     0   161   326   140   5   3     7  13 1 1
Total sbrk(): 106496/20:169. Odd ends: pad+heads+chain+tail: 0+8760+0+2048.
Memory allocation statistics after execution:   (buckets -8(4)..8180(8192)
   10640 free:     0     0     0    30   122     4   3   5     0   0 0 0
  112424 used:     0     0     0   162   326   140   5   3    12  24 1 1
Total sbrk(): 135168/27:176. Odd ends: pad+heads+chain+tail: 0+10056+0+2048.
[rogers@rgrjr memtest]$ PERL_DEBUG_MSTATS=2 perl ./test3.pl 100 1000
Memory allocation statistics after compilation: (buckets -8(4)..8180(8192)
   10740 free:     0     0     0    31    26     4   3   5     3   1 0 0
   84948 used:     0     0     0   161   326   140   5   3     7  13 1 1
Total sbrk(): 106496/20:169. Odd ends: pad+heads+chain+tail: 0+8760+0+2048.
Memory allocation statistics after execution:   (buckets -8(4)..8180(8192)
   10640 free:     0     0     0    30   122     4   3   5     0   0 0 0
  112424 used:     0     0     0   162   326   140   5   3    12  24 1 1
Total sbrk(): 135168/27:176. Odd ends: pad+heads+chain+tail: 0+10056+0+2048.
[rogers@rgrjr memtest]$ 

So the extra "used" memory would seem to have to do with entering/exiting the subroutine scope. But it has nothing to do with subroutines per se, since it is possible to remove the subroutine:

for (my $i = 0; $i < $outer; $i++) {
    my $n = $inner;
    my $state = [];

    for (my $i = 0; $i < $n; $i++) {
	push(@$state, [1, 2, 3, 4, 5]);
    }
}

The perlguts page tells about something called "scratchpads," which would seem to be the only thing that could explain this. Running the test3.pl script with "-DX" (see perlrun) suggests that scratchpads are not freed until perl exits (though I'll spare you the wallpaper). This happens after the dump_mstats call in perl_run in perl.c, so presumably in perl_destruct; I can't find exactly where, but this is too late to do any good in any case.

So my working theory is that Perl leaves all these scratchpads hanging around, with active pointers to values that therefore cannot be reclaimed until after the next scratchpad use. And, since my app uses tons of closures, expecting their storage to disappear when they become unreferenced, I am guessing that a scratchpad is created for each closure, and is never thrown away, perhaps because the closure itself is referenced from a scratchpad. (Lisp garbage collectors sometimes have the analogous problem with pointers to garbage being left in registers that are "dead" as far as the code is concerned, but still "live" to the GC. But that only delays reclamation, and never permanently inhibits it.) Or perhaps there is another bug relating to closures; I haven't gotten that far yet.

FWIW, there is a routine called "Perl_pad_reset" in op.c, but an an old comment in front of it says that it is disabled because it has bugs. So it may be that the Perl community has decided to live with scratchpad memory lossage . . .

[somewhat later . . .]

I did find that Perl 5.003 Changes includes the following:

NETaa13721: pad_findlex core dumps on bad CvOUTSIDE()
From: Carl Witty
Files patched: op.c sv.c toke.c
 Each CV has a reference to the CV containing it lexically.  Unfortunately,
 it didn't reference-count this reference, so when the outer CV was freed,
 we ended up with a pointer to memory that got reused later as some other kind
 of SV.

And a message from Dave Mitchell <davem at fdgroup.co.uk> to <perl5-porters at perl.org> (among others) on 8-Aug-01 titled Re: [ID 20010807.013] Garbage collection and/or memory leak problem in perl5 explains that scratchpads are created for each occurrence of an operator, and even quotes the same pad_reset comment. This ought to explain the memory usage patterns shown above (though I haven't tried to verify that).

Finally, the section of This Week on p5p 2001/03/12 titled "CvOUTSIDE" (scroll down a page or two to the third section) explains that subs within subs necessarily have circular refcounts. It describes this as "Alan's Great Subroutine Memory Leak (the problem with sub x { sub {} })". [At the time, I found a good page that describes this problem, but I can't find it now. -- rgr, 8-Jan-05.]

So it looks like I need to recode the thing again, this time in terms of Perl objects. Fortunately, that seems like a good thing to do in any case.

[much later . . .]

Well, I did rewrite my application in terms of perl objects, and it did indeed turn out to be all for the good. However, I still often want to use closures for various purposes, e.g. for passing to Getopt::Long, and it irks me that this makes the object and everything it contains "permanent." At least for Getopt::Long purposes, the objects that are affected are top-level ones that tend to live until the script exits anyway, so the loss is not appreciable. -- rgr, 8-Jan-05.