开发者

python memory leak, leaking frames

开发者 https://www.devze.com 2023-01-23 04:42 出处:网络
I have this code to get the caller of my function\'s file name, line number, and function. It seems to be leaking frames though and I don\'t understand why. Is this just throwing me off and my leak mu

I have this code to get the caller of my function's file name, line number, and function. It seems to be leaking frames though and I don't understand why. Is this just throwing me off and my leak must be elsewhere?

        rv = "(unknown file)", 0, "(unknown function)"

            for f in inspect.stack()[1:]:
                if __file__ in f:
                    continue
                else:
                    rv = f[1:4]
                    break

        return rv

I'm not saving a reference to the frame anywhere. But it's definitely frames that are leaking:

> objcallgraph.show_most_common_types()
>tuple                      24798
>frame                      9601
>...

Update: My frames are definitely being leaked. I did the suggestion about gc.set_debug() and frames are ve开发者_如何学运维ry slowly going into the gc.garbage list. Not even close to how many are being created though as show in show_most_common_types(). I have a question about scope though, in the above, doesn't f go out of scope after the for loop? Because I just tried this:


for f in range(20):
    l = 1

print f

and it printed 19. So could it be my f in the for loop leaking? This is a reference graph of a frame reference that was in my gc.garbage list:

python memory leak, leaking frames

Update2:

It looks like the inspect module itself is holding references to the frames. This is an objectgraph of a backreference from a live frame, not one on the garbage list.

python memory leak, leaking frames

Link here because it's too wide.

Is there a way to clear the inspect module? Where the hell are these frames being saved =\


EDIT I just realized that's wrong, the f reference and f_back both point the same way. I'll leave it in case it inspires someone else:

Each frame has an f_back pointer, so when you set f = inspect.stack()[1] then inspect.stack()[0][0].f_locals (which contains f) now has a reference to ...stack()[1] and ...stack()[1][0].f_back points to ...stack()[0][0]. So you created a circular reference which must be resolved by GC instead of simply by reference count. The GC is not tuned to deal with your rate of object creation so you consume more and more memory.

You could probably eliminate the circular reference just by setting f = None on your way out of the function. That breaks the circular reference.


Well I think I found the problem. It seems to be an issue with the inspect module and the underlying C code in a multi-threaded app. The module with the code above is being imported from different threads. And that 2nd graph points to the issue.

python memory leak, leaking frames

The function listed here in the 3rd node down is inspect.getmodule(). I couldn't fit it all in there and had to do some cropping.

(Pdb) objgraph.at(3510928)
<cell at 0x359290: dict object at 0x3849c0>

And inside the dict is all the frames

(Pdb) objgraph.at(0x3849c0)

     {(<frame object at 0x97a288>, '/lib/python26.zip/logging/__init__.py'): <module 'logging' from '/lib/python26.zip/logging/__init__.py'>,
     (<frame object at 0x896288>, '/lib/python26.zip/logging/__init__.py'): <module 'logging' from '/lib/python26.zip/logging/__init__.py'>,
     (<frame object at 0xa621b0>, '/lib/python26.zip/logging/__init__.py'): <module 'logging' from '/lib/python26.zip/logging/__init__.py'>,
     (<frame object at 0x11266e8>, '/lib/python26.zip/logging/__init__.py'): <module 'logging' from '/lib/python26.zip/logging/__init__.py'>,
    ...}

And if you get the outer frames of all those frames

(Pdb) inspect.getouterframes(objgraph.at(0x97a288))
[(<frame object at 0x97a288>, '/lib/python26.zip/logging/__init__.py', 1028, 'debug', ['            self._log(DEBUG, msg, args, **kwargs)\n'], 0),
 (<frame object at 0x794040>, '/lib/python26.zip/logging/__init__.py', 1505, 'debug', ['    root.debug(*((msg,)+args), **kwargs)\n'], 0),
 (<frame object at 0x794e58>, '/mmc/src/core/controller/main.py', 1046, '__startCharge', ['            self.chargeLock.release()\n'], 0),
 (<frame object at 0x5c4260>, '/mmc/src/core/controller/main.py', 1420, 'watchScheduleStartChargeCondition', ['                        ret = self.__startCharge(0, eventCode=eventCode)\n'], 0),
 (<frame object at 0x5c0dd0>, '/home/ephibian/Python2/_install/lib/python2.6/threading.py', 484, 'run', None, None),
 (<frame object at 0x5c3b48>, '/home/ephibian/Python2/_install/lib/python2.6/threading.py', 532, '__bootstrap_inner', None, None),
 (<frame object at 0x218170>, '/home/ephibian/Python2/_install/lib/python2.6/threading.py', 504, '__bootstrap', None, None)]

They all point to the __bootstrap method in threading. I could be on the wrong track here, but the context of some of these frames are nowhere near where I'm calling the method I posted.

0

精彩评论

暂无评论...
验证码 换一张
取 消