Excessive Garbage Collection ?

Application details: pyQt application, Qt 4.4 and python 2.5

Issue: Over the time ( using application for long time w/o closing ) , the application gets slow sometimes. I tried most of the profiling application, but did not find anything significant to point out any issue.

My application creates multiple tables in different tabs using QTabWidget. So to speed up the tab switching I only construct the table once and keep the QTableWidgetItem in memory. The count of QTableWidgetItem ranges in ~5000 - 10000. There are more of such objects (# ~ 600 - 800) that are constructed and needs to be keep in memory until anything changes.

I was thinking could GC be slowing down my application sometimes. Because the default GC thresholds are 700,10,10 . In my case, 700 seems to be very less. And these QTableWidgetItem stays in memory until any data has changed. So could GC be kicking in frequently in phase 1, phase 2 and phase 3 ? Once the data is changed I just de-reference it by doing mytable.setRowCount(0). In this case, I do not need GC to reclaim memory. Am I right ?

Can I tell GC to not track some of these objects?

Please suggest if my understanding is wrong ?

Also, please let me know if question is unclear.

    def fetchData (self, win=None):
"""Fetches data from the data sources.
    """
start = time.clock()
changed = self.adjustDateRange()
if not changed and self.chartEdition == self.fetchEdition: return

badIngredients = []
ingredientList = self.getIngredientList()
parentWin = win
if parentWin is None:
    parentWin = qApp.activeWindow()

B = 0
N = len(ingredientList)
if N == 0: return

    progress   = QProgressDialog(parentWin)
    progress.setWindowModality(Qt.WindowModal)
progress.setWindowTitle ("FetchData Progress")
progress.setRange (0, N)
    #progress.forceShow()
self.lock()
try:
    for i in xrange(len(ingredientList)):
    progress.setValue(i)
    if B > 0:
        progress.setLabelText("Fetching time series %s of %s\n"\
                  "Number of missing series: %s" % \
                  (i+1,N,B))
    else:
                progress.setLabelText("Fetching time series %s of %s" % \
                    (i+1,N))
                print self, "Fetching time series ", i+1, " of ", N
    qApp.processEvents()
    ingredient = ingredientList[i]
    if progress.wasCanceled():
        for ingredient in ingredientList[i:len(ingredientList)]:
        ingredient.error = 1
        badIngredients.append(ingredient)
        break
    try:
        ingredient.getTimeSeries (win)
        ingredient.error = 0
    except:
      #Handle Exception here
              # Create a badIngredient list here
    for ingredient in badIngredients:
    self.deleteIngredient(ingredient)
finally:
    progress.setValue(N)
        progress.close()
    self.callWhenFree (self, self.chartChanged, remove=True)
    self.unlock()
    self.emit (SIGNAL("dataChanged"))
self.fetchEdition = self.chartEdition
end = time.clock()
print "Data Fetched in %s sec" %(end-start)

EDIT: While executing this loop, over the time the progress Dialog keeps getting slower and slower. But there are lot of other things happening like creating huge table for these data and more Qt objects. Also, the chart drawing of QGraphicsItem gets slower. Currently, I have commented the qApp.processEvent() for testing purpose, in bugs list of Qt it says that QProgressDialog.setValue internally calls processEvents so I do not need to call explicitly. Also there is a freezing problem where application just freezes at this progress dialog and needs to kill it.

I used objgraph http://mg.pov.lt/objgraph/ to get the top most 50 objects in memory This is what I get when I open the application for first time

tuple                      16243
dict                       6325
function                   4220
instance                   3814
QTreeWidgetItem            2849
wrapper_descriptor         2642
weakref                    1447
getset_descriptor          1387
list                       934
method_descriptor          815
builtin_function_or_method 805
wrappertype                660
type                       652
classobj                   267
module                     235
member_descriptor          181
QAction                    154
instancemethod             47
property                   36
frame     开发者_StackOverflow                 34
QWidget                    33
QColor                     33
QVBoxLayout                27
_Condition                 20
QLabel                     18
QMenu                      17
QToolButton                14
QTableWidgetItem           13
QGridLayout                13
SplitResult                13
QTimer                     11
TypeInfo                   10
classmethod_descriptor     10
QComboBox                  9
QLineEdit                  9
QCheckBox                  9
QSpacerItem                8
QGroupBox                  7
PenStyle                   6
set                        6
ChartViewAction            6
QHBoxLayout                6
MouseButton                6
QRadioButton               5
QActionGroup               5
QtMsgType                  5
Thread                     4
QPushButton                4
QToolBar                   4
staticmethod               4

After running the code that might be leaking, the top most 30 objects in memory.

tuple                      19948
QTableWidgetItem           9298
dict                       7562
function                   4220
instance                   4157
QTreeWidgetItem            2849
wrapper_descriptor         2788
QGraphicsRectItem          2246
weakref                    1527
getset_descriptor          1392
list                       1198
QGraphicsLineItem          825
method_descriptor          816
QGraphicsTextItem          812
builtin_function_or_method 811
QColor                     780
DQEllipse                  748
wrappertype                660
type                       652
QAction                    280
classobj                   267
module                     235
member_descriptor          189
instancemethod             110
dqComboBoxTableItem        66
property                   36
frame                      35
QWidget                    33
QVBoxLayout                27
GlobalColor                24

The QTableWidgetItems must be in memory. Currently, I am testing this application with disabling gc. Most of the objects that I see, are created and keep in memory like QGraphicsLineItem, QGraphicsRectItem, etc. When I create new objects, these objects are over-ridden with new objects thus making the old one de-referenced. What are the other means of testing memory leak?

---

You could try turning off garbage collection to test your theory with :

gc.disable()

or better still:

gc.set_debug(gc.DEBUG_LEAK)

See the relevant page in the python reference for more.