Zope ZODB Pack error

When trying to pack my zope instance's ZODB, I get this error:

2011-02-28 10:47:21 ERROR ZODB.DB packing
Traceback (most recent call last):
  File "/usr/local/www/cstools/parts/zope2/lib/python/ZODB/DB.py", line 618, in pack
    self._storage.pack(t, referencesf)
  File "/usr/local/www/cstools/parts/zope2/lib/python/ZEO/ClientStorage.py", line 865, in pack
    return self._server.pack(t, wait)
  File "/usr/local/www/cstools/parts/zope2/lib/python/ZEO/ServerStub.py", line 161, in pack
    self.rpc.call('pack', t, wait)
  File "/usr/local/www/cstools/parts/zope2/lib/python/ZEO/zrpc/connection.py", line 539, in call
    raise inst # error raised by server
AttributeError: 'NoneType' object has no attribute 'append'
2011-02-28 10:47:21 ERROR Zope.SiteErrorLog https://conservationtoolstest.tnc.org:54321/Control_Panel/Database/main/manage_pack
Traceback (innermost last):
  Module ZPublisher.P开发者_运维百科ublish, line 115, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 41, in call_object
  Module <string>, line 3, in _facade
  Module AccessControl.requestmethod, line 64, in _curried
  Module App.ApplicationManager, line 430, in manage_pack
  Module ZODB.DB, line 618, in pack
  Module ZEO.ClientStorage, line 865, in pack
  Module ZEO.ServerStub, line 161, in pack
  Module ZEO.zrpc.connection, line 539, in call
AttributeError: 'NoneType' object has no attribute 'append'

Passed to the call method are:

call(self, method, *args)

self:  <ManagedClientConnection ('<IP REMOVED>', 8100)>
method:  pack
args: (1298908370.5440209, 1)

From the zeoserver.log:

(11117) Error raised in delayed method
Traceback (most recent call last):
  File "/usr/local/www/cstools/buildout-cache/eggs/ZODB3-3.8.1-py2.4-linux-i686.egg/ZEO/StorageServer.py", line 1247, in run
    result = self._method(*self._args)
  File "/usr/local/www/cstools/buildout-cache/eggs/ZODB3-3.8.1-py2.4-linux-i686.egg/ZEO/StorageServer.py", line 416, in _pack_impl
    self.storage.pack(time, referencesf)
  File "/usr/local/www/cstools/buildout-cache/eggs/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB/FileStorage/FileStorage.py", line 1247, in pack
    opos = p.pack()
  File "/usr/local/www/cstools/buildout-cache/eggs/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB/FileStorage/fspack.py", line 481, in pack
    self.gc.findReachable()
  File "/usr/local/www/cstools/buildout-cache/eggs/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB/FileStorage/fspack.py", line 227, in findReachable
    self.findReachableAtPacktime([z64])
  File "/usr/local/www/cstools/buildout-cache/eggs/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB/FileStorage/fspack.py", line 303, in findReachableAtPacktime
    todo.extend(self.findrefs(pos))
  File "/usr/local/www/cstools/buildout-cache/eggs/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB/FileStorage/fspack.py", line 376, in findrefs
    return referencesf(self._file.read(dh.plen))
  File "/usr/local/www/cstools/buildout-cache/eggs/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB/serialize.py", line 622, in referencesf
    u.noload()
AttributeError: 'NoneType' object has no attribute 'append'

And finally, here the pegged versions:

buildout.dumppickedversions=0.4
elementtree=1.2.6-20050316
hexagonit.recipe.cmmi=1.1.1
hexagonit.recipe.download=1.2.1
html5lib=0.11.1
infrae.subversion=1.1
ipython=0.8.4
lxml=2.1.5
MySQL-python=1.2.2
PILwoTk = 1.1.6.4
plone.recipe.command=1.0
plone.recipe.distros=1.3
plone.recipe.zope2install=2.2
plone.recipe.zope2instance=2.3.1
plone.recipe.zope2zeoserver=0.13
py=0.9.0
python-ldap=2.3.4
pisa=3.0.32
reportlab=2.4
SessionCrumbler=0.2
setuptools = 0.6c11
simplejson=2.0.9
topp.helpers=1.0
zc.buildout=1.1.1
z3c.recipe.staticlxml=0.7.1
zc.recipe.egg=1.1.0
zc.recipe.cmmi=1.1.5
ZODB3=3.8.1
ZConfig=2.6.1
zdaemon=2.0.2
zest.recipe.mysql=0.7
zlib=1.2.4
zope.interface=3.4.1
zope.proxy=3.4.0
zope.testing=3.5.1

Can anybody help me figure out why this is happening?

Thank you! :)

---

Note: this is a revised answer after the question was updated with a traceback and version info

Your pack fails during garbage collection because you have a corrupted pickle in your ZODB database. You can verify this by running the fsrefs.py script on your Data.fs. If you have a zopepy or similar part in your buildout (a script that just runs the arguments as a python script with the sys.path extended with all your eggs), you can do this:

bin/zopepy /usr/local/www/cstools/buildout-cache/eggs/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB/scripts/fsrefs.py var/filestorage/Data.fs

That's the full path to the fsrefs.py script. It'll only confirm if there is a pickle error. Most likely this is problem with an old transaction you don't care about anymore, and you can fix this problem by editing the serialize.py module to ignore this particular pickle error. You'll find the path to it at the end of your traceback.

Edit the referencesf function (line 608 and on) to catch the exception in question. Originally the top of that function looks like this:

refs = []
u = cPickle.Unpickler(cStringIO.StringIO(p))
u.persistent_load = refs
u.noload()
u.noload()

Change it to:

refs = []
u = cPickle.Unpickler(cStringIO.StringIO(p))
u.persistent_load = refs
u.noload()
try:
    u.noload()
except AttributeError:
    print "Ignoring AttributeError during garbage collection unpickling."
    pass

The risk is that the garbage collection algorithm will not detect a set of objects that are still in use and thus pack away too much, but that risk diminishes with the age of the affected transaction. The risk appears minimal, after all that pickle is already broken and any application code won't be able to reach any object referenced through whatever that pickle held anyway.