Detecting and diagnosing workers silently crashing

I'm running Celery 2 using daemonizing - http://ask.github.com/celery/cookbook/daemonizing.html with RabbitMQ. From time to time silent crash happens, the only thing i see in celeryd.log:

[2010-12-24 14:14:31,323: INFO/PoolWorker-1414] process shutting down
[2010-12-24 14:14:31,323: INFO/PoolWorker-1414] process exiting with exitcode 0
[2010-12-24 14:14:31,331: INFO/PoolWorker-1415] child process calling self.run()
[2010-12-24 14:14:48,673: INFO/MainProcess] Got task from broker: airsale.search.xxx.get_search_results[01bf5d36-7c0e-4f8a-af69-750ef1b24abc]
[2010-12-24 14:14:48,761: INFO/MainProcess] Got task from broker: airsale.search.xxx.get_search_results[2d5f9952-d493-4de4-9752-0eee1776147d]
[2010-12-24 14:14:48,861: INFO/MainProcess] Got task from broker: airsale.search.xxx.get_search_results[0c77c1ec-df6c-4e34-875c-44909fbf8b9f]
[2010-12-24 14:14:48,961: INFO/MainProcess] Got task from broker: airsale.search.xxx.get_search_results[3d83dd54-0be8-4cf9-9cd6-81e070d97170]
[2010-12-24 14:14:49,061: INFO/MainProcess] Got task from broker: airsale.search.xxx.get_search_results[2dd29e70-e085-4fd1-a7ef-12d06b21644c]
..........

Then - only "Got task from broker" without any task processing.

ps -C celeryd shows - that celery nodes are running.

If i do : /etc/init.d/celeryd restart - number of celeryd processes doubles. Seems that old processes are uncontrolled by daemon any more.

1. How to detect - why task processing is not performed, even if ta开发者_开发百科sk is received from broker?
2. Why old celeryd processes are not killed by /etc/init.d/celeryd restart?

---

Queue workers are stalled, so the main solution to this is to adjust a the task time limit for each task, in a way to restart the worker when the tasks exceed this time.

Add to your task the following

from celery.decorators import task
from celery.exceptions import SoftTimeLimitExceeded


@task()
def mytask():
    try:
        do something()
    except SoftTimeLimitExceeded:
        clean something()

in your settings.py add the following

CELERYD_TASK_TIME_LIMIT = 30 #sec
CELERYD_TASK_SOFT_TIME_LIMIT = 30 #sec