Task Chaining with Cursor issue on app engine. Exception: Too big query offset. Anyone else get this issue?

I'm not sure if anyone else has this problem, but I'm getting an exception "Too big query offset" when using a cursor for chaining tasks on appengine development server (not sure if it happens on live).

The error occurs when requesting a cursor after 4000+ records have been processed in a single query.

I wasn't aware that offsets had anything to do with cursors, and perhaps its just a quirk in sdk for app engine.

To fix, either shorten the time allowed before task is deferred (so fewer records get processed at a time) or when checking time elapsed you can also check the number of records processed is still within range. e.g, if time.time() > end_time or count == 2000.Reset count and defer task. 2000 is an arbitrary number, I'm not sure what the limit should be.

EDIT:

After making the above mentioned changes, the never finishes executing. The with_cursor(cursor) code is being called, but seems to start at the beginning each time. Am I missing something obvious?

The code that causes the exception is as follows:

The table "Transact" has 4800 rows. The error occurs when transacts.cursor() is called when time.time() > end_time is true. 4510 records have been processed at the time when the cursor is requested, which seems to cause the error (on development server, haven't tested elsewhere).

def some_task(trans):
  tts = db.get(trans)
  for t in tts:
    #logging.info('in some_task')
    pass


def test_cursor(request):
  ret = test_cursor_task()


def test_cursor_task(cursor = None):
  startDate = datetime.datetime(2010,7,30)
  endDate = datetime.datetime(2010,8,30)
  end_time = time.time() + 20.0   
  transacts = Transact.all().filter('transactionDate >', startDate).filter('transa开发者_StackOverflow中文版ctionDate <=',endDate)

  count =0
  if cursor:
      transacts.with_cursor(cursor)
  trans =[]
  logging.info('queue_trans')
  for tran in transacts:
    count+=1
    #trans.append(str(tran))   
    trans.append(str(tran.key()))   

    if len(trans)==20:
            deferred.defer(some_task, trans,  _countdown = 500)                
            trans =[]            
    if time.time() > end_time:
        logging.info(count)            
        if len(trans)>0:                
           deferred.defer(some_task, trans, _countdown = 500)
           trans =[]
        logging.info('time limit exceeded setting next call to queue')
        cursor = transacts.cursor()
        deferred.defer(test_cursor_task, cursor)
        logging.info('returning false')
        return False


    return True


  return HttpResponse('')

Hope this helps someone.

Thanks Bert

---

Try this again without using the iter functionality:

#...
CHUNK = 500
objs = transacts.fetch(CHUNK)

for tran in objs:
    do_your_stuff

if len(objs) == CHUNK:
    deferred.defer(my_task_again, cursor=str(transacts.cursor()))

This works for me.