Iterate over large collection in django - cache problem

I need to iterate over large collection (3 * 10^6 elements) in Django to do some kind of analysis that can't be done using single SQL statement.

• Is it possible to turn off collection caching in django? (Caching all the data is not to be acceptable data has around 0.5GB)
• Is it possible to make django fetch collection in chunks? It seems that it tries to pre fetch whole collection in to the memory and then iterate over it. I think that observing the speed of execution:
  • iter(Coll.objects.all()).next() - this takes forever
  • iter(Coll.objects.all()[:100开发者_开发知识库00]).next() - this takes less than a second

---

Use QuerySet.iterator() to walk over the results instead of loading them all first.

---

It seams that the problem was caused by the database backend (sqlite) that doesn't support reading in chunks. I've used sqlite as the database will be trashed after I do all the computations but it seems that sqlite isn't good even for that.

Here is what I've found in django source code of sqlite backend:

class DatabaseFeatures(BaseDatabaseFeatures):
    # SQLite cannot handle us only partially reading from a cursor's result set
    # and then writing the same rows to the database in another cursor. This
    # setting ensures we always read result sets fully into memory all in one
    # go.
    can_use_chunked_reads = False