开发者

Purging numpy.memmap

开发者 https://www.devze.com 2023-03-06 02:23 出处：网络

Given a numpy.memmap object created with mode=\'r\' (i.e. read-only), is there a way to force it to purge all loaded pages out of physical RAM, without deleting the object itself?

相关专题：large-data memory-mapped-files mmap numpy python

Given a numpy.memmap object created with mode='r' (i.e. read-only), is there a way to force it to purge all loaded pages out of physical RAM, without deleting the object itself?

In other words, I'd like the reference to the memmap instance to remain valid, but all physical memory that's being used to cache the on-disk data to be uncommitted. Any views onto to the memmap array must also remain valid.

I am hoping to use this as a d开发者_Go百科iagnostic tool, to help separate "real" memory requirements of a script from "transient" requirements induced by the use of memmap.

I'm using Python 2.7 on RedHat.

If you run "pmap SCRIPT-PID", the "real" memory shows as "[ anon ]" blocks, and all memory-mapped files show up with the file name in the last column.

Purging the pages is possible at C level, if you manage to get ahold of the pointer to the beginning of the mapping and call madvise(ptr, length, MADV_DONTNEED) on it, but it's going to be cludgy.