dictionary interface for large data sets

I have a set of key/values (all text) that is开发者_运维技巧 too large to load in memory at once. I would like to interact with this data via a Python dictionary-like interface.

Does such a module already exist?

Reading key values should be efficient and values compressed on disk to save space.

Edit:

• Ideally cross platform, but only using Linux for now
• Needs to be thread safe

---

Take a look at the shelve module. It provides dictionaries with various backing database systems include BerkleyDB and dbm.

---

Try bsddb(Python bindings for Berkley db)

---

ended up creating my own solution based on this existing module.

---

Assuming you are on a *nix platform, you are just BEGGING for tokyo-cabinet... It has a native set of ruby bindings... You can more information at 1978th.net...