Python⇒Speed: The mmap() copy-on-write trick: reducing memory usage of array copies

Let’s say you have an array, and you need to make some copies and modify those copies. Usually, memory usage scales with the number of copies: if your original array was 1GB of RAM, each copy will take 1GB of RAM. And that can add up.

But often, you’re just changing a small part of the array. Ideally, the memory cost would only be the parts of the copies that you changed.

As it turns out, there is an operating system facility that enables this: mmap()’s copy-on-write functionality.

In this article you will learn:

1. How normal memory copies work.
2. How to use mmap() copy-on-write with NumPy.
3. How the underlying mmap() copy-on-write mechanism works, and why it can be more efficient.

Read more...

from Planet Python
via read more