Python⇒Speed: Clinging to memory: how Python function calls can increase your memory usage

Unlike languages like C, much of the time Python will free up memory for you. But sometimes, it won’t work the way you expect it to.

Consider the following Python program—how much memory do you think it will use at peak?

import numpy as np

def load_1GB_of_data():
    return np.ones((2 ** 30), dtype=np.uint8)

def process_data():
    data = load_1GB_of_data()
    return modify2(modify1(data))

def modify1(data):
    return data * 2

def modify2(data):
    return data + 10

process_data()

Presuming we can’t mutate the original data, the best we can do is peak memory of 2GB: for a brief moment in time both the original 1GB of data and the modified copy of the data will need to be present. In practice, actual peak usage will be 3GB—lower down you’ll see an actual memory profiling result demonstrating that.

The best we can do is 2GB, actual use is 3GB: where did that extra 1GB of memory usage come from? The interaction of function calls with Python’s memory management.

To understand why, and what you can do to fix it, this will article will cover:

1. A quick overview of how Python automatically manages memory for you.
2. How functions impact Python’s memory tracking.
3. What you can do to fix this problem.

Read more...

from Planet Python
via read more