Get startedGet started for free

Code profiling for memory usage

1. Code profiling for memory usage

We've defined efficient code as code that has a minimal runtime and a small memory footprint. So far, we've only covered how to inspect the runtime of our code. In this lesson, we'll cover a few techniques on how to evaluate our code's memory footprint.

2. Quick and dirty approach

One basic approach for inspecting memory consumption is using Python's built-in module sys. This module contains system specific functions and contains one nice method, dot-getsizeof, that returns the size of an object in bytes. sys-dot-getsizeof is a quick and dirty way to see the size of an object. But, this only gives us the size of an individual object. What if we wanted to inspect the line-by-line memory footprint of our code? You guessed it! We'll have to use a code profiler!

3. Code profiling: memory

Just like we've used code profiling to gather detailed stats on runtimes, we can also use code profiling to analyze the memory allocation for each line of code in our code base. We'll use the memory_profiler package that is very similar to the line_profiler package. It can be downloaded via pip and comes with a handy magic command (%mprun) that uses the same syntax as %lprun.

4. Code profiling: memory

One drawback to using %mprun is that any function profiled for memory consumption must be defined in a file and imported. %mprun can only be used on functions defined in physical files, and not in the IPython session. In this example, the convert_units function was placed in a file named hero_funcs-dot-py. Now, we simply import our function from the hero_funcs file and use the magic command %mprun to gather statistics on its memory footprint.

5. %mprun output

%mprun output is similar to %lprun output. Here, we see a line-by-line description of the memory consumption for the function in question.

6. %mprun output

The first column represents the line number of the code that has been profiled.

7. %mprun output

The second column (Mem usage) is the memory used after that line has been executed.

8. %mprun output

Next, the Increment column shows the difference in memory of the current line with respect to the previous one. This shows us the impact of the current line on the total memory usage.

9. %mprun output

The last column (Line Contents) shows the source code that has been profiled.

10. %mprun output

Profiling a function with %mprun allows us to see what lines are taking up a large amount of memory and possibly develop a more efficient solution. Before moving on, I want to draw your attention to a few things.

11. %mprun output caveats

First, the memory is reported in mebibytes. Although one mebibyte is not exactly the same as one megabyte, for our purposes, we can assume they are close enough to mean the same thing.

12. %mprun output caveats

Second, the heroes list, hts array, and wts array used to generate this output are not the original datasets we've used in the past. Instead, a random sample of 35,000 heroes was used to create these datasets in order to clearly show the power of using the memory_profiler.

13. %mprun output caveats

If we had used the original datasets, the memory used would have been too small to report. Don't worry too much about this - the main take away is that small memory allocation may not show up when using %mprun and that is a perfectly fine result. The random sample of 35,000 heroes was used solely to show typical %mprun output when the memory was large enough to report.

14. %mprun output caveats

Finally, the memory_profiler package inspects memory consumption by querying the operating system. This might be slightly different from the amount of memory that is actually used by the Python interpreter. Thus, results may differ between platforms and even between runs. Regardless, the important information can still be observed.

15. Let's practice!

Now that you've got the tools to evaluate memory consumption, put your skills to the test. Let's start using the memory_profiler!

Create Your Free Account

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.