Code profiling for memory usage
1. Code profiling for memory usage
We've defined efficient code as code that has a minimal runtime and a small memory footprint. So far, we've only covered how to inspect the runtime of our code. In this lesson, we'll cover a few techniques on how to evaluate our code's memory footprint.2. Quick and dirty approach
One basic approach for inspecting memory consumption is using Python's built-in module sys. This module contains system specific functions and contains one nice method, dot-getsizeof, that returns the size of an object in bytes. sys-dot-getsizeof is a quick and dirty way to see the size of an object. But, this only gives us the size of an individual object. What if we wanted to inspect the line-by-line memory footprint of our code? You guessed it! We'll have to use a code profiler!3. Code profiling: memory
Just like we've used code profiling to gather detailed stats on runtimes, we can also use code profiling to analyze the memory allocation for each line of code in our code base. We'll use the memory_profiler package that is very similar to the line_profiler package. It can be downloaded via pip and comes with a handy magic command (%mprun) that uses the same syntax as %lprun.4. Code profiling: memory
One drawback to using %mprun is that any function profiled for memory consumption must be defined in a file and imported. %mprun can only be used on functions defined in physical files, and not in the IPython session. In this example, the convert_units function was placed in a file named hero_funcs-dot-py. Now, we simply import our function from the hero_funcs file and use the magic command %mprun to gather statistics on its memory footprint.5. %mprun output
%mprun output is similar to %lprun output. Here, we see a line-by-line description of the memory consumption for the function in question.6. %mprun output
The first column represents the line number of the code that has been profiled.7. %mprun output
The second column (Mem usage) is the memory used after that line has been executed.8. %mprun output
Next, the Increment column shows the difference in memory of the current line with respect to the previous one. This shows us the impact of the current line on the total memory usage.9. %mprun output
The last column (Line Contents) shows the source code that has been profiled.10. %mprun output
Profiling a function with %mprun allows us to see what lines are taking up a large amount of memory and possibly develop a more efficient solution. Before moving on, I want to draw your attention to a few things.11. %mprun output caveats
First, the memory is reported in mebibytes. Although one mebibyte is not exactly the same as one megabyte, for our purposes, we can assume they are close enough to mean the same thing.12. %mprun output caveats
Second, the heroes list, hts array, and wts array used to generate this output are not the original datasets we've used in the past. Instead, a random sample of 35,000 heroes was used to create these datasets in order to clearly show the power of using the memory_profiler.13. %mprun output caveats
If we had used the original datasets, the memory used would have been too small to report. Don't worry too much about this - the main take away is that small memory allocation may not show up when using %mprun and that is a perfectly fine result. The random sample of 35,000 heroes was used solely to show typical %mprun output when the memory was large enough to report.14. %mprun output caveats
Finally, the memory_profiler package inspects memory consumption by querying the operating system. This might be slightly different from the amount of memory that is actually used by the Python interpreter. Thus, results may differ between platforms and even between runs. Regardless, the important information can still be observed.15. Let's practice!
Now that you've got the tools to evaluate memory consumption, put your skills to the test. Let's start using the memory_profiler!Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.