1. Code profiling for runtime
We've covered how to time our code using the magic command %timeit, which works well with bite-sized code. But, what if we wanted to time a large code base or see the line-by-line runtimes within a function?
In this lesson, we'll cover a concept called code profiling that allows us to analyze code more efficiently.
2. Code profiling
Code profiling is a technique used to describe how long, and how often, various parts of a program are executed.
The beauty of a code profiler is its ability to gather summary statistics on individual pieces of our code without using magic commands like %timeit.
We'll focus on the line_profiler package to profile a function's runtime line-by-line.
Since this package isn't a part of Python's Standard Library, we need to install it separately. This can easily be done with a pip install command.
Let's explore using line_profiler with an example.
3. Code profiling: runtime
Suppose we have a list of superheroes along with each hero's height (in centimeters) and weight (in kilograms) loaded as NumPy arrays.
4. Code profiling: runtime
We've also developed a function called convert_units that converts each hero's height from centimeters to inches and weight from kilograms to pounds.
5. Code profiling: runtime
If we wanted to get an estimated runtime of this function, we could use %timeit. But, this will only give us the total execution time. What if we wanted to see how long each line within the function took to run?
6. Code profiling: runtime
We would have to use %timeit on each individual line of our convert_units function.
But, that's a lot of manual work and not very efficient.
7. Code profiling: line_profiler
Instead, we can profile our function with the line_profiler package.
To use this package, we first need to load it into our session. We can do this using the command %load_ext followed by line_profiler.
Now, we can use the magic command %lprun, from line_profiler, to gather runtimes for individual lines of code within the convert_units function.
%lprun uses a special syntax. First, we use the -f flag to indicate we'd like to profile a function.
8. Code profiling: line_profiler
Next, we specify the name of the function we'd like to profile. Note, the name of the function is passed without any parentheses.
9. Code profiling: line_profiler
Finally, we provide the exact function call we'd like to profile by including any arguments that are needed.
10. %lprun output
The output from %lprun provides a nice table that summarizes the profiling statistics.
11. %lprun output
First, a column specifying the line number followed by a column displaying the number of times that line was executed (called the Hits column).
12. %lprun output
Next, the Time column shows the total amount of time each line took to execute.
13. %lprun output
This column uses a specific timer unit that can be found in the first line of the output. Here, the timer unit is listed in microseconds using scientific notation.
14. %lprun output
We see that line three took 13 timer units, or, roughly 13 microseconds to run.
15. %lprun output
The Per Hit column gives the average amount of time spent executing a single line. This is calculated by dividing the Time column by the Hits column.
16. %lprun output
Notice that line 9 was executed three times and had a total run time of three microseconds, one microsecond per hit.
17. %lprun output
The % Time column shows the percentage of time spent on a line relative to the total amount of time spent in the function. This can be a nice way to see which lines of code are taking up the most time within a function.
18. %lprun output
Finally, the source code is displayed for each line in the Line Contents column.
19. %lprun output caveats
You may notice that the Total time reported when using %lprun and the time reported from using %timeit do not match.
Remember, %timeit uses multiple loops in order to calculate an average and standard deviation of time, so the time reported from each of these magic commands aren't expected to match exactly.
20. Let's practice your new profiling powers!
With great power comes great responsibility! Let's put your new profiling power to good use by profiling some code.