Limitations of hierarchical clustering

1. Limitations of hierarchical clustering

Hi everyone, welcome to the final video in the chapter! Now that you are familiar with hierarchical clustering let us look at the challenges when performing this type of clustering.

2. Measuring speed in hierarchical clustering

Let us design a small task to measure the speed of various iterations of hierarchical clustering in order to check how long it takes for iterations. We will use the timeit module to check runtime of functions. As the most time-consuming step in the process of hierarchical clustering is constructing the distance matrix through the linkage method, we will time the amount of time it takes to form the matrix. For the purpose of this exercise, we will use randomly generated data points on the XY plane. To test the limits of the algorithm, we will use an increasing number of data points.

3. Use of timeit module

To demonstrate the use of timeit to further check how much time it may take for a large number of points, let us look at how long it takes to run the linkage method for 100 points with randomly generated coordinates. We first import the random and timeit modules to generate the points and time the runtime of a function respectively. Next, we create a DataFrame with 100 points, and randomly generate 100 points within coordinates in the range of zero to a hundred. To check the time of a function in the interpreter, we use the percent symbol before the timeit keyword followed by the statement that we were about to run. The timeit module runs the function multiple times and reports the mean and standard deviations of the runtimes. When I run this code on a 2017 Macbook Air running Jupyter notebooks, the mean time to execute the statement is about 1-point-02 milliseconds. Let us now perform iterations for an increasing number of points, check how long it takes to run the linkage method and then plot a graph to compare the performace.

4. Comparison of runtime of linkage method

If you plot the runtime of the linkage method with the number of points, you can see that the runtime increases with the increase in number of data points. In addition to it, you would notice that the increase in run time is not linear with respect to the increase in data points, but quadratic. This makes the technique of hierarchical clustering infeasible for huge number of data points, for instance, the shopping habits of all customers in Walmart in a year.

5. Next up - exercises

Let us now try some exercises to measure the time of functions through timeit.

This exercise is part of the course

Cluster Analysis in Python

IntermediateSkill Level

4.8+

Start Course for Free

Before you are ready to classify news articles, you need to be introduced to the basics of clustering. This chapter familiarizes you with a class of machine learning algorithms called unsupervised learning and then introduces you to clustering, one of the popular unsupervised learning algorithms. You will know about two popular clustering techniques - hierarchical clustering and k-means clustering. The chapter concludes with basic pre-processing steps before you start clustering data.

Exercise 1: Unsupervised learning: basics Exercise 2: Unsupervised learning in real world Exercise 3: Pokémon sightings Exercise 4: Basics of cluster analysis Exercise 5: Pokémon sightings: hierarchical clustering Exercise 6: Pokémon sightings: k-means clustering Exercise 7: Data preparation for cluster analysis Exercise 8: Normalize basic list data Exercise 9: Visualize normalized data Exercise 10: Normalization of small numbers Exercise 11: FIFA 18: Normalize data

This chapter focuses on a popular clustering algorithm - hierarchical clustering - and its implementation in SciPy. In addition to the procedure to perform hierarchical clustering, it attempts to help you answer an important question - how many clusters are present in your data? The chapter concludes with a discussion on the limitations of hierarchical clustering and discusses considerations while using hierarchical clustering.

Exercise 1: Basics of hierarchical clustering Exercise 2: Hierarchical clustering: ward method Exercise 3: Hierarchical clustering: single method Exercise 4: Hierarchical clustering: complete method Exercise 5: Visualize clusters Exercise 6: Visualize clusters with matplotlib Exercise 7: Visualize clusters with seaborn Exercise 8: How many clusters?Exercise 9: Create a dendrogram Exercise 10: How many clusters in comic con data?Exercise 11: Limitations of hierarchical clustering

Current Exercise

Exercise 12: Timing run of hierarchical clustering Exercise 13: FIFA 18: exploring defenders

This chapter introduces a different clustering algorithm - k-means clustering - and its implementation in SciPy. K-means clustering overcomes the biggest drawback of hierarchical clustering that was discussed in the last chapter. As dendrograms are specific to hierarchical clustering, this chapter discusses one method to find the number of clusters before running k-means clustering. The chapter concludes with a discussion on the limitations of k-means clustering and discusses considerations while using this algorithm.

Exercise 1: Basics of k-means clustering Exercise 2: K-means clustering: first exercise Exercise 3: Runtime of k-means clustering Exercise 4: How many clusters?Exercise 5: Elbow method on distinct clusters Exercise 6: Elbow method on uniform data Exercise 7: Limitations of k-means clustering Exercise 8: Impact of seeds on distinct clusters Exercise 9: Uniform clustering patterns Exercise 10: FIFA 18: defenders revisited

Now that you are familiar with two of the most popular clustering techniques, this chapter helps you apply this knowledge to real-world problems. The chapter first discusses the process of finding dominant colors in an image, before moving on to the problem discussed in the introduction - clustering of news articles. The chapter concludes with a discussion on clustering with multiple variables, which makes it difficult to visualize all the data.

Exercise 1: Dominant colors in images Exercise 2: Extract RGB values from image Exercise 3: How many dominant colors?Exercise 4: Display dominant colors Exercise 5: Document clustering Exercise 6: TF-IDF of movie plots Exercise 7: Top terms in movie clusters Exercise 8: Clustering with multiple features Exercise 9: Clustering with many features Exercise 10: Basic checks on clusters Exercise 11: FIFA 18: what makes a complete player?Exercise 12: Farewell!