In
statistics,
hierarchical clustering is a method of
cluster analysis which seeks to build a
hierarchy of clusters. Strategies for hierarchical clustering generally fall into two types:
- Agglomerative: This is a "bottom up" approach: each observation starts in its own cluster, and pairs of clusters are merged as one moves up the hierarchy.
- Divisive: This is a "top down" approach: all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy.
In general, the merges and splits are determined in a
greedy manner. The results of hierarchical clustering are usually presented in a
dendrogram.
In the general case, the complexity of agglomerative clustering is
, which makes them too slow for large data sets. Divisive clustering with an exhaustive search is
, which is even worse. However, for some special cases, optimal efficient agglomerative methods (of complexity
) are known: SLINK
[1] for single-linkage and CLINK
[2] for complete-linkage clustering.
[edit] Cluster dissimilarity
In order to decide which clusters should be combined (for agglomerative), or where a cluster should be split (for divisive), a measure of dissimilarity between sets of observations is required. In most methods of hierarchical clustering, this is achieved by use of an appropriate
metric (a measure of
distance between pairs of observations), and a linkage criterion which specifies the dissimilarity of sets as a function of the pairwise distances of observations in the sets.
[edit] Metric
The choice of an appropriate metric will influence the shape of the clusters, as some elements may be close to one another according to one distance and farther away according to another. For example, in a 2-dimensional space, the distance between the point (1,0) and the origin (0,0) is always 1 according to the usual norms, but the distance between the point (1,1) and the origin (0,0) can be 2,
or 1 under Manhattan distance, Euclidean distance or maximum distance respectively.
Some commonly used metrics for hierarchical clustering are:
[3]
For text or other non-numeric data, metrics such as the
Hamming distance or
Levenshtein distance are often used.
A review of cluster analysis in health psychology research found that the most common distance measure in published studies in that research area is the Euclidean distance or the squared Euclidean distance.
[citation needed]
[edit] Linkage criteria
The linkage criteria determines the distance between sets of observations as a function of the pairwise distances between observations.
Some commonly used linkage criteria between two sets of observations
A and
B are:
[4][5]
where
d is the chosen metric. Other linkage criteria include:
- The sum of all intra-cluster variance.
- The increase in variance for the cluster being merged ( Ward's criterion).[6]
- The probability that candidate clusters spawn from the same distribution function (V-linkage).
[edit] Discussion
Hierarchical clustering has the distinct advantage that any valid measure of distance can be used. In fact, the observations themselves are not required: all that is used is a matrix of distances.
[edit] Example for Agglomerative Clustering
For example, suppose this data is to be clustered, and the
Euclidean distance is the
distance metric.
Cutting the tree at a given height will give a partitioning clustering at a selected precision. In this example, cutting after the second row will yield clusters {a} {b c} {d e} {f}. Cutting after the third row will yield clusters {a} {b c} {d e f}, which is a coarser clustering, with a smaller number of larger clusters.
Raw data
The hierarchical clustering
dendrogram would be as such:
Traditional representation
This method builds the hierarchy from the individual elements by progressively merging clusters. In our example, we have six elements {a} {b} {c} {d} {e} and {f}. The first step is to determine which elements to merge in a cluster. Usually, we want to take the two closest elements, according to the chosen distance.
Optionally, one can also construct a
distance matrix at this stage, where the number in the
i-th row
j-th column is the distance between the
i-th and
j-th elements. Then, as clustering progresses, rows and columns are merged as the clusters are merged and the distances updated. This is a common way to implement this type of clustering, and has the benefit of caching distances between clusters. A simple agglomerative clustering algorithm is described in the
single-linkage clustering page; it can easily be adapted to different types of linkage (see below).
Suppose we have merged the two closest elements
b and
c, we now have the following clusters {
a}, {
b,
c}, {
d}, {
e} and {
f}, and want to merge them further. To do that, we need to take the distance between {a} and {b c}, and therefore define the distance between two clusters. Usually the distance between two clusters
and
is one of the following:
-
-
- The mean distance between elements of each cluster (also called average linkage clustering, used e.g. in UPGMA):
-
- The sum of all intra-cluster variance.
- The increase in variance for the cluster being merged ( Ward's method[6])
- The probability that candidate clusters spawn from the same distribution function (V-linkage).
Each agglomeration occurs at a greater distance between clusters than the previous agglomeration, and one can decide to stop clustering either when the clusters are too far apart to be merged (distance criterion) or when there is a sufficiently small number of clusters (number criterion).
[edit] Software
- R has several functions for hierarchical clustering: see CRAN Task View: Cluster Analysis & Finite Mixture Models for more information.
- Orange, a free data mining software suite, module orngClustering for scripting in Python, or cluster analysis through visual programming.
- hcluster is Python software, based on NumPy, which supports hierarchical clustering and plotting.
- Cluster 3.0 provides a nice Graphical User Interface to access to different clustering routines and is available for Windows, Mac OS X, Linux, Unix. See: [1]
- ELKI includes multiple hierarchical clustering algorithms.
- figue A JavaScript package that implements some agglomerative clustering functions (single-linkage, complete-linkage, average-linkage) and functions to visualize clustering output (e.g. dendograms) (Online demo).
- MultiDendrograms An open source Java application for variable-group agglomerative hierarchical clustering,[7] with graphical user interface.
- CrimeStat implements two hierarchical clustering routines, a nearest neighbor (Nnh) and a risk-adjusted(Rnnh).
- Complete C# DEMO implemented as visual studio project that includes real text files processing, building of document-term matrix with stop words filtering and stemming. Same site offers comparison with other algorithms.
[edit] Commercial
[edit] See also
[edit] References and further reading
where S is the 
