-
Joseph K. Bradley authored
## What changes were proposed in this pull request? Made DataFrame-based API primary * Spark doc menu bar and other places now link to ml-guide.html, not mllib-guide.html * mllib-guide.html keeps RDD-specific list of features, with a link at the top redirecting people to ml-guide.html * ml-guide.html includes a "maintenance mode" announcement about the RDD-based API * **Reviewers: please check this carefully** * (minor) Titles for DF API no longer include "- spark.ml" suffix. Titles for RDD API have "- RDD-based API" suffix * Moved migration guide to ml-guide from mllib-guide * Also moved past guides from mllib-migration-guides to ml-migration-guides, with a redirect link on mllib-migration-guides * **Reviewers**: I did not change any of the content of the migration guides. Reorganized DataFrame-based guide: * ml-guide.html mimics the old mllib-guide.html page in terms of content: overview, migration guide, etc. * Moved Pipeline description into ml-pipeline.html and moved tuning into ml-tuning.html * **Reviewers**: I did not change the content of these guides, except some intro text. * Sidebar remains the same, but with pipeline and tuning sections added Other: * ml-classification-regression.html: Moved text about linear methods to new section in page ## How was this patch tested? Generated docs locally Author: Joseph K. Bradley <joseph@databricks.com> Closes #14213 from jkbradley/ml-guide-2.0.
Joseph K. Bradley authored## What changes were proposed in this pull request? Made DataFrame-based API primary * Spark doc menu bar and other places now link to ml-guide.html, not mllib-guide.html * mllib-guide.html keeps RDD-specific list of features, with a link at the top redirecting people to ml-guide.html * ml-guide.html includes a "maintenance mode" announcement about the RDD-based API * **Reviewers: please check this carefully** * (minor) Titles for DF API no longer include "- spark.ml" suffix. Titles for RDD API have "- RDD-based API" suffix * Moved migration guide to ml-guide from mllib-guide * Also moved past guides from mllib-migration-guides to ml-migration-guides, with a redirect link on mllib-migration-guides * **Reviewers**: I did not change any of the content of the migration guides. Reorganized DataFrame-based guide: * ml-guide.html mimics the old mllib-guide.html page in terms of content: overview, migration guide, etc. * Moved Pipeline description into ml-pipeline.html and moved tuning into ml-tuning.html * **Reviewers**: I did not change the content of these guides, except some intro text. * Sidebar remains the same, but with pipeline and tuning sections added Other: * ml-classification-regression.html: Moved text about linear methods to new section in page ## How was this patch tested? Generated docs locally Author: Joseph K. Bradley <joseph@databricks.com> Closes #14213 from jkbradley/ml-guide-2.0.
layout: global
title: Clustering
displayTitle: ClusteringThis page describes clustering algorithms in MLlib. The guide for clustering in the RDD-based API also has relevant information about these algorithms.
Table of Contents
- This will become a table of contents (this text will be scraped). {:toc}
K-means
k-means is one of the most commonly used clustering algorithms that clusters the data points into a predefined number of clusters. The MLlib implementation includes a parallelized variant of the k-means++ method called kmeans||.
KMeans is implemented as an Estimator and generates a KMeansModel as the base model.
Input Columns
| Param name | Type(s) | Default | Description |
|---|---|---|---|
| featuresCol | Vector | "features" | Feature vector |
Output Columns
| Param name | Type(s) | Default | Description |
|---|---|---|---|
| predictionCol | Int | "prediction" | Predicted cluster center |
Example
{% include_example scala/org/apache/spark/examples/ml/KMeansExample.scala %}
{% include_example java/org/apache/spark/examples/ml/JavaKMeansExample.java %}
{% include_example python/ml/kmeans_example.py %}
Latent Dirichlet allocation (LDA)
LDA is implemented as an Estimator that supports both EMLDAOptimizer and OnlineLDAOptimizer,
and generates a LDAModel as the base model. Expert users may cast a LDAModel generated by
EMLDAOptimizer to a DistributedLDAModel if needed.
Refer to the Scala API docs for more details.
{% include_example scala/org/apache/spark/examples/ml/LDAExample.scala %}
Refer to the Java API docs for more details.
{% include_example java/org/apache/spark/examples/ml/JavaLDAExample.java %}
Refer to the Python API docs for more details.
{% include_example python/ml/lda_example.py %}
Bisecting k-means
Bisecting k-means is a kind of hierarchical clustering using a divisive (or "top-down") approach: all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy.
Bisecting K-means can often be much faster than regular K-means, but it will generally produce a different clustering.
BisectingKMeans is implemented as an Estimator and generates a BisectingKMeansModel as the base model.
Example
{% include_example scala/org/apache/spark/examples/ml/BisectingKMeansExample.scala %}
{% include_example java/org/apache/spark/examples/ml/JavaBisectingKMeansExample.java %}
{% include_example python/ml/bisecting_k_means_example.py %}
Gaussian Mixture Model (GMM)
A Gaussian Mixture Model
represents a composite distribution whereby points are drawn from one of k Gaussian sub-distributions,
each with its own probability. The spark.ml implementation uses the
expectation-maximization
algorithm to induce the maximum-likelihood model given a set of samples.
GaussianMixture is implemented as an Estimator and generates a GaussianMixtureModel as the base
model.
Input Columns
| Param name | Type(s) | Default | Description |
|---|---|---|---|
| featuresCol | Vector | "features" | Feature vector |
Output Columns
| Param name | Type(s) | Default | Description |
|---|---|---|---|
| predictionCol | Int | "prediction" | Predicted cluster center |
| probabilityCol | Vector | "probability" | Probability of each cluster |
Example
{% include_example scala/org/apache/spark/examples/ml/GaussianMixtureExample.scala %}
{% include_example java/org/apache/spark/examples/ml/JavaGaussianMixtureExample.java %}
{% include_example python/ml/gaussian_mixture_example.py %}