org.apache.spark.ml.util

Class MetadataUtils

Object

org.apache.spark.ml.util.MetadataUtils

public class MetadataUtils
extends Object

Helper utilities for algorithms using ML metadata

Constructor Summary

Constructors

Constructor and Description
MetadataUtils()

Method Summary

Modifier and Type	Method and Description
static scala.collection.immutable.Map<Object,Object>	getCategoricalFeatures(StructField featuresSchema) Examine a schema to identify categorical (Binary and Nominal) features.
static int[]	getFeatureIndicesFromNames(StructField col, String[] names) Takes a Vector column and a list of feature names, and returns the corresponding list of feature indices in the column, in order.
static scala.Option<Object>	getNumClasses(StructField labelSchema) Examine a schema to identify the number of classes in a label column.
static int	getNumFeatures(Dataset<?> dataset, String vectorCol) Obtain the number of features in a vector column.
static scala.Option<Object>	getNumFeatures(StructField vectorSchema) Examine a schema to identify the number of features in a vector column.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MetadataUtils

public MetadataUtils()

Method Detail

getNumClasses

public static scala.Option<Object> getNumClasses(StructField labelSchema)

Examine a schema to identify the number of classes in a label column. Returns None if the number of labels is not specified, or if the label column is continuous.

Parameters:

labelSchema - (undocumented)

Returns:

(undocumented)

getNumFeatures

public static int getNumFeatures(Dataset<?> dataset,
                                 String vectorCol)

Obtain the number of features in a vector column. If no metadata is available, extract it from the dataset.

Parameters:

dataset - (undocumented)

vectorCol - (undocumented)

Returns:

(undocumented)

getNumFeatures

public static scala.Option<Object> getNumFeatures(StructField vectorSchema)

Examine a schema to identify the number of features in a vector column. Returns None if the number of features is not specified.

Parameters:

vectorSchema - (undocumented)

Returns:

(undocumented)

getCategoricalFeatures

public static scala.collection.immutable.Map<Object,Object> getCategoricalFeatures(StructField featuresSchema)

Examine a schema to identify categorical (Binary and Nominal) features.

Parameters:

featuresSchema - Schema of the features column. If a feature does not have metadata, it is assumed to be continuous. If a feature is Nominal, then it must have the number of values specified.

Returns:

Map: feature index to number of categories. The map's set of keys will be the set of categorical feature indices.

getFeatureIndicesFromNames

public static int[] getFeatureIndicesFromNames(StructField col,
                                               String[] names)

Takes a Vector column and a list of feature names, and returns the corresponding list of feature indices in the column, in order.

Parameters:

col - Vector column which must have feature names specified via attributes

names - List of feature names

Returns:

(undocumented)