Clustering statistics box-plot.

Description

Clustering statistics box-plot.

Usage

bdm.boxp(bdm, byVars = F, layer = 1)

Arguments

Details

If the number of clusters is large, only the first 25 clusters will be plotted. Note that the WTT algorithm numbers the clusters based on density value at the peak cell of the cluster. Thus, the numbering of the clusters is highly correlated with their relevance in terms of partial density. Therefore, in case of more than 25 clusters, the most relevant should always be included in the plot.

Value

None.

Examples

bdm.example()
bdm.boxp(exMap)
bdm.boxp(exMap, byVars = TRUE)

ptSNE cost & size plot.

Description

ptSNE cost & size plot.

Usage

bdm.cost(bdm, offset = 0)

Arguments

Value

None.

Examples

bdm.example()
bdm.cost(exMap)

Class density maps

Description

Compute the class density maps of a set of classes on the embedding grid. This function returns a fuzzy mapping of the set of classes on the grid cells. The classes can be whatever set of classes of interest and must be given as a vector of point-wise discrete labels (either numeric, string or factor).

Usage

bdm.dMap(bdm, threads = 2, type = "SOCK", data = NULL, layer = 1)

Arguments

Details

bdm.dMap() computes the join distribution P(V=v_{i},C=c_{j}) where V={v_{1},\dots,v_{l}} is the discrete covariate and C={c_{1},\dots, c_{g}} are the grid cells of the paKDE raster. That is, this function recomputes the paKDE but keeping track of the covariate (or class) label of each data-point. This results in a fuzzy distribution of the covariate (class) at each cell.

Usually, figuring out the join distribution P(V=v_{i},C=c_{j}) entails an intensive computation. Thus bdm.dMap() performs the computation and stores the result in a dedicated element named $dMap. Afterwards the class density maps can be visualized with the bdm.dMap.plot() function.

Value

A copy of the input bdm instance with element $dMap, a matrix with a soft clustering of the grid cells.

Examples

# --- load example dataset
bdm.example()
## Not run: 
exMap <- bdm.dMap(exMap, threads = 4)

## End(Not run)

Class density maps plot.

Description

Class density maps plot.

Usage

bdm.dMap.plot(bdm, classes = NULL, join = FALSE, class.pltt = NULL,
  pakde.pltt = NULL, pakde.lvls = 16, wtt.lwd = 1, plot.peaks = T,
  labels.cex = 1, layer = 1)

Arguments

Details

bdm.dMap.plot() yields a multi-plot layout where the first plot shows the dominating value of the covariate (or dominating class) in each cell, and the rest of the plots show the density map of each covariate value (or class).

The join distribution P(V=v_{i},C=c_{j}) will be affected by the bias present in the marginal distribution of the covariate. Therefore, the join distribution P(V=v_{i},C=c_{j}) is transformed, by default, into a conditional distribution P(c_{j}|V=v_{i}) (where the c_{j} are the grid cells of the embedding and V is the covariate (or class)). Thus, the first plot shows a hard classification of grid-cells, (cells are coloured based on the dominating value of the covariate (or dominating class), i.e. the v_{i} for which P(c_{j}|V=v_{i}) is maximum), and the rest of the plots show the conditional distributions P(C=c_{j}|V=v_{i}). This makes the plots of the different classes not directly comparable but the dominant areas of each class can be more easily identified.

However, the same plots can be depicted based on the join distribution by setting join = TRUE. This makes sense when the bias in the covariate values (or classes) is not significant. In this case the hard clustering shows the real dominance of each covariate value (or class) over the embedding area and the density maps are comparable one to each other (although, individually, they are not real density functions as they do not add up to one).

The multi-plot layout can be limited to a subset of the values of the covariate (or subset of classes) specified in parameter classes.

Value

None.

Examples

# --- load example dataset
bdm.example()
## Not run: 
exMap <- bdm.dMap(exMap, threads = 4)
bdm.dMap.plot(exMap)

## End(Not run)

Example dataset

Description

Loads an example of a mapping of a dataset.

Usage

bdm.example()

Details

A bdm instance is a list with elements: $dSet a name identifying the dataset (bdm.fName() use this name to generate a default file name); $data a matrix with raw data; $lbls a vector of datapoint labels (in case they are known); $N the dataset size; $is.distance a logical value that is set to TRUE when the raw data is a distance matrix. Downstream steps of the mapping protocol will add more elements to the list.

This example is based on a small synthetic dataset with n = 5000 observations drawn from a 4-variate Gaussian Mixture Model (GMM) with 16 Gaussian components.

Value

An example bdm instance named exMap.

Examples

# --- load example dataset
bdm.example()
str(exMap)

Default bdm file name

Description

Generates a default file name. The default file name is intended for functions bdm.save() and bdm.scp() to ease the task of working/organizing multiple runs on the same dataset.

Usage

bdm.fName(bdm)

Arguments

Details

The file name is generated based on bdm$dSet and main ptSNE parameters (threads, layers, rounds, boost and perplexity). In case that bdm.wtt() has been performed on any of the layers, the number of clusters in the first not null layer of bdm$wtt is also included.

Value

A *.RData file name based on bdm$dSet and main bdm parameters.

Examples

bdm.example()
str(exMap$dSet)
str(exMap$ptsne)
bdm.fName(exMap)

Create bdm instance

Description

Creates a bdm instance.

Usage

bdm.init(dSet.name, dSet.data, labels = NULL, is.distance = F,
  check.duplicates = T)

Arguments

Value

A bdm instance. A bdm instance is initially a list with a few elements to which new elements are added at each step of the mapping protocol.

Examples

# --- get a matrix with raw-data
mydata <- matrix(rnorm(10000, mean = 0, sd = 3),  ncol = 2)
mylabels <- apply(mydata, 1, function(row) round(sqrt(sum(row**2)), 0))
# --- create a \var{bdm} instance with our raw-data matrix
mybdm <- bdm.init('mydataset', mydata, labels = mylabels)
str(mybdm)

Get data-point clustering labels.

Description

Given that clusters are computed at grid-cell level, this function returns the clustering label for each data-point.

Usage

bdm.labels(bdm, merged = T, layer = 1)

Arguments

Value

A vector of data-point clustering labels.

Examples

bdm.example()
exMap.labels <- bdm.labels(exMap)

Set/get default local machine name or IP address

Description

Set/get default local machine name or IP address

Usage

bdm.local(dest = NULL)

Arguments

Value

The current value of local

Examples

# --- set default value of \var{local}
bdm.local('xxx.255.0.0')
bdm.local('mymachine.mydomain.cat')

Merging of clusters based on signal-to-noise-ratio.

Description

Performs a recursive merging of clusters based on minimum loss of signal-to-noise-ratio (S2NR) until reaching the desired number of clusters. The S2NR is the explained/unexplained variance ratio measured in the high dimensional space based on the given low dimensional clustering.

Usage

bdm.merge.s2nr(bdm, k = 10, plot.merge = T, ret.merge = F,
  info = T, layer = 1, ...)

Arguments

Details

See details in bdm.optk.s2nr().

Value

None if ret.merge = FALSE. Else, a copy of the input bdm instance with new element bdm$merge.

Examples

bdm.example()
exMap.labels <- bdm.labels(exMap)

Set/get default path for mybdm

Description

Set/get default path for mybdm

Usage

bdm.mybdm(path = NULL)

Arguments

Value

The current path value to mybdm

Examples

# --- set default path for \var{mybdm}
bdm.mybdm('~/mybdm')

Plots the signal-to-nois-ratio as a function of the number of clusters.

Description

The function bdm.optk.sn2r() computes the S2NR that results from recursively merging clusters and, by deafult, makes a plot of these values. For large datasets this computation can take a while, so we can save this result by setting ret.optk = TRUE. If this result is saved, we can plot it again at any time using this funcion.

Usage

bdm.optk.plot(bdm)

Arguments

Value

None.

Examples

bdm.example()
exMap <- bdm.optk.s2nr(exMap, ret.optk = TRUE)
bdm.optk.plot(exMap)

Find optimal number of clusters based on signal-to-noise-ratio.

Description

Performs a recursive merging of clusters based on minimum loss of signal-to-noise-ratio (S2NR). The S2NR is the explained/unexplained variance ratio measured in the high dimensional space based on the given low dimensional clustering. Merging is applied recursively until reaching a configuration of only 2 clusters and the S2NR is measured at each step.

Usage

bdm.optk.s2nr(bdm, info = T, plot.optk = T, ret.optk = F,
  layer = 1)

Arguments

Details

The logic under this heuristic is that neigbouring clusters in the embedding correspond to close clusters in the high dimensional space, i.e. it is a merging heuristic based on the spatial distribution of clusters. For each cluster (child cluster) we choose the neighboring cluster with steepest gradient along their common border (father cluster). Thus, we get a set of pairs of clusters (child/father) as potential mergings. Given this set of candidates, the merging is performed recursively choosing, at each step, the pair of child/father clusters that results in a minimum loss of S2NR. A typical situation is that some clusters dominate over all of their neighboring clusters. This clusters have no father. Thus, once all candidate mergings have been performed we reach a blocked state where only the dominant clusters remain. This situation identifies a hierarchy level in the clustering. When this situation is reached, the algorithm starts a new merging round, identifying the child/father relations at that level of hierarchy. The process stops when only two clusters remain. Usually, the clustering hierarchy is clearly depicted by singular points in the S2NR function. This is a hint that the low dimensional clustering configuration is an image of a hierarchycal spatial configuration in the high dimensional space. See bdm.optk.plot().

Value

None if ret.optk = FALSE. Else, a copy of the input bdm instance with new element bdm$optk (a matrix).

Examples

# --- load mapped dataset
bdm.example()
# --- compute optimal number of clusters and attach the computation
bdm.optk.s2nr(exMap, plot.optk = TRUE, ret.optk = FALSE)

Perplexity-adaptive kernel density estimation

Description

Starts the paKDE algorithm (second step of the mapping protocol).

Usage

bdm.pakde(bdm, layer = 1, threads = 2, type = "SOCK", ppx = 100,
  itr = 100, tol = 1e-05, g = 200, g.exp = 3)

Arguments

Details

When computing the paKDE the embedding area is discretized as a grid of size g*g cells. In order to avoid border effects, the limits of the grid are expanded by default so as to enclose at least the 0.9986 of the cumulative distribution function (3 \sigma) of the kernels of the most extreme mapped points in each direction.

The presence of outliers in the embedding can lead to undesired expansion of the grid limits. We can overcome this using lower values of g.exp. By setting g.exp = 0 the grid limits will be equal to the range of the embedding.

The values g.exp = c(1, 2, 3, 4, 5, 6) enclose cdf values of 0.8413, 0.9772, 0.9986, 0.99996, 0.99999, 1.0 respectively.

Value

A copy of the input bdm instance with new element bdm$pakde (paKDE output). bdm$pakde[[layer]]$layer = 'NC' stands for not computed layers.

Examples

# --- load mapped dataset
bdm.example()
# --- run paKDE
## Not run: 
exMap <- bdm.pakde(exMap, threads = 4, ppx = 200, g = 200, g.exp = 3)

## End(Not run)
# --- plot paKDE output
bdm.pakde.plot(exMap)

Plot paKDE (density landscape)

Description

Plot paKDE (density landscape)

Usage

bdm.pakde.plot(bdm, pakde.pltt = NULL, pakde.lvls = 16, layer = 1)

Arguments

Value

None.

Examples

bdm.example()
exMap <- bdm.pakde.plot(exMap)

Parallelized t-SNE

Description

Starts the ptSNE algorithm (first step of the mapping protocol).

Usage

bdm.ptsne(bdm, threads = 3, type = "SOCK", layers = 2, rounds = 1,
  boost = 2, whiten = 4, input.dim = NULL, ppx = 100, itr = 100,
  tol = 1e-05, alpha = 0.5, Y.init = NULL, info = 1)

Arguments

Details

By default the algorithm is structured in \sqrt{n} epochs of \sqrt{z} iterations each, where n is the dataset size and z is the thread-size (z=n*layers/threads). The running time of the algorithm is then determined by epochs*iters*t_i+ epochs*t_e where t_i is the running time of a single iteration and t_e is the inter-epoch running time.

The boost factor is meant to reduce the running time. With boost > 1 the algorithm is structured in n/boost epochs with z*boost iterations each. This structure performs the same total number of iterations but arranged into a lower number of epochs, thus decreasing the total running time to epochs*iters*t_i + 1/boost*epochs*t_e. When the number of threads is high, the inter-epoch time can be high, in particular when using 'MPI' parallelization, thus, reducing the number of epochs can result in a significant reduction of the total running time. The counterpart is that increasing the number of iterations per epoch might result in a lack of convergence, thus the boost factor must be used with caution. To the most of our knowledge using values up to boost=2.5 is generally safe.

In case of extremely large datasets, we strongly recommend to initialize the bdm instance with already preprocessed data and use whiten = 0. Fast principal components approximations can be computed by means of e.g. flashpcaR or scater R packages.

Value

A copy of the input bdm instance with new element bdm$ptsne (t-SNE output).

Examples

# --- load example dataset
bdm.example()
# --- perform ptSNE
## Not run: 
exMap <- bdm.ptsne(exMap, threads = 10, layers = 2, rounds = 2, ppx = 200)

## End(Not run)
# --- plot the Cost function
bdm.cost(exMap)
# --- plot ptSNE output
bdm.ptsne.plot(exMap)

Plot ptSNE (low-dimensional embedding)

Description

Plot ptSNE (low-dimensional embedding)

Usage

bdm.ptsne.plot(bdm, ptsne.cex = 0.5, ptsne.bg = "#FFFFFF",
  class.pltt = NULL, layer = 1)

Arguments

Value

None.

Examples

bdm.example()
exMap <- bdm.ptsne.plot(exMap)

ptSNE quantile-maps

Description

Shows the mapping of quantitative variables into the embedding space.

Usage

bdm.qMap(bdm, data = NULL, labels = NULL, subset = NULL,
  qMap.levels = 8, qMap.cex = 0.3, qMap.bg = "#FFFFFF",
  class.pltt = NULL, layer = 1)

Arguments

Details

This is not a heat-map but a quantile-map plot. This function splits the range of each variable into as many quantiles as specified by levels so that the color gradient will hardly ever correspond to a constant numeric gradient. Thus, the mapping will show more evenly distributed colors though at the expense of possibly exaggerating artifacts. For variables with very extrem distributions, it will be impossible to find as many quantiles as desired and the distribution of colors will not be so homogeneous.

Value

None.

Examples

bdm.example()
bdm.qMap(exMap)
# --- show only components (1, 2, 4, 8) of the GMM
bdm.qMap(exMap, subset = which(exMap$lbls %in% c(1, 4, 8, 16)))

Save bdm instance

Description

Saves a bdm instance with default path/file names, as given by bdm.mybdm()/bdm.fName(bdm). Default file name is generated based on bdm$dSet and ptSNE main parameters (threads, layers, boost, rounds, perplexity). The purpose of functions bdm.save() and bdm.scp() used with bdm.fName() is to ease the task of working/organizing multiple runs on the same dataset.

Usage

bdm.save(...)

Arguments

Value

None

Examples

# --- get a matrix with raw-data
mydata <- cbind(rnorm(10000, mean = 0, sd = 3),  ncol = 2)
mylabels <- apply(mydata, 1, function(row) round(sqrt(sum(row**2)), 0))
# --- create a \var{bdm} instance with our raw-data matrix
mybdm <- bdm.init('mydataset', mydata, labels = mylabels)
str(mybdm)
# --- save it
## Not run: 
bdm.save(mybdm)

## End(Not run)

Transfer bdm instance to a remote machine.

Description

Transfers a bdm instance to a remote machine. By default a file name is generated based on bdm$dSet and t-SNE main parameters (threads, layers, rounds, perplexity). The purpose of functions bdm.save() and bdm.scp() used with bdm.fName() is to ease the task of working/organizing multiple runs on the same dataset.

Usage

bdm.scp(..., dest = NULL)

Arguments

Value

None

Examples

## Not run: 
# --- load example
bdm.example()
# --- scp to \var{bdm.local()} with default file name
bdm.scp(exMap)
# --- scp to IP address 'xxx.xxx.0.0' with default file name
bdm.scp(exMap, dest = 'xxx.xxx.0.0')

## End(Not run)

Watertrack transform (WTT)

Description

Starts the WTT algorithm (third setp of the mapping protocol).

Usage

bdm.wtt(bdm, layer = 1)

Arguments

Details

This function requires the up-stream step bdm.pakde().

Value

A copy of the input bdm instance with bdm$wtt (WTT output). bdm$wtt[[layer]]$layer = 'NC' stands for not computed layers.

Examples

# --- load mapped dataset
bdm.example()
# --- perform WTT
exMap <- bdm.wtt(exMap)
# --- plot WTT output
bdm.wtt.plot(exMap)

Plot WTT (clustering)

Description

Plot WTT (clustering)

Usage

bdm.wtt.plot(bdm, pakde.pltt = NULL, pakde.lvls = 16, wtt.lwd = 1,
  plot.peaks = T, labels.cex = 1, layer = 1)

Arguments

Value

None.

Examples

bdm.example()
exMap <- bdm.wtt.plot(exMap)