Package {cumulcalib}

Title: Cumulative Calibration Assessment for Prediction Models
Version: 0.1.0
Description: Tools for visualization of, and inference on, the calibration of prediction models on the cumulative domain. This provides a method for evaluating calibration of risk prediction models without having to group the data or use tuning parameters (e.g., loess bandwidth). This package implements the methodology described in Sadatsafavi and Petkau (2024) <doi:10.1002/sim.10138>. The core of the package is cumulcalib(), which takes in vectors of binary responses and predicted risks. The package also implements non-parametric assessment of the calibration of individualized treatment effect (ITE) models using data from a randomized trial, via cumulcalibITE(), as described in Sadatsafavi et al. (2025) <doi:10.48550/arXiv.2512.08140>. The plot() and summary() methods are implemented for the results returned by cumulcalib() and cumulcalibITE().
License: MIT + file LICENSE
URL: https://github.com/resplab/cumulcalib
Imports: graphics, stats
Suggests: knitr, predtools, rmarkdown, markdown, spelling, testthat (≥ 3.0.0)
VignetteBuilder: knitr
Config/testthat/edition: 3
Encoding: UTF-8
Language: en-US
Config/roxygen2/version: 8.0.0
RoxygenNote: 7.3.3
NeedsCompilation: no
Packaged: 2026-07-24 22:41:19 UTC; msafavi
Author: Mohsen Sadatsafavi ORCID iD [aut, cre]
Maintainer: Mohsen Sadatsafavi <mohsen.sadatsafavi@ubc.ca>
Repository: CRAN
Date/Publication: 2026-07-25 17:10:08 UTC

cumulcalib: Cumulative Calibration Assessment for Prediction Models

Description

Tools for visualization of, and inference on, the calibration of prediction models on the cumulative domain. This provides a method for evaluating calibration of risk prediction models without having to group the data or use tuning parameters (e.g., loess bandwidth). This package implements the methodology described in Sadatsafavi and Petkau (2024) doi:10.1002/sim.10138. The core of the package is cumulcalib(), which takes in vectors of binary responses and predicted risks. The package also implements non-parametric assessment of the calibration of individualized treatment effect (ITE) models using data from a randomized trial, via cumulcalibITE(), as described in Sadatsafavi et al. (2025) doi:10.48550/arXiv.2512.08140. The plot() and summary() methods are implemented for the results returned by cumulcalib() and cumulcalibITE().

Author(s)

Maintainer: Mohsen Sadatsafavi mohsen.sadatsafavi@ubc.ca (ORCID)

See Also

Useful links:

Cumulative calibration assessment

Description

This is the core function for performing cumulative calibration assessment

Usage

cumulcalib(y, p, method = c("BB", "BM"), ordered = FALSE, n_sim = 0)

Arguments

y

vector of binary responses

p

vector of predicted probabilities.

method

string with either BB (Brownian bridge test, default method), BM (Brownian motion test), BM2p (two-part BM test - experimental), BB1p (one-part BB test wit only the 'bridge' component). Multiple methods can be specified. The first one will be the 'main' method (e.g., when submitting the resulting object to plot()). Default is c("BB","BM")

ordered

if TRUE, y and p are already ordered based on ascending values of p. This is to speed up simulations.

n_sim

if >0, indicates a simulation-based test is requested for inference.

Value

an objective of class cumulcalib that can be printed or plotted

Examples

pi <- rbeta(1000,1,2)
Y <- rbinom(length(pi),1,pi)
res <- cumulcalib(Y, pi, method="BB")
summary(res)
plot(res)

Cumulative calibration assessment for individual treatment effect (ITE) predictions

Description

This is the core function for performing cumulative calibration assessment for models that predict individual treatment effects (treatment benefit).

Usage

cumulcalibITE(
  y,
  h,
  a,
  p = NULL,
  method = c("BB", "BM"),
  ordered = FALSE,
  n_sim = 0,
  aux = FALSE
)

Arguments

y

vector of binary responses

h

vector of predicted treatment benefits (the predicted reduction in outcome risk due to treatment).

a

treatment indicator (1 if treated, 0 if control).

p

optional vector of predicted baseline risks (the risk without treatment). If omitted (NULL), the marginal test is performed using observed event rates in the treated and control groups; if supplied, the conditional test is performed. Default is NULL.

method

string with either BB (Brownian bridge test, default method), BM (Brownian motion test), BM2p (two-part BM test - experimental), BB1p (one-part BB test wit only the 'bridge' component). Multiple methods can be specified. The first one will be the 'main' method (e.g., when submitting the resulting object to plot()). Default is c("BB","BM")

ordered

if TRUE, the data are already ordered based on ascending values of h. This is to speed up simulations.

n_sim

if >0, indicates a simulation-based test is requested for inference.

aux

if TRUE, auxiliary quantities (used internally and for diagnostics) are returned in the result. Default is FALSE.

Value

an objective of class cumulcalib that can be printed or plotted

Examples

p <- rbeta(1000, 1, 2)
a <- rbinom(length(p), 1, 0.5)
h <- runif(length(p), 0, 0.05)
Y <- rbinom(length(p), 1, pmax(0, pmin(1, p - a * h)))
res <- cumulcalibITE(Y, h = h, a = a)
summary(res)
plot(res)

CDF of the Kolmogorov distribution

Description

CDF of the Kolmogorov distribution

Usage

pKolmogorov(q, summands = ceiling(q * sqrt(72) + 3/2))

Arguments

q

the quantity at which CDF will be evaluated. Currently accepts only a scalar

summands

maximum number of terms to be evaluated in the infinite series (default=ceiling(q*sqrt(72)+3/2))

Value

a scalar value

CDF of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval

Description

CDF of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval

Usage

pMAD_BM(q, summands = 100)

Arguments

q

the quantity at which CDF will be evaluated. Currently accepts only a scalar

summands

maximum number of terms to be evaluated in the infinite series (default=100)

Value

a scalar value

CDF of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval, conditional on its terminal value

Description

CDF of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval, conditional on its terminal value

Usage

pMAD_BM_c(
  q,
  w1,
  method = 1,
  exp_tolerance = -30,
  summands = ceiling(q * sqrt(72) + 3/2)
)

Arguments

q

the quantity at which CDF will be evaluated. Currently accepts only a scalar

w1

the terminal value

method

different infinite series to use (1,2,3)

exp_tolerance

numerical tolerance as the stopping rule when evaluating the infinite sum (default -30 on the exponential scale)

summands

number of terms to evaluate (default is ceiling(q * sqrt(72) + 3/2))

Value

a scalar value

Generates cumulative calibration plots

Description

Plot method for objects returned by cumulcalib() and cumulcalibITE().

Usage

## S3 method for class 'cumulcalib'
plot(
  x,
  method = NULL,
  stats_config = list(),
  x2axis = TRUE,
  y2axis = TRUE,
  draw_polygon = TRUE,
  ...
)

Arguments

x

An object of class cumulcalib generated by cumulcalib()

method

Which method to use. See details. If unspecified, uses the default method used in the cumulcalib() call.

stats_config

A list specifying the visualization parameters for the statistical test(s). Defaults to an empty list (list()), which by default draws only the statistic line(s) ("v", in blue and red), not their significance threshold(s) ("h", at the 0.95 level). If set to NULL, no statistical lines are drawn. Recognized elements are sig (significance level(s)), lines (which lines to draw: include "v" for the vertical statistic line and/or "h" for the horizontal significance threshold, e.g. "vh" to draw both), and col (color(s)).

x2axis

If true, draws a second x-axis (on top) showing predicted risks

y2axis

If true, draws a second y-axis (on right) showing scaled partial sums

draw_polygon

If true, draws the black triangle marker at the origin. Default is TRUE

...

Parameters to be passed to plot()

Details

For method, options are: BM1p (or simply BM): One-part Brownian motion BM2p: Two-part Brownian motion BB1p: One-part Brownian bridge BB2p (or simply BB): Two-part Brownian bridge

Value

None

Concise printing of a cumulcalib object

Description

Prints a one-screen overview of a calibration assessment returned by cumulcalib() or cumulcalibITE(). For a fuller breakdown use summary().

Usage

## S3 method for class 'cumulcalib'
print(x, ...)

Arguments

x

An object of class cumulcalib generated by cumulcalib() or cumulcalibITE()

...

Not used

Value

The input object, invisibly.

Print a cumulcalib summary

Description

Print a cumulcalib summary

Usage

## S3 method for class 'summary.cumulcalib'
print(x, ...)

Arguments

x

An object of class summary.cumulcalib generated by summary()

...

Not used

Value

The input object, invisibly.

Quantile function of the Kolmogorov distribution

Description

Quantile function of the Kolmogorov distribution

Usage

qKolmogorov(p)

Arguments

p

the quantity at which the quantile function will be evaluated. Currently accepts only a scalar

Value

a scalar value

Quantile function of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval

Description

Quantile function of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval

Usage

qMAD_BM(p)

Arguments

p

the quantity at which the quantile function will be evaluated. Currently accepts only a scalar

Value

a scalar value

Quantile function of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval, conditional on its terminal value

Description

Quantile function of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval, conditional on its terminal value

Usage

qMAD_BM_c(p, w1)

Arguments

p

the quantity at which the quantile function will be evaluated. Currently accepts only a scalar

w1

the terminal value

Value

a scalar value

Summarize a cumulcalib object

Description

Builds a structured summary of a calibration assessment for the selected method. The returned object has class summary.cumulcalib and is displayed by print.summary.cumulcalib().

Usage

## S3 method for class 'cumulcalib'
summary(object, method = NULL, shape_threshold = 1.5, ...)

Arguments

object

An object of class cumulcalib generated by cumulcalib() or cumulcalibITE()

method

Which method to use. Options are BB (Brownian bridge test), BM (Brownian motion test), BB1p (1-part Brownian bridge test), and BM2p (2-part Brownian bridge test). If unspecified, uses the default method used in the cumulcalib() call

shape_threshold

Minimum standardized maximum deviation (S*) for the shape of the miscalibration around C* to be described. Below this, the cumulative process is treated as unremarkable and no shape is reported (default 1.5).

...

Not used

Value

An object of class summary.cumulcalib: a list holding the key statistics for the selected method (mean calibration, distance/bridge statistic, p-values) together with the maximum cumulative calibration error C* and, since C* is the only quantity that admits a signed interpretation, its direction (C_star_sign, C_star_direction) and the location of that maximum (C_star_time, C_star_pred). The test statistics are reported unsigned. When the standardized maximum deviation (S*) is at least shape_threshold, a crossover element describes the shape of the miscalibration around the C* location: whether the cumulative error reverses (an interior peak, with opposite directions on either side) or is one-directional (a monotone accumulation), together with the direction(s). Otherwise crossover is NULL.