| Title: | Cumulative Calibration Assessment for Prediction Models |
|---|---|
| Description: | Tools for visualization of, and inference on, the calibration of prediction models on the cumulative domain. This provides a method for evaluating calibration of risk prediction models without having to group the data or use tuning parameters (e.g., loess bandwidth). This package implements the methodology described in Sadatsafavi and Petkau (2024) <doi:10.1002/sim.10138>. The core of the package is cumulcalib(), which takes in vectors of binary responses and predicted risks. The package also implements non-parametric assessment of the calibration of individualized treatment effect (ITE) models using data from a randomized trial, via cumulcalibITE(), as described in Sadatsafavi et al. (2025) <doi:10.48550/arXiv.2512.08140>. The plot() and summary() methods are implemented for the results returned by cumulcalib() and cumulcalibITE(). |
| Authors: | Mohsen Sadatsafavi [aut, cre] (ORCID: <https://orcid.org/0000-0002-0419-7862>) |
| Maintainer: | Mohsen Sadatsafavi <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.0 |
| Built: | 2026-07-24 22:46:07 UTC |
| Source: | https://github.com/resplab/cumulcalib |
This is the core function for performing cumulative calibration assessment
cumulcalib(y, p, method = c("BB", "BM"), ordered = FALSE, n_sim = 0)cumulcalib(y, p, method = c("BB", "BM"), ordered = FALSE, n_sim = 0)
y |
vector of binary responses |
p |
vector of predicted probabilities. |
method |
string with either BB (Brownian bridge test, default method), BM (Brownian motion test), BM2p (two-part BM test - experimental), BB1p (one-part BB test wit only the 'bridge' component). Multiple methods can be specified. The first one will be the 'main' method (e.g., when submitting the resulting object to plot()). Default is c("BB","BM") |
ordered |
if TRUE, y and p are already ordered based on ascending values of p. This is to speed up simulations. |
n_sim |
if >0, indicates a simulation-based test is requested for inference. |
an objective of class cumulcalib that can be printed or plotted
pi <- rbeta(1000,1,2) Y <- rbinom(length(pi),1,pi) res <- cumulcalib(Y, pi, method="BB") summary(res) plot(res)pi <- rbeta(1000,1,2) Y <- rbinom(length(pi),1,pi) res <- cumulcalib(Y, pi, method="BB") summary(res) plot(res)
This is the core function for performing cumulative calibration assessment for models that predict individual treatment effects (treatment benefit).
cumulcalibITE( y, h, a, p = NULL, method = c("BB", "BM"), ordered = FALSE, n_sim = 0, aux = FALSE )cumulcalibITE( y, h, a, p = NULL, method = c("BB", "BM"), ordered = FALSE, n_sim = 0, aux = FALSE )
y |
vector of binary responses |
h |
vector of predicted treatment benefits (the predicted reduction in outcome risk due to treatment). |
a |
treatment indicator (1 if treated, 0 if control). |
p |
optional vector of predicted baseline risks (the risk without treatment). If omitted (NULL), the marginal test is performed using observed event rates in the treated and control groups; if supplied, the conditional test is performed. Default is NULL. |
method |
string with either BB (Brownian bridge test, default method), BM (Brownian motion test), BM2p (two-part BM test - experimental), BB1p (one-part BB test wit only the 'bridge' component). Multiple methods can be specified. The first one will be the 'main' method (e.g., when submitting the resulting object to plot()). Default is c("BB","BM") |
ordered |
if TRUE, the data are already ordered based on ascending values of h. This is to speed up simulations. |
n_sim |
if >0, indicates a simulation-based test is requested for inference. |
aux |
if TRUE, auxiliary quantities (used internally and for diagnostics) are returned in the result. Default is FALSE. |
an objective of class cumulcalib that can be printed or plotted
p <- rbeta(1000, 1, 2) a <- rbinom(length(p), 1, 0.5) h <- runif(length(p), 0, 0.05) Y <- rbinom(length(p), 1, pmax(0, pmin(1, p - a * h))) res <- cumulcalibITE(Y, h = h, a = a) summary(res) plot(res)p <- rbeta(1000, 1, 2) a <- rbinom(length(p), 1, 0.5) h <- runif(length(p), 0, 0.05) Y <- rbinom(length(p), 1, pmax(0, pmin(1, p - a * h))) res <- cumulcalibITE(Y, h = h, a = a) summary(res) plot(res)
CDF of the Kolmogorov distribution
pKolmogorov(q, summands = ceiling(q * sqrt(72) + 3/2))pKolmogorov(q, summands = ceiling(q * sqrt(72) + 3/2))
q |
the quantity at which CDF will be evaluated. Currently accepts only a scalar |
summands |
maximum number of terms to be evaluated in the infinite series (default=ceiling(q*sqrt(72)+3/2)) |
a scalar value
Plot method for objects returned by cumulcalib() and cumulcalibITE().
## S3 method for class 'cumulcalib' plot( x, method = NULL, stats_config = list(), x2axis = TRUE, y2axis = TRUE, draw_polygon = TRUE, ... )## S3 method for class 'cumulcalib' plot( x, method = NULL, stats_config = list(), x2axis = TRUE, y2axis = TRUE, draw_polygon = TRUE, ... )
x |
An object of class cumulcalib generated by cumulcalib() |
method |
Which method to use. See details. If unspecified, uses the default method used in the cumulcalib() call. |
stats_config |
A list specifying the visualization parameters for the
statistical test(s). Defaults to an empty list ( |
x2axis |
If true, draws a second x-axis (on top) showing predicted risks |
y2axis |
If true, draws a second y-axis (on right) showing scaled partial sums |
draw_polygon |
If true, draws the black triangle marker at the origin. Default is TRUE |
... |
Parameters to be passed to plot() |
For method, options are: BM1p (or simply BM): One-part Brownian motion BM2p: Two-part Brownian motion BB1p: One-part Brownian bridge BB2p (or simply BB): Two-part Brownian bridge
None
CDF of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval
pMAD_BM(q, summands = 100)pMAD_BM(q, summands = 100)
q |
the quantity at which CDF will be evaluated. Currently accepts only a scalar |
summands |
maximum number of terms to be evaluated in the infinite series (default=100) |
a scalar value
CDF of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval, conditional on its terminal value
pMAD_BM_c( q, w1, method = 1, exp_tolerance = -30, summands = ceiling(q * sqrt(72) + 3/2) )pMAD_BM_c( q, w1, method = 1, exp_tolerance = -30, summands = ceiling(q * sqrt(72) + 3/2) )
q |
the quantity at which CDF will be evaluated. Currently accepts only a scalar |
w1 |
the terminal value |
method |
different infinite series to use (1,2,3) |
exp_tolerance |
numerical tolerance as the stopping rule when evaluating the infinite sum (default -30 on the exponential scale) |
summands |
number of terms to evaluate (default is ceiling(q * sqrt(72) + 3/2)) |
a scalar value
Prints a one-screen overview of a calibration assessment returned by
cumulcalib() or cumulcalibITE(). For a fuller breakdown use summary().
## S3 method for class 'cumulcalib' print(x, ...)## S3 method for class 'cumulcalib' print(x, ...)
x |
An object of class cumulcalib generated by cumulcalib() or cumulcalibITE() |
... |
Not used |
The input object, invisibly.
Print a cumulcalib summary
## S3 method for class 'summary.cumulcalib' print(x, ...)## S3 method for class 'summary.cumulcalib' print(x, ...)
x |
An object of class summary.cumulcalib generated by summary() |
... |
Not used |
The input object, invisibly.
Quantile function of the Kolmogorov distribution
qKolmogorov(p)qKolmogorov(p)
p |
the quantity at which the quantile function will be evaluated. Currently accepts only a scalar |
a scalar value
Quantile function of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval
qMAD_BM(p)qMAD_BM(p)
p |
the quantity at which the quantile function will be evaluated. Currently accepts only a scalar |
a scalar value
Quantile function of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval, conditional on its terminal value
qMAD_BM_c(p, w1)qMAD_BM_c(p, w1)
p |
the quantity at which the quantile function will be evaluated. Currently accepts only a scalar |
w1 |
the terminal value |
a scalar value
Builds a structured summary of a calibration assessment for the selected
method. The returned object has class summary.cumulcalib and is
displayed by print.summary.cumulcalib().
## S3 method for class 'cumulcalib' summary(object, method = NULL, shape_threshold = 1.5, ...)## S3 method for class 'cumulcalib' summary(object, method = NULL, shape_threshold = 1.5, ...)
object |
An object of class cumulcalib generated by cumulcalib() or cumulcalibITE() |
method |
Which method to use. Options are BB (Brownian bridge test), BM (Brownian motion test), BB1p (1-part Brownian bridge test), and BM2p (2-part Brownian bridge test). If unspecified, uses the default method used in the cumulcalib() call |
shape_threshold |
Minimum standardized maximum deviation (S*) for the shape of the miscalibration around C* to be described. Below this, the cumulative process is treated as unremarkable and no shape is reported (default 1.5). |
... |
Not used |
An object of class summary.cumulcalib: a list holding the key
statistics for the selected method (mean calibration, distance/bridge
statistic, p-values) together with the maximum cumulative calibration
error C* and, since C* is the only quantity that admits a signed
interpretation, its direction (C_star_sign,
C_star_direction) and the location of that maximum
(C_star_time, C_star_pred). The test statistics are reported
unsigned. When the standardized maximum deviation (S*) is at least
shape_threshold, a crossover element describes the shape of
the miscalibration around the C* location: whether the cumulative error
reverses (an interior peak, with opposite directions on either side) or is
one-directional (a monotone accumulation), together with the direction(s).
Otherwise crossover is NULL.