| Type: | Package |
| Title: | Quantifying Performance of a Binary Classifier Through Weight of Evidence |
| Version: | 0.6.2 |
| Date: | 2019-09-12 |
| Description: | The distributions of the weight of evidence (log Bayes factor) favouring case over noncase status in a test dataset (or test folds generated by cross-validation) can be used to quantify the performance of a diagnostic test (McKeigue (2019), <doi:10.1177/0962280218776989>). The package can be used with any test dataset on which you have observed case-control status and have computed prior and posterior probabilities of case status using a model learned on a training dataset. To quantify how the predictor will behave as a risk stratifier, the quantiles of the distributions of weight of evidence in cases and controls can be calculated and plotted. |
| Depends: | R (≥ 2.10) |
| License: | GPL-3 |
| URL: | http://www.homepages.ed.ac.uk/pmckeigu/preprints/classify/wevidtutorial.html |
| LazyLoad: | yes |
| Imports: | ggplot2, mclust, pROC (≥ 1.9), reshape2, zoo |
| Suggests: | testthat (≥ 2.0.0) |
| ByteCompile: | TRUE |
| Encoding: | UTF-8 |
| RoxygenNote: | 6.1.1 |
| NeedsCompilation: | no |
| Packaged: | 2019-09-12 15:18:09 UTC; mcolombo |
| Author: | Paul McKeigue 
[aut],
Marco Colombo
[ctb, cre] |
| Maintainer: | Marco Colombo <mar.colombo13@gmail.com> |
| Repository: | CRAN |
| Date/Publication: | 2019-09-12 15:40:02 UTC |
Quantifying performance of a diagnostic test using the sampling distribution of the weight of evidence favouring case over noncase status
Description
The wevid package provides functions for quantifying the performance of a diagnostic test (or any other binary classifier) by calculating and plotting the distributions in cases and noncases of the weight of evidence favouring case over noncase status.
Details
The distributions of the weight of evidence (log Bayes factor) favouring case over noncase status in a test dataset (or test folds generated by cross-validation) can be used to quantify the performance of a diagnostic test.
In comparison with the C-statistic (area under ROC curve), the expected weight of evidence (expected information for discrimination) has several advantages as a summary measure of predictive performance. To quantify how the predictor will behave as a risk stratifier, the quantiles of the distributions of weight of evidence in cases and controls can be calculated and plotted.
This package can be used with any test dataset on which you have observed case-control status and have computed prior and posterior probabilities of case status using a model learned on a training dataset. Therefore, you should have computed on a test dataset (or on test folds used for cross-validation):
The prior probability of case status (this may be just the frequency of cases in the training data).
The posterior probability of case status (using the model learned on the training data to predict on the test data).
The observed case status (coded as 0 for noncases, 1 for cases).
The main function of the package is Wdensities which computes
the crude and model-based densities of weight of evidence in cases and
controls. Once these are computed, they can be plotted with
plotWdists and plotcumfreqs. Summary statistics
can be reported with summary.
Author(s)
Paul McKeigue paul.mckeigue@ed.ac.uk
References
Paul McKeigue (2019), Quantifying performance of a diagnostic test as the expected information for discrimination: Relation to the C-statistic. Statistical Methods for Medical Research, 28 (6), 1841-1851. https://doi.org/10.1177/0962280218776989.
See Also
Useful links:
Compute densities of weights of evidence in cases and controls
Description
The function computes smoothed densities of the weight of evidence in cases and in controls from the crude probabilities, then adjusts them to make them mathematically consistent so that p(W_ctrl) = exp(-W) p(W_case).
Usage
Wdensities(y, posterior.p, prior.p, range.xseq = c(-25, 25),
x.stepsize = 0.01, adjust.bw = 1, recalibrate = TRUE,
debug = FALSE)
Arguments
y |
Binary outcome label (0 for controls, 1 for cases). |
posterior.p |
Vector of posterior probabilities generated by using model to predict on test data. |
prior.p |
Vector of prior probabilities. |
range.xseq |
Range of points where the curves should be sampled. |
x.stepsize |
Distance between each point. |
adjust.bw |
Bandwidth adjustment for the Gaussian kernel density estimator. By default it is set to 1 (no adjustment), setting it to a value smaller/larger than 1 reduces/increases the smoothing of the kernel. This argument is ignored if more than one mixture component is identified. |
recalibrate |
If |
debug |
If |
Details
If the sample distributions in cases and controls support a 2-component mixture model (based on model comparison with BIC) for the densities, this will be detected and a 2-component mixture model will be fitted before adjustment.
Value
A densities object that contains the information necessary to compute summary measures and generate plots.
Examples
data(cleveland)
densities <- with(cleveland, Wdensities(y, posterior.p, prior.p))
# Example which requires fitting a mixture distribution
data(fitonly)
densities <- with(fitonly, Wdensities(y, posterior.p, prior.p))
Calculate the crude smoothed densities of W in cases and in controls
Description
These functions allow to compute the smoothed densities of the weight of
evidence in cases and in controls from the crude probabilities.
Wdensities.crude is designed for the general case; if the model
probabilities reflect a spike-slab mixture distribution, where a high
proportion of values of the predictor are zero, then Wdensities.mix
will be more appropriate.
Usage
Wdensities.crude(y, W, xseq, adjust.bw = 1)
Wdensities.mix(y, W, xseq, mixcomponent)
Arguments
y |
Binary outcome label (0 for controls, 1 for cases). |
W |
Weight of evidence. |
xseq |
Sequence of points where the curves should be sampled. |
adjust.bw |
Bandwidth adjustment. |
mixcomponent |
integer vector same length as |
Value
List of crude densities for controls and cases sampled at each point in the range provided.
Plot the distribution of the weight of evidence in cases and in controls
Description
Plot the distribution of the weight of evidence in cases and in controls
Usage
plotWdists(densities, distlabels = c("Crude", "Model-based"))
Arguments
densities |
Densities object produced by |
distlabels |
Character vector of length 2 to be used to label the crude and the model-based curves (in that order). |
Value
A ggplot object representing the distributions of crude and model-based weights of evidence in cases and in controls.
Examples
data(cleveland)
densities <- with(cleveland, Wdensities(y, posterior.p, prior.p))
plotWdists(densities)
# Example which requires fitting a mixture distribution
data(fitonly)
densities <- with(fitonly, Wdensities(y, posterior.p, prior.p))
# truncate spike
plotWdists(densities) + ggplot2::scale_y_continuous(limits=c(0, 0.5))
Plot the cumulative frequency distributions in cases and in controls
Description
Plot the cumulative frequency distributions in cases and in controls
Usage
plotcumfreqs(densities)
Arguments
densities |
Densities object produced by |
Value
A ggplot object representing the cumulative frequency distributions of the smoothed densities of the weights of evidence in cases and in controls.
Examples
data(cleveland)
densities <- with(cleveland, Wdensities(y, posterior.p, prior.p))
plotcumfreqs(densities)
Plot crude and model-based ROC curves
Description
While the crude ROC curve can be non-concave and is generally not smooth, the model-based ROC curve is always concave, as the corresponding densities have been adjusted to be mathematically consistent.
Usage
plotroc(densities)
Arguments
densities |
Densities object produced by |
Value
A ggplot object representing crude and model-based ROC curves.
Examples
data(cleveland)
densities <- with(cleveland, Wdensities(y, posterior.p, prior.p))
plotroc(densities)
Proportions of cases and controls below a threshold of weight of evidence
Description
Proportions of cases and controls below a threshold of weight of evidence
Usage
prop.belowthreshold(densities, w.threshold)
Arguments
densities |
Densities object produced by |
w.threshold |
Threshold value of weight of evidence (natural logs). |
Value
Numeric vector of length 2 listing the proportions of controls and cases with weight of evidence below the given threshold.
Examples
data(cleveland)
densities <- with(cleveland, Wdensities(y, posterior.p, prior.p))
w.threshold <- log(4) # threshold Bayes factor of 4
prop.belowthreshold(densities, w.threshold)
Recalibrate posterior probabilities
Description
Transforms posterior probabilities to logits, fits a logistic regression model and returns the predictive probabilities from this model.
Usage
recalibrate.p(y, posterior.p)
Arguments
y |
Binary outcome label (0 for controls, 1 for cases). |
posterior.p |
Vector of posterior probabilities. |
Value
Recalibrated posterior probabilities.
Summary evaluation of predictive performance
Description
Summary evaluation of predictive performance
Usage
## S3 method for class 'Wdensities'
summary(object, ...)
## S3 method for class 'Wdensities'
mean(x, ...)
auroc.crude(densities)
auroc.model(densities)
lambda.crude(densities)
lambda.model(densities)
Arguments
object, x, densities |
Densities object produced by
|
... |
Further arguments passed to or from other methods. These are currently ignored. |
Value
summary returns a data frame that reports the number of cases and
controls, the test log-likelihood, the crude and model-based C-statistic
and expected weight of evidence Lambda.
mean returns a numeric vector listing the mean densities of the weight
of evidence in controls and in cases.
auroc.crude and auroc.model return the area under the ROC curve
according to the crude and the model-based densities of weight of evidence,
respectively.
lambda.crude and lambda.model return the expected weight of
evidence (expected information for discrimination) in bits from the crude
and the model-based densities, respectively.
Examples
data(cleveland)
densities <- with(cleveland, Wdensities(y, posterior.p, prior.p))
summary(densities)
mean(densities)
auroc.model(densities)
lambda.model(densities)
Calculate weights of evidence in natural log units
Description
Calculate weights of evidence in natural log units
Usage
weightsofevidence(posterior.p, prior.p)
Arguments
posterior.p |
Vector of posterior probabilities generated by using model to predict on test data. |
prior.p |
Vector of prior probabilities. |
Value
The weight of evidence in nats for each observation.
Examples
data(cleveland) # load example dataset
W <- with(cleveland, weightsofevidence(posterior.p, prior.p))
Example datasets
Description
Example datasets
The wevid package comes with the following dataset:
-
clevelandis based on cross-validated prediction of coronary disease in the Cleveland Heart Study (297 observations).
-
pimais based on cross-validated prediction of diabetes in Pima Native Americans (768 observations).
-
fitonlyis based on cross-validated prediction of colorectal cancer from fecal immunochemical test (FIT) only in Michigan (242 observations). As most controls and some cases have have zero values in the FIT test, to fit densities to the sampled values of weight of evidence in controls and cases it is necessary to specify spike-slab mixtures.
Format
Each dataset consists of a data frame with the following variables:
prior.p: Prior probabilities of case status.
posterior.p: Posterior probabilities of case status.
y: Case-control status.
Source
http://www.homepages.ed.ac.uk/pmckeigu/preprints/classify/wevidtutorial.html