Package {modelcardr}

Type: Package
Title: Machine Learning Prediction Auditing and Model Card Reporting
Version: 0.1.0
Description: Audits predictions from fitted machine learning models without requiring retraining or access to the fitted model object. It supports binary classification, multiclass classification, and regression, providing performance metrics with bootstrap confidence intervals, calibration and threshold analyses, residual diagnostics, subgroup comparisons, user-defined acceptance criteria, risk warnings, and self-contained reports. For binary-classification and regression models, the package can also assemble and render structured model cards documenting intended use, evaluation data, performance, subgroup results, assumptions, and limitations. The model-card reporting approach is described by Mitchell et al. (2019) "Model Cards for Model Reporting" <doi:10.1145/3287560.3287596>.
License: MIT + file LICENSE
Encoding: UTF-8
Depends: R (≥ 4.1.0)
Imports: graphics, stats, tools, utils, withr
Suggests: testthat (≥ 3.0.0), jsonlite
Config/testthat/edition: 3
Config/roxygen2/version: 8.0.0
NeedsCompilation: no
Packaged: 2026-07-21 11:52:36 UTC; georg
Author: Georgios P. Georgiou [aut, cre]
Maintainer: Georgios P. Georgiou <georgiou.georg@unic.ac.cy>
Repository: CRAN
Date/Publication: 2026-07-30 16:40:02 UTC

Audit predictions from a completed model

Description

Evaluates predictions without retraining or requiring the fitted model. For binary classification it calculates bootstrap confidence intervals, threshold trade-offs, ROC and precision-recall curves, probability calibration, subgroup performance, and automatic warning rules.

Usage

.audit_binary_predictions(
  truth,
  probability = NULL,
  estimate = NULL,
  positive = NULL,
  threshold = 0.5,
  thresholds = seq(0.05, 0.95, by = 0.01),
  groups = NULL,
  bootstrap = 500L,
  conf_level = 0.95,
  seed = NULL,
  threshold_objective = c("balanced_accuracy", "f1", "youden", "cost"),
  false_positive_cost = 1,
  false_negative_cost = 1,
  calibration_bins = 10L,
  reference_probability = NULL,
  drift_bins = 10L,
  subgroup_min_n = 30L,
  title = "Prediction Audit",
  requirements = NULL
)

Arguments

truth

Observed outcomes.

probability

Numeric probability of the positive class.

estimate

Optional predicted classes. When omitted, classes are created from probability and threshold.

positive

Label of the positive class.

threshold

Threshold used for the main confusion matrix and metrics.

thresholds

Candidate thresholds to evaluate.

groups

Optional subgroup data frame, named list, or vector.

bootstrap

Number of stratified bootstrap samples for confidence intervals. Use zero to disable.

conf_level

Confidence level for bootstrap percentile intervals.

seed

Optional random seed used for bootstrap resampling. The default NULL uses the current random-number generator state without resetting it.

threshold_objective

Criterion for the recommended threshold: "balanced_accuracy", "f1", "youden", or "cost".

false_positive_cost

Relative cost of a false positive.

false_negative_cost

Relative cost of a false negative.

calibration_bins

Number of probability bins used in the calibration table.

reference_probability

Optional probabilities from an earlier or reference population for prediction-distribution drift checks.

drift_bins

Number of bins used for the population stability index.

subgroup_min_n

Subgroups below this size are flagged.

title

Report title.

requirements

Optional pass/fail criteria created by audit_requirements().

Value

An object of class model_audit.

Add data documentation

Description

Add data documentation

Usage

add_data_description(
  card,
  source = NULL,
  collection_period = NULL,
  population = NULL,
  sampling = NULL,
  exclusions = NULL,
  preprocessing = NULL,
  license = NULL,
  notes = NULL
)

Arguments

card

A model card.

source

Data source.

collection_period

Collection period.

population

Population represented by the data.

sampling

Sampling method.

exclusions

Exclusion criteria.

preprocessing

Important preprocessing steps.

license

Data license or access restrictions.

notes

Additional notes.

Value

The updated card.

Add known limitations

Description

Add known limitations

Usage

add_limitations(card, ...)

Arguments

card

A model card.

...

Character vectors containing limitations.

Value

The updated card.

Add intended-use information

Description

Add intended-use information

Usage

add_purpose(
  card,
  intended_use,
  users = NULL,
  out_of_scope = NULL,
  decision_context = NULL,
  owner = NULL
)

Arguments

card

A model card.

intended_use

Intended use of the model.

users

Intended users.

out_of_scope

Unsupported or prohibited uses.

decision_context

How predictions enter a decision process.

owner

Responsible person or team.

Value

The updated card.

Run model-specific diagnostic checks

Description

Diagnostics are screening evidence, not certification that assumptions hold.

Usage

audit_assumptions(card)

Arguments

card

A model card.

Value

The updated card.

Audit subgroup performance and fairness

Description

This function reports subgroup metrics and disparities. It does not declare a model to be fair or unfair; interpretation depends on the use case.

Usage

audit_fairness(card, groups = NULL, min_n = 30L)

Arguments

card

A model card.

groups

Optional character vector naming stored sensitive features. Defaults to all sensitive features supplied to model_card().

min_n

Minimum subgroup size before a subgroup is marked as small.

Value

The updated card.

Calculate model performance

Description

Calculate model performance

Usage

audit_performance(card)

Arguments

card

A model card.

Value

The updated card.

Audit Predictions from a Completed Model

Description

Audits completed binary-classification, multiclass-classification, or regression predictions without requiring the fitted model.

Usage

audit_predictions(
  truth,
  probability = NULL,
  estimate = NULL,
  task = c("auto", "binary_classification", "multiclass_classification", "regression"),
  positive = NULL,
  class_names = NULL,
  threshold = 0.5,
  thresholds = seq(0.05, 0.95, by = 0.01),
  groups = NULL,
  bootstrap = 500L,
  conf_level = 0.95,
  seed = NULL,
  threshold_objective = c("balanced_accuracy", "f1", "youden", "cost"),
  false_positive_cost = 1,
  false_negative_cost = 1,
  calibration_bins = 10L,
  reference_probability = NULL,
  drift_bins = 10L,
  subgroup_min_n = 30L,
  title = "Prediction Audit",
  requirements = NULL
)

Arguments

truth

Observed outcomes.

probability

A binary positive-class probability vector or multiclass probability matrix.

estimate

Predicted classes or numeric regression estimates.

task

Task type, or "auto".

positive

Positive class for binary classification.

class_names

Optional multiclass probability-column names.

threshold

Binary decision threshold.

thresholds

Candidate binary thresholds.

groups

Optional subgroup variables.

bootstrap

Number of bootstrap samples.

conf_level

Bootstrap confidence level.

seed

Optional random seed used for bootstrap resampling. The default NULL uses the current random-number generator state without resetting it.

threshold_objective

Binary threshold objective.

false_positive_cost

Relative binary false-positive cost.

false_negative_cost

Relative binary false-negative cost.

calibration_bins

Number of calibration bins.

reference_probability

Optional binary reference probabilities for drift.

drift_bins

Number of drift bins.

subgroup_min_n

Minimum subgroup size before a warning.

title

Audit title.

requirements

Optional criteria from audit_requirements().

Value

An object of class model_audit.

Examples

audit_predictions(c(1, 2, 3), estimate = c(1.1, 1.9, 3.2), task = "regression", bootstrap = 0)

Define prediction-audit requirements

Description

Creates optional acceptance criteria for binary classification, multiclass classification, and regression audits. Requirements that cannot be evaluated from the supplied data are reported as INCONCLUSIVE.

Usage

audit_requirements(
  min_n = NULL,
  min_accuracy = NULL,
  min_balanced_accuracy = NULL,
  min_sensitivity = NULL,
  min_specificity = NULL,
  min_precision = NULL,
  min_f1 = NULL,
  min_roc_auc = NULL,
  min_average_precision = NULL,
  min_macro_f1 = NULL,
  min_weighted_f1 = NULL,
  min_macro_roc_auc = NULL,
  min_minimum_class_recall = NULL,
  min_r_squared = NULL,
  max_false_positive_rate = NULL,
  max_false_negative_rate = NULL,
  max_brier_score = NULL,
  max_log_loss = NULL,
  max_multiclass_log_loss = NULL,
  max_multiclass_brier_score = NULL,
  max_rmse = NULL,
  max_mae = NULL,
  max_absolute_bias = NULL,
  max_mape = NULL,
  max_smape = NULL,
  max_expected_calibration_error = NULL,
  calibration_slope_range = NULL,
  max_threshold_difference = NULL,
  max_subgroup_sensitivity_gap = NULL,
  max_subgroup_specificity_gap = NULL,
  max_subgroup_selection_rate_gap = NULL,
  max_subgroup_accuracy_gap = NULL,
  max_subgroup_macro_f1_gap = NULL,
  max_subgroup_rmse_gap = NULL,
  max_subgroup_mae_gap = NULL,
  max_population_stability_index = NULL,
  use_confidence_bounds = TRUE
)

Arguments

min_n

Minimum required number of evaluation observations. Use NULL to disable this requirement.

min_accuracy

Minimum acceptable classification accuracy. Use NULL to disable this requirement.

min_balanced_accuracy

Minimum acceptable balanced accuracy. Use NULL to disable this requirement.

min_sensitivity

Minimum acceptable binary-classification sensitivity. Use NULL to disable this requirement.

min_specificity

Minimum acceptable binary-classification specificity. Use NULL to disable this requirement.

min_precision

Minimum acceptable binary-classification precision. Use NULL to disable this requirement.

min_f1

Minimum acceptable binary-classification F1 score. Use NULL to disable this requirement.

min_roc_auc

Minimum acceptable binary-classification ROC AUC. Use NULL to disable this requirement.

min_average_precision

Minimum acceptable average precision for binary classification. Use NULL to disable this requirement.

min_macro_f1

Minimum acceptable macro F1 score for multiclass classification. Use NULL to disable this requirement.

min_weighted_f1

Minimum acceptable weighted F1 score for multiclass classification. Use NULL to disable this requirement.

min_macro_roc_auc

Minimum acceptable macro ROC AUC for multiclass classification. Use NULL to disable this requirement.

min_minimum_class_recall

Minimum acceptable recall for the worst-performing class in multiclass classification. Use NULL to disable this requirement.

min_r_squared

Minimum acceptable R-squared value for regression. Use NULL to disable this requirement.

max_false_positive_rate

Maximum acceptable false-positive rate for binary classification. Use NULL to disable this requirement.

max_false_negative_rate

Maximum acceptable false-negative rate for binary classification. Use NULL to disable this requirement.

max_brier_score

Maximum acceptable Brier score for binary classification. Use NULL to disable this requirement.

max_log_loss

Maximum acceptable log loss for binary classification. Use NULL to disable this requirement.

max_multiclass_log_loss

Maximum acceptable log loss for multiclass classification. Use NULL to disable this requirement.

max_multiclass_brier_score

Maximum acceptable Brier score for multiclass classification. Use NULL to disable this requirement.

max_rmse

Maximum acceptable root mean squared error for regression. Use NULL to disable this requirement.

max_mae

Maximum acceptable mean absolute error for regression. Use NULL to disable this requirement.

max_absolute_bias

Maximum acceptable absolute mean bias for regression. Use NULL to disable this requirement.

max_mape

Maximum acceptable mean absolute percentage error for regression. Use NULL to disable this requirement.

max_smape

Maximum acceptable symmetric mean absolute percentage error for regression. Use NULL to disable this requirement.

max_expected_calibration_error

Maximum acceptable expected calibration error for classification. Use NULL to disable this requirement.

calibration_slope_range

Numeric vector of length two containing the minimum and maximum acceptable binary calibration slope. Use NULL to disable this requirement.

max_threshold_difference

Maximum acceptable absolute difference between the selected binary threshold and the package-recommended threshold. Use NULL to disable this requirement.

max_subgroup_sensitivity_gap

Maximum acceptable difference between the highest and lowest subgroup sensitivities. Use NULL to disable this requirement.

max_subgroup_specificity_gap

Maximum acceptable difference between the highest and lowest subgroup specificities. Use NULL to disable this requirement.

max_subgroup_selection_rate_gap

Maximum acceptable difference between the highest and lowest subgroup selection rates. Use NULL to disable this requirement.

max_subgroup_accuracy_gap

Maximum acceptable difference between the highest and lowest subgroup accuracies. Use NULL to disable this requirement.

max_subgroup_macro_f1_gap

Maximum acceptable difference between the highest and lowest subgroup macro F1 scores. Use NULL to disable this requirement.

max_subgroup_rmse_gap

Maximum acceptable difference between the highest and lowest subgroup RMSE values. Use NULL to disable this requirement.

max_subgroup_mae_gap

Maximum acceptable difference between the highest and lowest subgroup MAE values. Use NULL to disable this requirement.

max_population_stability_index

Maximum acceptable population stability index for prediction drift. Use NULL to disable this requirement.

use_confidence_bounds

Logical. When TRUE, minimum requirements use lower confidence bounds and maximum requirements use upper confidence bounds when bootstrap intervals are available. When FALSE, point estimates are used.

Value

An object of class audit_requirements containing the configured audit criteria.

Examples

rules <- audit_requirements(
  min_n = 100,
  min_accuracy = 0.75,
  min_roc_auc = 0.80,
  max_log_loss = 0.60
)

print(rules)

Apply pass/fail requirements to a prediction audit

Description

Evaluates configured acceptance criteria and adds a rule table plus an overall PASS, FAIL, or INCONCLUSIVE result to a model audit. Missing evidence is marked INCONCLUSIVE rather than passing automatically.

Usage

check_requirements(audit, requirements)

Arguments

audit

An object of class model_audit.

requirements

An object created by audit_requirements().

Value

The updated model_audit object.

Examples

truth <- c("no", "no", "yes", "yes")
probability <- c(0.1, 0.2, 0.8, 0.9)
audit <- audit_predictions(truth, probability, positive = "yes", bootstrap = 0)
rules <- audit_requirements(min_roc_auc = 0.80, min_sensitivity = 0.80)
audit <- check_requirements(audit, rules)
audit$overall_status
audit$rule_results

Compare two model cards

Description

Compare two model cards

Usage

compare_cards(old, new)

Arguments

old

Older model card.

new

Newer model card.

Value

A list containing metadata and metric comparisons.

Compare two sets of binary-classification predictions

Description

Uses paired stratified bootstrap resampling so each model is evaluated on the same resampled observations.

Usage

compare_predictions(
  truth,
  probability_a,
  probability_b,
  positive = NULL,
  threshold = 0.5,
  bootstrap = 1000L,
  conf_level = 0.95,
  seed = NULL,
  model_names = c("model_a", "model_b")
)

Arguments

truth

Observed outcomes.

probability_a

Positive-class probabilities from model A.

probability_b

Positive-class probabilities from model B.

positive

Positive-class label.

threshold

Threshold applied to both models.

bootstrap

Number of paired bootstrap samples.

conf_level

Confidence level for differences.

seed

Optional random seed used for paired bootstrap resampling. The default NULL uses the current random-number generator state without resetting it.

model_names

Two model names.

Value

An object of class prediction_comparison.

Create a model card

Description

Creates a structured model-card object for a regression or binary classification model. Predictions may be generated from model, or supplied directly through estimate and probability.

Usage

model_card(
  model = NULL,
  evaluation_data = NULL,
  truth,
  estimate = NULL,
  probability = NULL,
  task = c("auto", "regression", "binary_classification"),
  positive = NULL,
  threshold = 0.5,
  title = NULL,
  version = "0.1.0",
  sensitive_features = NULL,
  metadata = list()
)

Arguments

model

A fitted model object, or NULL for prediction-only cards.

evaluation_data

A data frame used for evaluation.

truth

Outcome vector or the name of its column in evaluation_data.

estimate

Predicted values/classes or a column name.

probability

Positive-class probabilities or a column name.

task

One of "auto", "regression", "binary_classification", or "multiclass_classification".

positive

Positive outcome label for classification.

threshold

Probability threshold used to create class predictions.

title

Card title.

version

Model/card version.

sensitive_features

Character vector of columns to retain for subgroup evaluation. Raw predictor columns are not stored.

metadata

Named list of extra metadata.

Value

An object of class model_card.

Compatibility Interface for Prediction Audits

Description

Calls audit_predictions() for all supported prediction tasks.

Usage

model_card_from_predictions(
  truth,
  estimate = NULL,
  probability = NULL,
  task = c("auto", "regression", "binary_classification", "multiclass_classification"),
  positive = NULL,
  class_names = NULL,
  threshold = 0.5,
  title = "Prediction Audit",
  groups = NULL,
  bootstrap = 500L,
  ...
)

Arguments

truth

Observed outcomes.

estimate

Predicted classes or numeric estimates.

probability

Binary probability vector or multiclass probability matrix.

task

Task type.

positive

Binary positive class.

class_names

Optional multiclass probability-column names.

threshold

Binary classification threshold.

title

Audit title.

groups

Optional subgroup variables.

bootstrap

Bootstrap samples.

...

Additional arguments passed to audit_predictions().

Value

A model_audit object.

Plot a prediction audit

Description

Plot a prediction audit

Usage

## S3 method for class 'model_audit'
plot(x, type = NULL, metric = "balanced_accuracy", ...)

Arguments

x

A model_audit object.

type

Plot type. Binary audits support "roc", "pr", "calibration", and "threshold"; multiclass audits support "calibration"; regression audits support "actual_predicted", "residuals", and "residual_histogram".

metric

Metric shown for a binary threshold plot.

...

Additional graphical arguments.

Value

The input model_audit object, returned invisibly. The function is called for its side effect of drawing the selected diagnostic plot.

Print audit requirements

Description

Print audit requirements

Usage

## S3 method for class 'audit_requirements'
print(x, ...)

Arguments

x

An audit_requirements object.

...

Unused.

Value

The input audit_requirements object, returned invisibly. The function is called for its side effect of printing the configured audit criteria and whether confidence bounds are used.

Print a prediction audit

Description

Print a prediction audit

Usage

## S3 method for class 'model_audit'
print(x, ...)

Arguments

x

A model_audit object.

...

Unused.

Value

The input model_audit object, returned invisibly. The function is called for its side effect of printing a concise overview of the audit, including its task, sample size, selected metrics, status, and number of findings.

Print a model card

Description

Print a model card

Usage

## S3 method for class 'model_card'
print(x, ...)

Arguments

x

A model_card object.

...

Unused.

Value

The input model_card object, returned invisibly. The function is called for its side effect of printing the card title, version, task, model class, evaluation sample size, and number of performance metrics.

Print a prediction comparison

Description

Print a prediction comparison

Usage

## S3 method for class 'prediction_comparison'
print(x, ...)

Arguments

x

A prediction_comparison object.

...

Unused.

Value

The input prediction_comparison object, returned invisibly. The function is called for its side effect of printing the compared model names, evaluation sample size, and selected comparison metrics.

Render a Prediction Audit

Description

Renders task-specific evidence and omits sections that do not apply.

Usage

render_audit(audit, output_file, format = NULL, detailed = FALSE)

Arguments

audit

A model_audit object.

output_file

Destination HTML, Markdown, or JSON file.

format

Optional explicit output format.

detailed

Whether to include detailed diagnostic tables.

Value

The output path, invisibly.

Render a model card

Description

Render a model card

Usage

render_card(card, output_file, format = NULL)

Arguments

card

A model card.

output_file

Destination file ending in .md, .html, or .json.

format

Optional explicit format: "markdown", "html", or "json".

Value

The normalized output path, invisibly.

Summarize a prediction audit

Description

Summarize a prediction audit

Usage

## S3 method for class 'model_audit'
summary(object, ...)

Arguments

object

A model_audit object.

...

Unused.

Value

A named list summarizing the audit. It contains the audit title, task, evaluation-data information, overall and class-level metrics, recommended threshold, calibration summary, subgroup disparities, diagnostics, requirement-checking results, findings, and risk warnings. Components that are not applicable to the audit task may be NULL.

Summarize a model card

Description

Summarize a model card

Usage

## S3 method for class 'model_card'
summary(object, ...)

Arguments

object

A model_card object.

...

Unused.

Value

A named list containing the model-card title, version, task, model metadata, intended purpose, performance results, fairness summary, assumption checks, limitations, and completeness-validation results.

Validate model-card completeness

Description

Validate model-card completeness

Usage

validate_card(card)

Arguments

card

A model card.

Value

A data frame of validation checks.