--- title: "Confidence intervals and p-values" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Confidence intervals and p-values} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) library(smartcor) ``` ## Every estimate ships with its uncertainty A correlation coefficient on its own is hard to act on. `smartcor` therefore returns a full inference payload with every estimate: a confidence interval, a p-value, the null hypothesis being tested, and a short provenance string naming the source of each quantity. The guiding principle is that the *method* dictates the *inference*: a Kendall interval should not be computed the way a Pearson interval is, and the package picks the right machinery for you and tells you which it used. ## The inference payload Consider a routine continuous-continuous pair. ```{r payload} r = smart_cor(mtcars$mpg, mtcars$wt) ``` The console block already surfaces the interval, the p-value, the test, the null, and the source. For programmatic access, `tidy()` lays the same payload out as columns. ```{r payload-tidy} library(tibble) # for printing infcols = c("estimate", "statistic", "p.value", "p_method", "null_hypothesis", "ci_lower", "ci_upper", "conf_level", "ci_method", "ci_source") as.data.frame(tidy(r)[infcols]) ``` Two fields are worth dwelling on. `ci_method` is either `"analytic"` (a closed-form or asymptotic interval) or `"bootstrap"` (a resampled interval), and `ci_source` names the exact reference or procedure behind it. Together they make the interval auditable: you can always see *how* it was produced, not just its endpoints. ## Analytic intervals are the default Thirteen of the fourteen methods carry a closed-form or asymptotic confidence interval, and `smartcor` uses it by default. Only Theil's~U, which has no widely accepted analytic interval, falls back to resampling. The table below is generated by the package itself: for each method we force that estimator on a suitable pair and read back the inference provenance, so it cannot drift from what the code actually does. ```{r source-map} set.seed(1) colour = factor(sample(c("red", "blue", "green"), 90, replace = TRUE)) shape = factor(sample(c("circle", "square", "triangle"), 90, replace = TRUE)) one = function(method, x, y) { tidy(smart_cor(x, y, method = method, verbose = FALSE))[ c("method_label", "ci_method", "ci_source", "p_method", "null_hypothesis")] } spec = list( one("pearson", mtcars$mpg, mtcars$wt), one("spearman", mtcars$mpg, mtcars$wt), one("kendall", mtcars$mpg, mtcars$wt), one("point_biserial", mtcars$mpg, mtcars$vs), one("phi", mtcars$vs, mtcars$am), one("tetrachoric", mtcars$vs, mtcars$am), one("yules_q", mtcars$vs, mtcars$am), one("polychoric", mtcars$gear, mtcars$carb), one("polyserial", mtcars$mpg, mtcars$gear), one("gamma", mtcars$gear, mtcars$carb), one("rank_biserial", mtcars$vs, mtcars$gear), one("cramers_v", colour, shape), one("theils_u", colour, shape), one("tschuprows_t", colour, shape) ) map = do.call(rbind, spec) knitr::kable(map[c("method_label", "ci_method", "ci_source")], row.names = FALSE, caption = "How each method's confidence interval is computed.") ``` The analytic sources are the standard ones: Fisher's~$z$ transform for Pearson and the point-biserial coefficient; Bonett and Wright (2000) for Spearman; Fieller, Hartley, and Pearson (1957) for Kendall; the asymptotic standard errors of Olsson (1979) and Olsson, Drasgow, and Dorans (1982) on the Fisher-$z$ scale for the latent-variable methods (polychoric, tetrachoric, polyserial); Brown and Benedetti (1977) for Goodman-Kruskal's gamma and Yule's~Q; Cliff (1996) for the rank-biserial coefficient; and a noncentral chi-square pivot for the chi-square-based association measures (phi, Cramer's~V, Tschuprow's~T). The noncentral pivot deserves a note: it inverts the chi-square test for the noncentrality parameter, so its intervals are asymmetric and correctly bound at zero when the association is weak, rather than spilling below it. One caveat: the point-biserial interval reuses the Fisher-$z$ variance $1/(n-3)$, which assumes bivariate normality; a binary margin cannot satisfy that exactly, so treat this interval as an approximation. ## The `bootstrap` argument The `bootstrap` argument is a tri-state control. - `"auto"` (the default) uses the analytic interval when one exists and is finite, and resamples only when it does not, for example when a latent-variable optimiser returns a singular Hessian or a contingency table is too sparse for the pivot. - `TRUE` always adds a percentile bootstrap, which is useful for cross-checking an analytic interval against a distribution-free one. - `FALSE` reports only the analytic interval, returning `NA` endpoints if none is available. The number of resamples is set by `n_boot` (default 500). Here we ask for a bootstrap interval on the same Pearson pair and compare it against the Fisher-$z$ interval above. ```{r bootstrap-crosscheck} rb = smart_cor(mtcars$mpg, mtcars$wt, bootstrap = TRUE, n_boot = 1000, verbose = FALSE) tidy(rb)[c("estimate", "ci_lower", "ci_upper", "ci_method", "ci_source")] ``` The two intervals agree closely here, so the asymptotic approximation holds up on this pair. Pass `seed` if you need the resampled interval to be reproducible; a seeded call restores your `.Random.seed` on exit. The confidence level is controlled by `conf_level`; lowering it narrows the interval. ```{r conf-level} rbind( `90%` = tidy(smart_cor(mtcars$mpg, mtcars$wt, conf_level = 0.90, verbose = FALSE))[c("ci_lower", "ci_upper")], `95%` = tidy(smart_cor(mtcars$mpg, mtcars$wt, conf_level = 0.95, verbose = FALSE))[c("ci_lower", "ci_upper")] ) ``` ## The null hypothesis is method-specific A correlation p-value is only meaningful against a stated null, and the natural null differs across methods. Pearson, Spearman, Kendall, and the latent-variable methods test whether the (latent) correlation is zero; the chi-square-based measures test independence of the two variables; Theil's~U tests whether one variable carries no information about the other. The `null_hypothesis` and `p_method` columns make this explicit, so a small p-value is never ambiguous about what has been rejected. ```{r nulls} knitr::kable(unique(map[c("method_label", "p_method", "null_hypothesis")]), row.names = FALSE, caption = "The null hypothesis and test behind each p-value.") ``` ## Reading interval widths Interval widths are worth reading, not just the point estimates. `compare_methods()` reports every applicable method for a pair side by side, with each interval on its own terms. ```{r compare} compare_methods(mtcars$gear, mtcars$carb) ``` The Goodman-Kruskal gamma interval is much wider than the rank-based ones. Gamma discards all tied pairs, so its effective sample size is smaller and its interval is correspondingly less precise. Two methods with similar coefficients can differ a lot in how firmly those coefficients are pinned down. ## Summary Every result carries an interval, a p-value, an explicit null, and a named source. Analytic intervals are the default for all but one method, and the `bootstrap` argument covers the rest: a distribution-free cross-check when you want one, and a fallback when an analytic interval is unavailable.