The data

Radar measurements of the ionosphere – the upper layer of the atmosphere – under high-frequency waves. The goal is to detect free electrons there. 351 observations, 33 numeric attributes, and a two-class target: g for a “good” return, one that shows structure in the ionosphere, b for a return that does not.

The methods under study are logistic regression, CART, bagging, random forests, AdaBoost and gradient boosting.

data (ionosphere)
dim (ionosphere)
#> [1] 351  34
table (ionosphere [, 34])
#> 
#>   b   g 
#> 126 225

plotdata (ionosphere [, -34], ionosphere [, 34], type = "pca")

Question 1. Which method predicts best?

BAGGING and ADABOOST need to be told what to ensemble; performance passes learningmethod on to them.

# Variable on both counts: the bootstrap draws the ten resamples, and four of the six methods
# are randomised in themselves -- bagging and the forest draw their samples and their
# variables, the two boosting methods their subsamples. Without 'seed' the table moves by a
# few thousandths at every run, which is more than the gap between the two leaders.
performance (c (LR, CART, BAGGING, RANDOMFOREST, ADABOOST, GRADIENTBOOSTING),
             ionosphere [, -34], ionosphere [, 34], type = "evaluation",
             protocol = "bootstrap", eval = "accuracy", nruns = 10, seed = 0,
             learningmethod = CART)
#>                   accuracy
#> LR               0.8626247
#> CART             0.8864160
#> BAGGING          0.9071374
#> RANDOMFOREST     0.9447429
#> ADABOOST         0.9447429
#> GRADIENTBOOSTING 0.9178818

Answer. Under a bootstrap evaluation, random forests and AdaBoost give the best accuracies. Note the shape of the ranking rather than its exact order: every ensemble of trees beats the single tree it is built on, and the single tree beats the linear model.

Question 2. How do the two best differ, in false positives and false negatives?

An accuracy hides which of the two errors a model makes. Three views of the same difference – the ROC curves first, which judge the ranking of the observations rather than the decision:

performance (c (RANDOMFOREST, ADABOOST), ionosphere [, -34], ionosphere [, 34],
             type = "roc", protocol = "bootstrap", nruns = 10, fuzzy = TRUE, seed = 0,
             learningmethod = CART)

then precision and recall, which are defined for one class against the rest – b here, since performance takes the first level of the target unless positive says otherwise:

performance (c (RANDOMFOREST, ADABOOST), ionosphere [, -34], ionosphere [, 34],
             type = "evaluation", protocol = "bootstrap",
             eval = c ("precision", "recall"), nruns = 10, seed = 0, learningmethod = CART)
#>              precision    recall
#> RANDOMFOREST 0.9503386 0.8863158
#> ADABOOST     0.9627907 0.8715789

then the two confusion matrices:

performance (RANDOMFOREST, ionosphere [, -34], ionosphere [, 34], type = "confusion",
             protocol = "bootstrap", nruns = 10, seed = 0)

#>            Predicted labels
#> True labels          b          g
#>           b 0.89052632 0.10947368
#>           g 0.02415459 0.97584541
performance (ADABOOST, ionosphere [, -34], ionosphere [, 34], type = "confusion",
             protocol = "bootstrap", nruns = 10, seed = 0, learningmethod = CART)

#>            Predicted labels
#> True labels          b          g
#>           b 0.87789474 0.12210526
#>           g 0.01449275 0.98550725

Answer. The two are level on accuracy but do not distribute their errors the same way. AdaBoost is the more reluctant of the two to answer b: it recovers g slightly better and b slightly worse, so it is the more precise and the less sensitive on the minority class. Which of the two that makes preferable is not a question the data answers – it depends on what a missed b costs against a false one.