The data
The crabs dataset of the MASS package, on
Leptograpsus variegatus crabs collected at Fremantle,
Australia. Two colour forms, orange and blue, and both sexes. Five
measurements were taken on each individual (in mm): the size of the
frontal lobe (FL), the rear width (RW), the
length of the carapace (CL), its width (CW)
and the depth of the body (BD).
There are two targets rather than one: the species (sp,
O or B) and the sex (sex,
F or M), with 50 males and 50 females of each
species.
data (crabs, package = "MASS")
summary (crabs)
#> sp sex index FL RW CL
#> B:100 F:100 Min. : 1.0 Min. : 7.20 Min. : 6.50 Min. :14.70
#> O:100 M:100 1st Qu.:13.0 1st Qu.:12.90 1st Qu.:11.00 1st Qu.:27.27
#> Median :25.5 Median :15.55 Median :12.80 Median :32.10
#> Mean :25.5 Mean :15.58 Mean :12.74 Mean :32.11
#> 3rd Qu.:38.0 3rd Qu.:18.05 3rd Qu.:14.30 3rd Qu.:37.23
#> Max. :50.0 Max. :23.10 Max. :20.20 Max. :47.60
#> CW BD
#> Min. :17.10 Min. : 6.10
#> 1st Qu.:31.50 1st Qu.:11.40
#> Median :36.80 Median :13.90
#> Mean :36.41 Mean :14.03
#> 3rd Qu.:42.00 3rd Qu.:16.60
#> Max. :54.60 Max. :21.60
plotdata (crabs [, 4:8], crabs [, 1], type = "pairs")
plotdata (crabs [, 4:8], crabs [, 2], type = "pairs")
The five measurements are almost perfectly correlated with each other
– a crab is simply bigger or smaller – so most of what any one of them
says, the others say too. That is what makes the selection the
interesting question here rather than the classifier.
round (cor (crabs [, 4:8]), 3)
#> FL RW CL CW BD
#> FL 1.000 0.907 0.979 0.965 0.988
#> RW 0.907 1.000 0.893 0.900 0.889
#> CL 0.979 0.893 1.000 0.995 0.983
#> CW 0.965 0.900 0.995 1.000 0.968
#> BD 0.988 0.889 0.983 0.968 1.000
Question 1. Selecting for the species
Using a ranking algorithm and Fisher’s index as the univariate
criterion, which multivariate criterion – the F statistic, mRMR, or a
wrapper – gives the best predictions of the species with a naive Bayes
classifier?
# The two filters are deterministic; the wrapper is not -- it judges each subset by fitting a
# naive Bayes classifier under a bootstrap, so without 'seed' it can stop at a different
# subset from one run to the next. That is the criterion's own variance, not the data's.
s.fstat1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
unieval = "fisher", multieval = "fstat", seed = 0)
s.mrmr1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
unieval = "fisher", multieval = "mrmr", seed = 0)
s.wrap1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
seed = 0)
s.fstat1
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : fstat
#> features kept : 1: FL
#> score : 46.99
s.mrmr1
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : mrmr
#> features kept : 1: FL
#> score : 0.1974
s.wrap1
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : wrapper
#> features kept : 1: FL
#> score : 0.6779
performance (NB, crabs [, 4:8], crabs [, 1], nruns = 100, seed = 0)
#> accuracy
#> 0.6191188
performance (NB, crabs [, 4:8] [, s.fstat1$selection], crabs [, 1], nruns = 100, seed = 0)
#> accuracy
#> 0.6626326
performance (NB, crabs [, 4:8] [, s.mrmr1$selection], crabs [, 1], nruns = 100, seed = 0)
#> accuracy
#> 0.6626326
performance (NB, crabs [, 4:8] [, s.wrap1$selection], crabs [, 1], nruns = 100, seed = 0)
#> accuracy
#> 0.6626326
Answer. All three criteria stop at the same
subset and therefore give the same performance – which is better than
using all five variables. Adding correlated measurements to a naive
Bayes classifier, whose whole assumption is that they are independent,
costs accuracy.
Question 2. Selecting for the sex
The same question, for the sex of the crabs.
s.fstat2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
unieval = "fisher", multieval = "fstat", seed = 0)
s.mrmr2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
unieval = "fisher", multieval = "mrmr", seed = 0)
s.wrap2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
seed = 0)
s.fstat2
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : fstat
#> features kept : 1: RW
#> score : 18.4
s.mrmr2
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : mrmr
#> features kept : 1: RW
#> score : 0.1264
s.wrap2
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : wrapper
#> features kept : 3: RW, CL, BD
#> score : 0.7358
performance (NB, crabs [, 4:8], crabs [, 2], nruns = 100, seed = 0)
#> accuracy
#> 0.6624966
performance (NB, crabs [, 4:8] [, s.fstat2$selection], crabs [, 2], nruns = 100, seed = 0)
#> accuracy
#> 0.6236062
performance (NB, crabs [, 4:8] [, s.mrmr2$selection], crabs [, 2], nruns = 100, seed = 0)
#> accuracy
#> 0.6236062
performance (NB, crabs [, 4:8] [, s.wrap2$selection], crabs [, 2], nruns = 100, seed = 0)
#> accuracy
#> 0.7129453
Answer. Only the wrapper improves on the full
set of variables. The two filter criteria stop at a single variable and
lose accuracy – which is the trade-off of the family: a filter judges a
subset without ever fitting the model it is selecting for, and here that
judgement is wrong.
Question 3. Which variables, in the end?
colnames (crabs) [4:8] [s.fstat1$selection]
#> [1] "FL"
colnames (crabs) [4:8] [s.wrap2$selection]
#> [1] "RW" "CL" "BD"
Answer. The frontal lobe alone is what
distinguishes the two species; distinguishing the sexes takes three
measurements – the rear width, the length of the carapace and the depth
of the body. Two different questions asked of the same five
measurements, and two different answers.