compare {CMA} | R Documentation

## Compare different classifiers

### Description

Classifiers can be evaluated separately using the method `evaluation`. Normally, several classifiers are used for the same dataset and their performance is compared. This comparison procedure is essentially facilitated by this method. For S4 method information, s. `compare-methods`

### Usage

    compare(clresultlist, measure = c("misclassification", "sensitivity", "specifity", "average probability", "brier score", "auc"), aggfun = mean, plot = FALSE, ...)

### Arguments

`clresultlist` | A list of lists (!) of objects of class `cloutput` or `clvarseloutput`. Each inner list is usually returned by `classification`. Additionally, the different list elements of the outer list should have been created by different classifiers, s. also example below.

`measure`

A character vector containing one or more of the elements listed below. By default, all measures are computed, using `evaluation` with `scheme = "iterationwise"`. Note that `"sensitivity", "specifity", "auc"` cannot be computed for the multiclass case.

`"misclassification"`
  The missclassifcation rate.
`"sensitivity"`
  The sensitivity or 1-false negative rate. Can only be computed for binary classifcation.
`"specifity"`
  The specifity or 1-false positive rate. Can only be computed for binary classification.
`"average probability"`
  The average probability assigned to the correct class. Requirement is that the used classifier provides probability estimations. The optimum performance is 1.
`"brier score"`
  The Brier Score is generally defined as `<sum over all observation i> <sum over all classes k> (I(y_i=k)-P(k))^2`, with `I()` denoting the indicator function and `P(k)` the estimated probability for class `k`. The optimum performance is 0.
`"auc"`
  The Area under the Curve (AUC) belonging to the empirical ROC curve computed from the estimated probabilities and the true class labels. Can only be computed for binary classification and if `"scheme = iterationwise"`, s. below. S. also `roc,cloutput-method`.

`aggfun` | Function that determines how performance among different iterations are aggregared. Default is `mean`, other possible choices are quantiles.
`plot` | Should the performance of different classifiers be visualized by a joint boxplot ? Default is `FALSE`.
`...` | Further arguments passed to `boxplot` in the case that `plot = TRUE`.

### Value

A `data.frame` with rows corresponding to the compared classifiers and columns to the performance measures, aggregated by `aggfun`, s. above.

### Note

If more than one measure is computed and `plot = TRUE`, one separate plot is created for each of them.

### Author(s)

Martin Slawski martin.slawski@campus.lmu.de

Anne-Laure Boulesteix http://www.slcmsr.net/boulesteix

### References

Dudoit, S., Fridlyand, J., Speed, T. P. (2002)

Comparison of discrimination methods for the classification of tumors using gene expression data.

Journal of the American Statistical Association 97, 77-87

### See Also

`classification`, `evaluation`

### Examples

    ## Not run:
    ### compare the performance of several discriminant analysis methods
    ### for the Khan dataset:
    data(khan)
    khanX <- as.matrix(khan[,-1])
    khanY <- khan[,1]
    set.seed(27611)
    fiveCV10iter <- GenerateLearningsets(y=khanY, method = "CV", fold = 5, niter = 2, strat = TRUE)
    ### candidate methods:  DLDA, LDA, QDA, pls_LDA, sclda
    class_dlda <- classification(X = khanX, y=khanY, learningsets = fiveCV10iter, classifier = dldaCMA)
    ### peform GeneSlection for LDA, FDA, QDA (using F-Tests):
    genesel_da <- GeneSelection(X=khanX, y=khanY, learningsets = fiveCV10iter, method = "f.test")
    ###
    class_lda <- classification(X = khanX, y=khanY, learningsets = fiveCV10iter, classifier = ldaCMA, genesel= genesel_da, nbgene = 10)

    class_qda <- classification(X = khanX, y=khanY, learningsets = fiveCV10iter, classifier = qdaCMA, genesel = genesel_da, nbgene = 2)

    ### We now make a comparison concerning the performance (sev. measures):
    ### first, collect in a list:
    dalike <- list(class_dlda, class_lda, class_qda)
    ### use pre-defined compare function:
    comparison <- compare(dalike, plot = TRUE, measure = c("misclassification", "brier score", "average probability"))
    print(comparison)
    ## End(Not run)

[Package CMA version 1.2.0 Index]