compare                 package:CMA                 R Documentation

_C_o_m_p_a_r_e _d_i_f_f_e_r_e_n_t _c_l_a_s_s_i_f_i_e_r_s

_D_e_s_c_r_i_p_t_i_o_n:

     Classifiers can be evaluated separately using the method
     'evaluation'. Normally, several classifiers are used for the same
     dataset and their performance is compared. This comparison
     procedure is essentially facilitated by this method. For S4 method
     information, s. 'compare-methods'

_U_s_a_g_e:

     compare(clresultlist, measure = c("misclassification", "sensitivity", "specifity", "average probability", "brier score", "auc"), aggfun = mean, plot = FALSE, ...)

_A_r_g_u_m_e_n_t_s:

clresultlist: A list of lists (!) of objects of class 'cloutput' or
          'clvarseloutput'. Each inner list is usually returned by
          'classification'. Additionally, the different list elements
          of the outer list should have been created by different
          classifiers, s. also example below.

 measure: A character vector containing one or more of the elements
          listed below. By default, all measures are computed, using
          'evaluation' with 'scheme = "iterationwise"'. Note that
          '"sensitivity", "specifity", "auc"' cannot be computed for
          the multiclass case.

          '"_m_i_s_c_l_a_s_s_i_f_i_c_a_t_i_o_n"' The missclassifcation rate.

          '"_s_e_n_s_i_t_i_v_i_t_y"' The sensitivity or 1-false negative rate. Can
               only be computed for binary classifcation.

          '"_s_p_e_c_i_f_i_t_y"' The specifity or 1-false positive rate. Can
               only be computed for binary classification.

          '"_a_v_e_r_a_g_e _p_r_o_b_a_b_i_l_i_t_y"' The average probability assigned to
               the correct class. Requirement is that the used
               classifier provides probability estimations. The optimum
               performance is 1.

          '"_b_r_i_e_r _s_c_o_r_e"' The Brier Score is generally defined as '<sum
               over all observation i> <sum over all classes k>
               (I(y_i=k)-P(k))^2', with 'I()' denoting the indicator
               function and 'P(k)' the estimated probability for class
               'k'. The optimum performance is 0.

          '"_a_u_c"' The Area under the Curve (AUC) belonging to the
               empirical ROC curve computed from the estimated
               probabilities and the true class labels. Can only be
               computed for binary classification and if '"scheme =
               iterationwise"', s. below. S. also
               'roc,cloutput-method'.

  aggfun: Function that determines how performance among different
          iterations are aggregared. Default is 'mean', other possible
          choices are quantiles.

    plot: Should the performance of different classifiers be visualized
          by a joint boxplot ? Default is 'FALSE'.

     ...: Further arguments passed to 'boxplot' in the case that 'plot
          = TRUE'.

_V_a_l_u_e:

     A 'data.frame' with rows corresponding to the compared classifiers
     and columns to the performance measures, aggregated by 'aggfun',
     s. above.

_N_o_t_e:

     If more than one measure is computed and 'plot = TRUE', one
     separate plot is created for each of them.

_A_u_t_h_o_r(_s):

     Martin Slawski martin.slawski@campus.lmu.de

     Anne-Laure Boulesteix <URL: http://www.slcmsr.net/boulesteix>

_R_e_f_e_r_e_n_c_e_s:

     Dudoit, S., Fridlyand, J., Speed, T. P. (2002)

     Comparison of discrimination methods for the classification of
     tumors using gene expression data.

     _Journal of the American Statistical Association 97, 77-87_

_S_e_e _A_l_s_o:

     'classification', 'evaluation'

_E_x_a_m_p_l_e_s:

     ## Not run: 
     ### compare the performance of several discriminant analysis methods
     ### for the Khan dataset:
     data(khan)
     khanX <- as.matrix(khan[,-1])
     khanY <- khan[,1]
     set.seed(27611)
     fiveCV10iter <- GenerateLearningsets(y=khanY, method = "CV", fold = 5, niter = 2, strat = TRUE)
     ### candidate methods:  DLDA, LDA, QDA, pls_LDA, sclda
     class_dlda <- classification(X = khanX, y=khanY, learningsets = fiveCV10iter, classifier = dldaCMA)
     ### peform GeneSlection for LDA, FDA, QDA (using F-Tests):
     genesel_da <- GeneSelection(X=khanX, y=khanY, learningsets = fiveCV10iter, method = "f.test")
     ###
     class_lda <- classification(X = khanX, y=khanY, learningsets = fiveCV10iter, classifier = ldaCMA, genesel= genesel_da, nbgene = 10)

     class_qda <- classification(X = khanX, y=khanY, learningsets = fiveCV10iter, classifier = qdaCMA, genesel = genesel_da, nbgene = 2)

     ### We now make a comparison concerning the performance (sev. measures):
     ### first, collect in a list:
     dalike <- list(class_dlda, class_lda, class_qda)
     ### use pre-defined compare function:
     comparison <- compare(dalike, plot = TRUE, measure = c("misclassification", "brier score", "average probability"))
     print(comparison)
     ## End(Not run)

