evaluation                package:CMA                R Documentation

_E_v_a_l_u_a_t_i_o_n _o_f _c_l_a_s_s_i_f_i_e_r_s

_D_e_s_c_r_i_p_t_i_o_n:

     The performance of classifiers can be evaluted by six different
     measures and two different schemes that are described more
     precisely below.

     For S4 method information, s. 'evaluation-methods'.

_U_s_a_g_e:

     evaluation(clresult, cltrain = NULL, cost = NULL, y = NULL, measure = c("misclassification", "sensitivity", "specifity", "average probability", "brier score", "auc", "0.632", "0.632+"),
                          scheme = c("iterationwise", "observationwise", "classwise"))

_A_r_g_u_m_e_n_t_s:

clresult: A list of objects of class 'cloutput' or 'clvarseloutput' 

 cltrain: An object of class 'cloutput' in which the _whole_ dataset
          was used as learning set. Only used if 'method = "0.632"' or
          'method = "0.632+"' in order to obtain an estimation for the
          resubsitution error rate.

    cost: An optional cost matrix used if 'measure =
          "misclassification"'. If it is not specified (default), the
          cost is the usual indicator loss. Otherwise, entry 'i,j' of
          'cost' quantifies the loss when the true class is class 'i-1'
          and the predicted class is 'j-1', provided the conventional
          coding '0,...,K-1' in the case of 'K' classes is used.
          Usually, the matrix contains only non-negative entries with
          zeros on the diagonal, but this is not obligatory. Make sure
          that the dimension of the matrix matches the number of
          classes.

       y: A vector containing the true class labels. Only needed if
          'scheme = "classwise"'.

 measure: Peformance measure to be used:

          '"_m_i_s_c_l_a_s_s_i_f_i_c_a_t_i_o_n"' The missclassifcation rate.

          '"_s_e_n_s_i_t_i_v_i_t_y"' The sensitivity or 1-false negative rate. Can
               only be computed for binary classifcation.

          '"_s_p_e_c_i_f_i_t_y"' The specifity or 1-false positive rate. Can
               only be computed for binary classification.

          '"_a_v_e_r_a_g_e _p_r_o_b_a_b_i_l_i_t_y"' The average probability assigned to
               the correct class. Requirement is that the used
               classifier provides probability estimations. The optimum
               performance is 1.

          '"_b_r_i_e_r _s_c_o_r_e"' The Brier Score is generally defined as '<sum
               over all observation i> <sum over all classes k>
               (I(y_i=k)-P(k))^2', with 'I()' denoting the indicator
               function and 'P(k)' the estimated probability for class
               'k'. The optimum performance is 0.

          '"_a_u_c"' The Area under the Curve (AUC) belonging to the
               empirical ROC curve computed from the estimated
               probabilities and the true class labels. Can only be
               computed for binary classification and if '"scheme =
               iterationwise"', s. below. S. also
               'roc,cloutput-method'.

          '"_0._6_3_2"' The 0.632 estimator (s. reference) for the
               misclassification rate (applied iteration- or)
               observationwise, if bootstrap learning sets have been
               used. Note that 'cltrain' must be provided.

          '"_0._6_3_2+"' The 0.632+ estimator (s. reference) for the
               misclassification rate (applied iteration- or)
               observationwise, if bootstrap learning sets have been
               used. Note that 'cltrain' must be provided.



          '"_i_t_e_r_a_t_i_o_n_w_i_s_e"' The performance measures listed above are
               computed for each different iteration, i.e. each
               different 'learningset'

          '"_o_b_s_e_r_v_a_t_i_o_n_w_i_s_e"' The performance measures listed above
               (except for '"auc"') are computed separately for each
               observation classified one or several times, depending
               on the 'learningset' scheme.

          '"_c_l_a_s_s_w_i_s_e"' The performance measures (exceptions: '"auc",
               "0.632", "0.632+"') are computed separately for each
               class, averaged over both iterations and observations.

_V_a_l_u_e:

     An object of class 'evaloutput'.

_A_u_t_h_o_r(_s):

     Martin Slawski martin.slawski@campus.lmu.de

     Anne-Laure Boulesteix <URL: http://www.slcmsr.net/boulesteix>

_R_e_f_e_r_e_n_c_e_s:

     Efron, B. and Tibshirani, R. (1997).  Improvements on
     cross-validation: The .632+ bootstrap method.

     _Journal of the American Statistical Association, 92, 548-560_.

_S_e_e _A_l_s_o:

     'evaloutput', 'classification', 'compare'

_E_x_a_m_p_l_e_s:

     ### simple linear discriminant analysis example using bootstrap datasets:
     ### datasets:
     data(golub)
     golubY <- golub[,1]
     ### extract gene expression from first 10 genes
     golubX <- as.matrix(golub[,2:11])
     ### generate 25 bootstrap datasets
     set.seed(333)
     bootds <- GenerateLearningsets(y = golubY, method = "bootstrap", ntrain = 30, niter = 10, strat = TRUE)
     ### run classification()
     ldalist <- classification(X=golubX, y=golubY, learningsets = bootds, classifier=ldaCMA)
     ### Evaluation:
     eval_iter <- evaluation(ldalist, scheme = "iter")
     eval_obs <- evaluation(ldalist, scheme = "obs")
     show(eval_iter)
     show(eval_obs)
     summary(eval_iter)
     summary(eval_obs)
     ### auc with boxplot
     eval_auc <- evaluation(ldalist, scheme = "iter", measure = "auc")
     boxplot(eval_auc)
     ### which observations have often been misclassified ?
     obsinfo(eval_obs, threshold = 0.75)

