Modeling the adaptive immune system: predictions and simulations

Lundegaard, Claus; Lund, Ole; Kesmir, Can; Brunak, Søren; Nielsen, Morten

The choice of an evaluation set is also absolutely crucial and several considerations must be taken. A large and diverse dataset is to be preferred to avoid any biases in prediction space. Extreme care should also be taken to ensure that none of the predictors have been trained on the data used for evaluation even though that might not always be possible. To make the evaluation as broad as possible cross-validation is often used, i.e. the method is trained on a large part of the available data and a smaller part is left out for evaluation. This is done until all data has been included in the evaluation set and in this way it is possible to estimate the performance on the complete dataset. Caution has to be taken, however, that the part used for training is not too similar to the evaluation part, as this will lead to an overestimation of the performance due to overtraining. This is especially true when using the leave-one-out version of cross-validation where everything except one data point is used for training, and the evaluation is then performed on the ensemble of the left out data points. Equations are as follows: