156 CHAPTER5 Credibility: Evaluating What's Been Learned e = 0 . 632 × e test instances + 0 . 368 × e training instances Then, the whole bootstrap procedure is repeated several times, with different replacement samples for the training set, and the results are averaged. The bootstrap procedure may be the best way of estimating the error rate for very small datasets. However, like leave-one-out cross-validation, it has disadvan- tages that can be illustrated by considering a special, artificial situation. In fact, the very dataset we considered above will do: a completely random dataset with two classes of equal size. The true error rate is 50% for any prediction rule. But a scheme that memorized the training set would give a perfect resubstitution score of 100%, so that e training instances = 0, and the 0.632 bootstrap will mix this in with a weight of 0.368 to give an overall error rate of only 31.6% (0.632 × 50% + 0.368 × 0%), which is misleadingly optimistic. 5.5 COMPARINGDATAMININGSCHEMES We often need to compare two different learning schemes on the same problem to