The bootstrap: A technique for data-driven statistics. Using computer-intensive analyses to explore experimental data

Abstract

Background

The concept of resampling data – more commonly referred to as bootstrapping – has been in use for more than three decades. Bootstrapping has considerable theoretical advantages when it is applied to non-Gaussian data. Most of the published literature is concerned with the mathematical aspects of the bootstrap but increasingly this technique is being utilized in medical and other fields.

Methods

I reviewed the published literature following a 1994 publication assessing the transfer of technology, including the bootstrap, to the biomedical literature.

Results

In the ten-year period following that 1994 paper there were 1679 published references to the technique in Medline. In that same time period the following citations were found in the four major medical journals—British Medical Journal (48), JAMA (51), Lancet (52) and the New England Journal of Medicine (45).

Content

I introduce the basic theory of the bootstrap, the jackknife, and permutation tests. The bootstrap is used to estimate the accuracy of an estimator such as the standard error, a confidence interval, or the bias of an estimator. The technique may be useful for analysing smallish expensive-to-collect data sets where prior information is sparse, distributional assumptions are unclear, and where further data may be difficult to acquire. Some of the elementary uses of bootstrapping are illustrated by considering the calculation of confidence intervals such as for reference ranges or for experimental data findings, hypothesis testing such as comparing experimental findings, linear regression, and correlation when studying association and prediction of variables, non-linear regression such as used in immunoassay techniques, and ROC curve processing.

Conclusions

These techniques can supplement current nonparametric statistical methods and should be included, where appropriate, in the armamentarium of data processing methodologies.

Introduction

In a 1994 review Altman and Goodman [1] identified influential statistical articles and the time pattern of their citations in the medical literature. One such article described the bootstrap [2]—the topic of this review. I used an Ovid Technologies Medline keyword search [“bootstrap” or “resampling”] for the period 1995 to 2004 to assess the subsequent pattern of citations in the medical literature and recovered 1679 references.¹ These citations increased year-by-year since 1995 (Fig. 1). I also performed a full-text search (numbers of citations in parenthesis) of research articles in the journals BMJ (48), JAMA (51), Lancet (52), and the New England Journal of Medicine (45) over the same period (due to archive limitations some of these searches were for shorter periods). These findings suggest that bootstrap methods are increasingly being utilised in the medical literature. These techniques have also found wide application in such diverse fields as astronomy, biology, economics, engineering, genetics, molecular biology, and finance [3].

In exploring aspects of the bootstrap in this review I largely used S-Plus, version 6.2 (Insightful Corp.), R, version 1.9 [4], a series of available S-Plus and R libraries [5], [6], [7], Confidence Interval Calculator, version 2 [8], [9], Analyse-it for Microsoft Excel, version 1.72 [10], and CBstat, version 5 [11]. In passing, it is worth noting that it is perfectly possible to implement the bootstrap and jackknife within a spreadsheet program [12], [13], [14] although the random number generator may not be entirely satisfactory (see Random number generators).

All the data used to illustrate this review are typical of the type of data analysed in the practice of clinical chemistry. These data were obtained from Hand et al. [15], Harris and Boyd [16], Beck and Shultz [17], Krzanowski [18], results of proficiency testing of enzyme determinations in Ontario [19], [20], and a study of the rate of removal of lactate dehydrogenase-1 (LD-1) from serum following a myocardial infarction [21].