or it may be performed retrospectively, for example,
by using regression analysis to evaluate the effects of
demographic variables.
META-ANALYSIS
Most promising laboratory studies are the
subject of numerous performance evaluations within
a short time following their initial descriptions.
Especially interesting studies are likely to engender a
daunting bibliography. Given a profusion of
reports, how does one integrate the findings to arrive
at an accurate appraisal of the performance of the
study? Clearly, a well-defined, systematic approach
is called for, one that is able to deal in a quantitative
way with the numerical data generated in perform-
ance evaluations. The field of study concerned with
this question is the discipline of meta-analysis
(Jenicek 1989). Research in meta-analytic
techniques has been conducted for barely 30 years
and applications in clinical medicine have appeared
only recently. To date, most medical meta-analyses
have dealt with the assessment of treatment effec-
tiveness and cause-effect relationships. However,
some work has been done in developing a meta-
analytic approach to the assessment of classification
performance (Irwig
et al.
1994).
Performance evaluation meta-analysis proceeds
in three steps: 1) the assembly of the pertinent
evaluation reports, 2) qualitative meta-analysis, and
3) quantitative meta-analysis (Jenicek 1989). The
first step, retrieval of the relevant literature, is not a
trivial aspect of meta-analysis. Indeed, literature
retrieval is itself an area of research within the disci-
pline of medical informatics. Because of shortcom-
ings in any single approach to searching the
literature, it is recommended that multiple methods
be employed, including searching computerized
literature databases, reviewing appropriate journals,
and consulting expert practitioners and laboratorians
(Irwig
et al.
1994). When relevant research remains
unpublished because the findings are negative, i.e.
show poor study performance, meta-analysis will
overestimate study performance. This is a form of
publication bias (Dickersin and Berlin 1992). It is
clear that publication bias is a common problem in
the meta-analysis of therapeutic research but the
extent of this difficulty in the meta-analysis of classi-
fication study performance is not known.
Qualitative meta-analysis consists of the catego-
rization of study reports according to the design of
the performance evaluation and the assessment of the
quality of the individual evaluations. Nierenberg
and Feinstein (1988) have proposed the five category
scheme of diagnostic study design shown in Table
4.3. Each successive category in the scheme is
characterized by an increase in the breadth and rigor
of the evaluation until, in category V, an ideal
evaluation is achieved. Category IV evaluations are
those that fall somewhat short of ideal, usually
because of some limitations in the spectrum of the
study populations. Category V and category IV
evaluations are the ones upon which further meta-
analysis should be performed. The findings of the
exploratory evaluations constituting categories I, II,
and III must be considered preliminary or provi-
sional so these studies should not be included in the
meta-analysis. A three category scheme for
prognostic study design has been suggested by
Simon and Altman (1994). In their scheme,
category 1 consists of early exploratory evaluations,
comparable to Nierenberg and Feinstein's categories
I, II, and III. Category 2 represents evaluations of
the clinical performance of a study as a means of
classifying prognostic groups and category 3 consists
of clinical evaluations designed to identify subsets of
patients who will benefit from a given therapy.
Depending upon the clinical spectrum and number of
patients studied, the evaluations in these two catego-
ries correspond to Nierenberg and Feinstein's
categories IV or V.
Evaluating Classification Studies
4-9
0
0.2
0.4
0.6
0.8
1
Predicted probability
0
0.2
0.4
0.6
0.8
1
Observed probability
Figure 4.4
Hypothetical calibration curve from a validation
study. The squares represent the mean values of the
subgroup probabilities and the continuous line is the line of
identity.
