Question 14

Illustrate and describe the Receiver Operator Curve (ROC) and the information gained

from it.

For a good answerthe following areas should have been addressed. Diagnostic tests may be

correct or incorrect. The accuracy of a test is assessed by its sensitivity (true positive rate)

and its specificity (true negative rate). The ROC provides a graphical representation of the

trade-off between sensitivity on the y axis and specificity or 1-specificity (false positive rate)

on the x axis. Any increase in sensitivity will be accompanied by a decrease in specificity. It

accounts for an arbitrary cut off level made for a test or comparing two or more diagnostic

tests. A gradient of 1 (area under the curve of 0.5) suggests that the test has no predictive

ability. A steeper gradient has increased area under the curve (ideally > 0.75) and improved

predictive ability. The best point on the curve is dependent on the consequences of a false

positive compared with a false negative of the test and is usually the L elbow of the curve.

The ROC is not affected by changes in prevalence as sensitivity and specificity are not

dependant on prevalence. An illustration of the ROC, with correctly labelled axis and

features was essential to answer this question. Few candidates scored well in this question,

but of those that did they generally achieved a good score. This area is well covered in the

recommended text Statistical Methods for Anaesthesia and Intensive Care by P Myles and T

Gin pages 98 to 99.

As far as college answers go, this is one of the good ones.

In brief:

- The ROC curve is a plot of sensitivity vs. false positive rate (1-specificity)
- Sensitivity is on the y-axis, from 0% to 100%
- The ROC is for tests which produce results on a numerical scale, rather than binary (positive vs. negative results)
- The ROC curve can be used to determine the cut off point at which the sensitivity and specificity are optimal.
- All possible combinations of sensitivity and specificity that can be achieved by changing the test's cutoff value can be summarised using a single parameter, the area under the ROC curve (AUC).
- The higher the AUC, the more accurate the test
- An AUC of 1.0 means the test is 100% accurate (i.e. the curve is square)
- An AUC of 0.5 (50%) means the ROC curve is a a straight diagonal line, which represents the "ideal bad test", one which is only ever accurate by pure chance.

- When comparing two tests, the more accurate test is the one with an ROC curve further to the top left corner of the graph, with a higher AUC.
- The best cutoff point for a test (which separates positive from negative values) is the point on the ROC curve which is closest to the top left corner of the graph.
- The cutoff values can be selected according to whether one wants more sensitivity or more specificity.

Information gained from the ROC curve:

- A simple graphical comparison of accuracy between diagnostic tests
- Optimal cut-off value can be derived from the shape of the curve - simplest method is to choose the point which is closest to 1,0 coordinate (i.e. the top left corner)
- The tanget at any point is the likelihood ratio for the single test value at that point
- Slope between the origin and any point along the curve is the positive likelihood ratio where the selected point is the criterion for positivity
- The slope between any two points is the likelihood ratio for a test with a defined level, bounded by the two points.

Hanley, James A., and Barbara J. McNeil. "The meaning and use of the area under a receiver operating characteristic (ROC) curve." *Radiology* 143.1 (1982): 29-36.

Zweig, Mark H., and Gregory Campbell. "Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine." *Clinical chemistry* 39.4 (1993): 561-577.

Cook, Nancy R. "Use and misuse of the receiver operating characteristic curve in risk prediction." *Circulation* 115.7 (2007): 928-935.

Jones, Catherine M., and Thanos Athanasiou. "Summary receiver operating characteristic curve analysis techniques in the evaluation of diagnostic tests."*The Annals of thoracic surgery* 79.1 (2005): 16-20.

Metz, Charles E. "ROC methodology in radiologic imaging." *Investigative radiology* 21.9 (1986): 720-733.

Søreide, Kjetil. "Receiver-operating characteristic curve analysis in diagnostic, prognostic and predictive biomarker research." *Journal of clinical pathology* 62.1 (2009): 1-5.

Lusted, Lee B. "ROC recollected." *Medical Decision Making* (1984): 131-135.