This study quantified the clinical consistency of expert panelists' ratings of appropriateness of pre-operative and post-operative chemotherapy plus radiation for rectal cancer.  For this study, a panel of nine physicians (two surgeons, four medical oncologists, three radiation oncologists) rated the appropriateness of providing pre-operative and post-operative treatments for rectal cancer, utilizing a modified-Delphi (RAND/UCLA) approach.  Clinical scenarios were paired so that each component of a pair differed by only one clinical feature (e.g., tumor stage).  A pair of appropriateness ratings was defined as inconsistent when the clinical scenario that should have had the higher (or at least equal) appropriateness rating was given a lower rating.  The rate of inconsistency was analyzed for panelists' ratings of pre- and post-operative chemotherapy plus radiation.

The final panel rating was inconsistent for 1.19% of pre-operative scenario pairs, and 0.77% of post-operative scenario pairs.  Using the conventional RAND/UCLA definition of appropriateness, the magnitude of the inconsistency would produce inconsistent appropriateness ratings in 0.43% of pre-operative and 0.11% of post-operative scenario pairs.  There was significant variation in the rate of inconsistency among individual panelists' final ratings of both pre-operative (range: 0.43-5.17%) and post-operative (range: 0.51-2.34%) scenarios.  Panelists' overall average rate of inconsistency improved significantly after the panel meeting and discussion (from 5.62 to 2.25% for pre-operative scenarios, and from 1.47 to 1.24% for post-operative scenarios, both).  There was no clear difference between specialty groups.  Inconsistency was related to the structure of the rating manual: in the second round there were no inconsistent ratings when scenario pairs occurred on the same page of the manual.