Evaluating Multimembership Classifiers: A Methodology and Application to the MEDAS Diagnostic System

Performance evaluation measures for multimembership classifiers are presented and applied in a retrospective study on the diagnostic performance of the MEDAS (Medical Emergency Decision Assistance System) system. Admission and discharge diagnoses for 122 patients with one or more of 26 distinct disorders in five major disorder categories were gathered. The average number of disorders per patient was 2 with 36 (29.5 percent) patients having 3 or more disorders simultaneously. The features (symptoms, signs, and laboratory data) available at admission were entered into a multimembership Bayesian pattern recognition algorithm which permits for diagnosis of multiple disorders. When the top five computer-ranked diagnoses were considered, all of the correct diagnoses for 86.1 percent of the patients were displayed by the fifth position. In 71.6 percent of these cases, no false diagnosis preceded any correct diagnosis. In ten cases a discharge diagnosis which was suggested by the available findings was omitted by the admitting physician. In six of these ten cases, the overlooked diagnoses appeared at the computer ranked list above all false diagnoses. Considering the urgency of diagnosis in the Emergency Department, the high uncertainty involved due to the limited availability of data, and the high frequency with which multiple disorders coexist, this limited study encourages our confidence in the MEDAS knowledge base and algorithm as a useful diagnostic support tool.