The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists

• Published in: Gastrointestinal endoscopy Vol. 75; no. 3; pp. 591 - 597
• Main Authors: Barton, John Roger, Corbett, Sally, van der Vleuten, Cees Petronella
• Format: Journal Article
• Language: English
• Published: Maryland heights, MO Mosby, Inc 01.03.2012; Elsevier
• Subjects: Biological and medical sciences; Clinical Competence; Colonoscopy - standards; Digestive system. Abdomen; Endoscopy; Gastroenterology and Hepatology; Gastroenterology. Liver. Pancreas. Abdomen; Humans; Investigative techniques, diagnostic techniques (general aspects); Medical sciences; Reproducibility of Results; NHS; BCSP; JAG; MCQ; DOPS; WBA; National Health Service; Joint Advisory Group for Gastrointestinal Endoscopy; multiple-choice questionnaire; Bowel Cancer Screening Program; workplace-based assessment; Direct Observation of Procedural Skills; Reliability; Validity
• ISSN: 0016-5107; 1097-6779; 1097-6779
• DOI: 10.1016/j.gie.2011.09.053

Practitioners increasingly need to be able to evidence the quality of their care and their clinical competence for purposes of recredentialing and relicensing. Although this may be accomplished by audit and performance data, detailed and robust assessments of competence may be valuable in certain circumstances. To develop and evaluate a detailed assessment of performance of colonoscopy. Evaluation of a Direct Observation of Procedural Skills (DOPS) method developed by an expert group of colonoscopists and clinical educationalists. English National Health Service National Bowel Cancer Screening Programme (BCSP). Aspirant colonoscopists wishing to participate in the BCSP were assessed by using the DOPS. Reliability was estimated by using generalizability theory (G), and the candidates' and assessors' perspectives on validity were evaluated by questionnaire. Grading of performance by 2 assessors over 2 consecutive real cases. DOPS grades, global expert evaluation, performance data, evaluation questionnaire scores. The assessment had high relative reliability: G = 0.81. The DOPS grades correlated highly with a global expert assessment. The candidates and assessors believed that the DOPS was a valid assessment of competence. Not guaranteed to assess therapeutic skills; evaluation questionnaire influenced by result of assessment. This is the first evaluation of a DOPS assessment on independent practitioners. It performs well, with good levels of reliability and validity, and is sufficient to be used in a high-stakes assessment. Similar approaches should be considered for assessment of competence in other areas of clinical practice for relicensing or recredentialing.