Reliability Testing of the CollaboRATE and SHARED Patient-Reported Measures of Shared Decision Making within Surgical Treatment Pathways: A Mixed-Methods Study

Abstract

ABSTRACT Objectives To evaluate the test-retest reliability of two instruments measuring shared decision-making (SDM) and to explore factors affecting the stability of participant’s lived experience of SDM during the test-retest interval. Design Mixed-methods prospective cohort study, nested within an ongoing National Health Service (NHS, United Kingdom) Trust Quality Improvement study Setting Seven surgical specialties within a tertiary NHS Hospital Trust. Participants/Procedures Patients (>18 years) booked for elective surgical interventions were invited to complete two measurement instruments at each timepoint: CollaboRATE (a 3-item questionnaire developed to assess patients’ perception of professionals SDM skills) and SHARED (a 10-item questionnaire to assess patients’ experience of making a treatment decision with health professionals). Instruments were completed at baseline timepoint (date of surgery booking consultation) and 5-10 days after consultation, but prior to surgery (retest timepoint). Main outcome measures Intra-class correlation (two-way, mixed, absolute agreement) and Kappa coefficients at item, total- and top-score levels were calculated. Bland-Altman plots were used to describe agreement between initial and retest measurement for each instrument. Semi-structured interviews to explore participants’ lived experience were conducted remotely after retest measurement with a purposively selected sample of patients of varying socio-demographic characteristics, surgical specialties, and direction of score change. Transcriptions underwent thematic analysis using inductive coding approaches to identify themes. Results 86 patients completed retest measurements (median time to completion = 8 days). Test-retest reliability was weak for CollaboRATE (ICC=0.34, p<0.001) and moderate for SHARED (ICC=0.52, p<0.001) total scores. Test-retest reliability was moderate for CollaboRATE top score (κ = 0.47, p<0.001). Interviews with nine patients identified two key themes driving instability in the test-retest interval: 1) ongoing reflection on the SDM process, and 2) a need for more support for SDM. Conclusion Our study demonstrated weak-moderate reliability in measuring patient-reported SDM which may be explained by patients’ continued reflection on decisions after surgical consultations. Future research should consider the fact that SDM is a process, and work is needed to understand how and when to optimally measure SDM so that the impact of continued reflection and reasoning are not missed. KEY MESSAGES What is already known on this topic? Instruments (e.g. self-reported questionnaires) measuring SDM from the patient perspective are widely used in clinical pathways, in research, and to guide the development of healthcare services. Few studies have evaluated the test-retest reliability of these instruments, and those done to date report poor reliability. What this study adds? This mixed-methods study explored the test-retest reliability of two SDM measurement instruments in the context of decision-making around surgical interventions. It is the first to supplement standard test-retest methods with interviews to explore patients’ perceptions of SDM over time in this setting. We found that patients reflect on decision-making processes and consider how this may affect the interpretation of reliability metrics in the context of SDM measurement. Our results affirm the dynamic nature of SDM and suggest that test-retest reliability may therefore not be an indicator of the overall quality of an SDM measure when used in this context. How this study might affect research, practice or policy – summarise the implications of this study SDM measurement instruments have different levels of reliability. Routine SDM measurement may identify a subgroup of patients who reflectively question their decisions for medical treatment in the time following consultation. There is scope to target interventions to improve SDM in the period between consultation and procedure. Clinicians and policymakers should design pathways with the timing of SDM measurement kept in mind.

The authors' abstract, as published at the source. Medical Decision Making, 2026 · DOI ↗