Common errors in the design of orthopaedic trials: Has anything changed?

  • Aaron Gazendam
    Affiliations
    Division of Orthopaedics, Department of Surgery, McMaster University, Hamilton, ON Canada

    Centre for Evidence-Based Orthopaedics, 293 Wellington St. N, Suite 110, Hamilton, ON L8L 8E7, Canada
    Search for articles by this author
  • Seper Ekhtiari
    Affiliations
    Division of Orthopaedics, Department of Surgery, McMaster University, Hamilton, ON Canada

    Centre for Evidence-Based Orthopaedics, 293 Wellington St. N, Suite 110, Hamilton, ON L8L 8E7, Canada
    Search for articles by this author
  • Luc Rubinger
    Correspondence
    Corresponding author at: Division of Orthopaedics, Department of Surgery, McMaster University, Hamilton, ON Canada.
    Affiliations
    Division of Orthopaedics, Department of Surgery, McMaster University, Hamilton, ON Canada
    Search for articles by this author
  • Mohit Bhandari
    Affiliations
    Division of Orthopaedics, Department of Surgery, McMaster University, Hamilton, ON Canada

    Centre for Evidence-Based Orthopaedics, 293 Wellington St. N, Suite 110, Hamilton, ON L8L 8E7, Canada
    Search for articles by this author
Published:December 08, 2021DOI:https://doi.org/10.1016/j.injury.2021.12.010

      Highlights

      • An increased focus on sample size calculations and adequately powered studies will allow for more robust and valid findings in clinical trials.
      • Blinding in orthopaedic trials is challenging, but there has been effort to increase blinding to reduce the risk of exaggerated effect sizes.
      • Expertise-based trials represent a novel design in surgical based trials, with the potential to reduce bias and increase validity and feasibility.
      • Patient reported outcomes and minimal important differences represent important trial outcomes that have seen increased uptake.

      Abstract

      Introduction

      The adoption of evidence-based orthopaedics has shifted the focus from expert base opinions and anecdotal evidence to a focus on integrating the best available clinical research. This shift has led to an increased focus on randomized controlled trials (RCTs) within the field. Although RCTs are considered the highest level of evidence, methodologic errors can introduce bias and limit the validity of the results. Early trials were hampered by lack of blinding, inadequate sample sizes and other design flaws. The objective of this review was to examine the current literature to determine if the design and execution of RCTs has improved.

      Design Errors

      The awareness of the importance of sample size increased over time with substantially more trials reporting sample size calculations. However, many contemporary RCTs are still underpowered and fail to reach their calculated sample size. Given the challenges of surgically based RCTs, the majority of historical trials lacked blinding, increasing the risk of bias. There is evidence that there has been a concerted effort to increase the blinding in RCTs, particularly in outcome assessors. A more recent development in the design of surgical trials is the introduction of expertise-based trial designs in which patients are randomized to a surgeon with expertise in a particular intervention. These trials minimize the bias that can arise from differential expertise bias and have the potential to improve the validity and feasibility of RCTs. Finally, there has been an increased focus on the reporting of patient reported outcomes (PROs) in orthopaedic RCTs. Alongside this movement has been the development of minimal important differences (MIDs) to define the changes that are relevant and meaningful to patients. Both PROs and MIDs should be taken into consideration when calculating the sample size and study power in clinical trials.

      Conclusions

      Although marked improvements have been made in the design and implementation of trials, there is still considerable room for improvement. Adequately blinded and powered studies evaluating clinically important outcomes and differences should be key considerations in trial design moving forward.

      Keywords

      To read this article in full you will need to make a payment

      Subscribe:

      Subscribe to Injury
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Bhandari M.
        • Tornetta P.
        Editorial comment: evidence-based orthopaedics: a paradigm shift.
        Clin Orthopaed Relat Res (1976-2007). 2003; 413: 8-10
        • Bhandari M.
        • Giannoudis P.V.
        Evidence-based medicine: what it is and what it is not.
        Injury. 2006; 37: 302-306https://doi.org/10.1016/j.injury.2006.01.034
        • Gazendam A.M.
        • Nucci N.W.
        • Ekhtiari S.
        • Lanting B.A.
        • MacDonald S.J.
        • Wood T.J.
        Quantifying the level of evidence of podium presentations at the american association of hip and knee surgeons from 2015 to 2019.
        J Arthroplasty. 2021; 36: 2219-2222
        • Smith C.S.
        • Mollon B.
        • Vannabouathong C.
        • Fu J.M.
        • Sales B.
        • Bhandari M.
        • et al.
        An assessment of randomized controlled trial quality in the journal of bone & joint surgery: update from 2001 to 2013.
        JBJS. 2020; 102: e116https://doi.org/10.2106/JBJS.18.00653
        • Farrokhyar F.
        • Karanicolas P.J.
        • Thoma A.
        • Simunovic M.
        • Bhandari M.
        • Devereaux P.J.
        • et al.
        Randomized controlled trials of surgical interventions.
        Ann Surg. 2010; 251: 409-416https://doi.org/10.1097/SLA.0b013e3181cf863d
        • Bhandari M.
        • Richards R.R.
        • Sprague S.
        • Schemitsch E.H.
        The quality of reporting of randomized trials in the journal of bone and joint surgery from 1988 through 2000.
        JBJS. 2002; 84: 388-396
        • Koucheki R.
        • Gazendam A.M.
        • Perera J.R.
        • Griffin A.
        • Ferguson P.
        • Wunder J.
        • et al.
        Assessment of risk of bias in osteosarcoma and Ewing's sarcoma randomized controlled trials: a systematic review.
        Curr Oncol. 2021; 28: 3771-3794
        • Farrow L.
        • Gardner W.T.
        • Ablett A.D.
        • Kutuzov V.
        • Johnstone A.
        A review of trauma and orthopaedic randomised clinical trials published in high-impact general medical journals.
        Eur J Orthopaed Surg Traumatol. 2021; : 1-11
        • Lochner H.V.
        • Bhandari M.
        • Tornetta I.II.P
        Type-II error rates (beta errors) of randomized trials in orthopaedic trauma.
        JBJS. 2001; 83: 1650-1655
        • Bhandari M.
        • Tornetta P.
        • Rampersad S.A.
        • Sprague S.
        • Heels-Ansdell D.
        • Sanders D.W.
        • et al.
        (Sample) size matters! an examination of sample size from the SPRINT trial.
        J Orthop Trauma. 2013; 27: 183-188https://doi.org/10.1097/BOT.0b013e3182647e0e
        • Ekhtiari S.
        • Gazendam A.
        • Nucci N.
        • Kruse C.
        • Bhandari M.
        The fragility of statistically significant findings from randomized controlled trials in Hip and knee arthroplasty.
        J Arthroplasty. 2020; 0https://doi.org/10.1016/j.arth.2020.12.015
        • Poolman R.W.
        • Struijs P.A.
        • Krips R.
        • Sierevelt I.N.
        • Marti R.K.
        • Farrokhyar F.
        • et al.
        Reporting of outcomes in orthopaedic randomized trials: does blinding of outcome assessors matter?.
        JBJS. 2007; 89: 550-558
        • Karanicolas P.J.
        • Bhandari M.
        • Taromi B.
        • Akl E.A.
        • Bassler D.
        • Alonso-Coello P.
        • et al.
        Blinding of outcomes in trials of orthopaedic trauma: an opportunity to enhance the validity of clinical trials.
        JBJS. 2008; 90: 1026-1033https://doi.org/10.2106/JBJS.G.00963
        • Kay J.
        • Memon M.
        • Simunovic N.
        • Musahl V.
        • Fu F.H.
        • Karlsson J.
        • et al.
        A historical analysis of randomized controlled trials in anterior cruciate ligament surgery.
        JBJS. 2017; 99: 2062-2068
        • Investigators S to PERIN in P with TF (SPRINT)
        Randomized trial of reamed and unreamed intramedullary nailing of tibial shaft fractures.
        J. Bone Joint Surg Am Volume. 2008; 90: 2567
        • Devereaux P.J.
        • Bhandari M.
        • Clarke M.
        • Montori V.M.
        • Cook D.J.
        • Yusuf S.
        • et al.
        Need for expertise based randomised controlled trials.
        BMJ. 2005; 330: 88https://doi.org/10.1136/bmj.330.7482.88
        • Scholtes V.A.
        • Nijman T.H.
        • van Beers L.
        • Devereaux P.J.
        • Poolman R.W.
        Emerging designs in orthopaedics: expertise-based randomized controlled trials.
        JBJS. 2012; 94: 24-28
        • Bednarska E.
        • Bryant D.
        • Devereaux P.J.
        Orthopaedic surgeons prefer to participate in expertise-based randomized trials.
        Clin Orthop Relat Res. 2008; 466: 1734-1744
        • Cook J.A.
        • Elders A.
        • Boachie C.
        • Bassinga T.
        • Fraser C.
        • Altman D.G.
        • et al.
        A systematic review of the use of an expertise-based randomised controlled trial design.
        Trials. 2015; 16: 241https://doi.org/10.1186/s13063-015-0739-5
        • Investigators H.
        Total hip arthroplasty or hemiarthroplasty for hip fracture.
        N Engl J Med. 2019; 381: 2199-2208
        • Beard D.J.
        • Davies L.J.
        • Cook J.A.
        • MacLennan G.
        • Price A.
        • Kent S.
        • et al.
        The clinical and cost-effectiveness of total versus partial knee replacement in patients with medial compartment osteoarthritis (TOPKAT): 5-year outcomes of a randomised controlled trial.
        Lancet. 2019; 394: 746-756https://doi.org/10.1016/S0140-6736(19)31281-4
        • Porter M.E.
        What is value in health care.
        N Engl J Med. 2010; 363: 2477-2481
        • Jaeschke R.
        • Singer J.
        • Guyatt G.H.
        Measurement of health status: ascertaining the minimal clinically important difference.
        Control Clin Trials. 1989; 10: 407-415
        • Danoff J.R.
        • Goel R.
        • Sutton R.
        • Maltenfort M.G.
        • Austin M.S.
        How much pain is significant? Defining the minimal clinically important difference for the visual analog scale for pain after total joint arthroplasty.
        J Arthroplasty. 2018; 33 (e2): S71-S75https://doi.org/10.1016/j.arth.2018.02.029
        • Maltenfort M.
        • Díaz-Ledezma C.
        Statistics in brief: minimum clinically important difference – availability of reliable estimates.
        Clin Orthopaed Relat Res. 2017; 475: 933-946https://doi.org/10.1007/s11999-016-5204-6
        • Leopold S.S.
        • Porcher R.
        Editorial: the minimum clinically important difference – the least we can do.
        Clin Orthopaed Relat Res. 2017; 475: 929-932https://doi.org/10.1007/s11999-017-5253-5
        • Reito A.
        • Raittio L.
        • Helminen O.
        Revisiting the sample size and statistical power of randomized controlled trials in orthopaedics After 2 Decades.
        JBJS Rev. 2020; 8: e0079https://doi.org/10.2106/JBJS.RVW.19.00079