#3256 Quality, power and fragility of randomised controlled trials in ANCA-associated vasculitis

Abstract Background and Aims Randomised controlled trials (RCTs) make up the backbone of high-grade clinical evidence. A number of recent RCTs have been published, changing the landscape of therapy in ANCA-associated vasculitis (AAV). However, not all RCTs are of high quality and they can be underpowered to detect the outcomes which might translate to sub-optimal therapeutic strategies. Using three key metrics—quality, statistical power and fragility (reversal of trial findings if the outcome of a small number of patients is changed)—we evaluated the robustness of all RCTs in AAV. Method We have searched PubMed for RCTs published from beginning of indexation to May 2024 on granulomatosis with polyangiitis, microscopic polyangiitis, and eosinophilic granulomatosis with polyangiitis RCT quality was assessed using modified Jadad 8-point scale which takes into account randomization, blinding, inclusion and exclusion criteria, withdrawal/drop-outs, adverse effects and statistical analysis. It stratified RCTs according to score (mJS) to poor (3 points or less), moderate (4 or 5 points) and high quality (6 to 8 points). RCTs were further assessed for reporting of statistical sample size calculation or power analysis. Lastly, fragility index (FI) was calculated representing the number of patients switched from event to non-event needed to change the RCT result from statistically significant to non-significant. Conversely, for studies reporting a non-significant result, reverse FI was calculated. This represents the number of patients needed to be switched from a non-event to an event to shift the results towards statistical significance. For example, if the FI is 3 then this means that if 3 or more patients did not experience trial outcome, the results would no longer be significant. Results A PubMed search resulted in 572 hits which were filtered through several rounds to include only original (main) RCTs of which 56 were found. The RCTs enrolled 5,336 patients (45.26% women, mean age 57 ± 7 years). The median sample size was 69 patients (IQR 43 to 121 patients, min-max: 17 to 704 patients), with a median of 33 events (IQR 20 to 72 events, min–max: 4 to 235 events). RCTs had a mean mJS of 5.7 ± 1.4 (min-max 3 to 8). Three (5%) RCTs were of poor quality, 17 (30%) had moderate quality, while 35 (63%) were of high quality. RCT quality increased over time (B = 0.08, p < 0.001, Fig. 1). Quality was also associated with larger RCT size (B = 0.004, p = 0.039, Fig. 2). A total of 36 (64%) RCTs reported sample size or power calculation with 35 of these providing details on calculation. Twenty RCTs (56%) reporting sample size or power analysis did not enroll a sufficient number of patients. FI was calculated for 12 RCTs while reverse FI was calculated for 33 RCTs. RCTs had a mean FI of 4.0 ± 3.3 (min-max: 0 to 10). A total of 5 (42%) RCTs had number of patients lost to follow-up, withdrawn or dropped out higher than the FI. Median reverse FI was 5.0 (IQR: 2.8 to 6.0, min-max: 0 to 15). A total of 15 (45%) of RCTs had number of patients lost to follow-up, withdrawn or dropped out higher than the reverse FI. Mean FI and reverse FI of RCTs published in the last 10 years (2015 or later) were 5 and 6.7, respectively. Reverse FI was associated with RCT size (B = 0.02, p < 0.001, Fig. 3) and quality (B = 1.1, p = 0.01, Fig. 4). There was no temporal trend of FI or reverse FI (both p > 0.05). Conclusion The quality of RCTs in AAV is rising, but they are frequently underpowered and highly fragile. On average, shifting 4 or 5 patients from event to non-event or vice versa, which is frequently less than the number of patients lost to follow-up, is enough to change the whole outcome of the trial which shows that results from these trials should be interpreted with caution.