The ISMRM Open Science Initiative for Perfusion Imaging (OSIPI): Results from the OSIPI-Dynamic Contrast-Enhanced challenge

Eve S. Shalom, School of Physics and Astronomy, University of Leeds, Leeds, UK.
Harrison Kim, Department of Radiology, University of Alabama, Birmingham, Alabama, USA.
Rianne A. van der Heijden, Department of Radiology & Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, The Netherlands.
Zaki Ahmed, Corewell Health William Beaumont University Hospital, Royal Oak, Michigan, USA.
Reyna Patel, Department of Radiology, Neuroradiology Division, Mayo Clinic, Scottsdale, Arizona, USA.
David A. Hormuth, Oden Institute for Computational Engineering and Sciences, The University of Texas, Austin, Texas, USA.
Julie C. DiCarlo, Biomedical Imaging Center, Livestrong Cancer Institutes, University of Texas at Austin, Austin, Texas, USA.
Thomas E. Yankeelov, Departments of Biomedical Engineering, Diagnostic Medicine, Oncology, Livestrong Cancer Institutes, Oden Institute for Computational Engineering and Sciences, The University of Texas, Austin, Texas, USA.
Nicholas J. Sisco, Department of Translational Neuroscience, Barrow Neurological Institute, Phoenix, Arizona, USA.
Richard D. Dortch, Department of Translational Neuroscience, Barrow Neurological Institute, Phoenix, Arizona, USA.
Ashley M. Stokes, Department of Translational Neuroscience, Barrow Neurological Institute, Phoenix, Arizona, USA.Follow
Marianna Inglese, Department of Biomedicine and Prevention, University of Rome, Tor Vergata, Italy.
Matthew Grech-Sollars, Department of Surgery and Cancer, Imperial College, London, UK.
Nicola Toschi, Department of Biomedicine and Prevention, University of Rome, Tor Vergata, Italy.
Prativa Sahoo, University Medical Center Göttingen, Göttingen, Germany.
Anup Singh, Center for Biomedical Engineering, Indian Institute of Technology Delhi, New Delhi, India.
Sanjay K. Verma, Institute of Bioengineering and Bioimaging, Singapore, Singapore.
Divya K. Rathore, Institute of Psychiatry, Psychology & Neuroscience, King's College, London, UK.
Anum S. Kazerouni, Department of Radiology, University of Washington, Seattle, Washington, USA.
Savannah C. Partridge, Department of Radiology, University of Washington, Seattle, Washington, USA.
Eve LoCastro, Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA.
Ramesh Paudyal, Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA.
Ivan A. Wolansky, Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA.
Amita Shukla-Dave, Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA.
Pepijn Schouten, Department of Radiology and Nuclear Medicine, University of Amsterdam, Amsterdam, The Netherlands.
Oliver J. Gurney-Champion, Department of Radiology and Nuclear Medicine, University of Amsterdam, Amsterdam, The Netherlands.
Radovan Jiřík, Czech Academy of Sciences, Institute of Scientific Instruments, Brno, Czech Republic.
Ondřej Macíček, Czech Academy of Sciences, Institute of Scientific Instruments, Brno, Czech Republic.
Michal Bartoš, Czech Academy of Sciences, Institute of Information Theory and Automation, Praha, Czech Republic.
Jiří Vitouš, Czech Academy of Sciences, Institute of Scientific Instruments, Brno, Czech Republic.
Ayesha Bharadwaj Das, Department of Radiology, Weill Cornell Medical College, New York, New York, USA.

Document Type

Article

Abstract

PURPOSE: has often been proposed as a quantitative imaging biomarker for diagnosis, prognosis, and treatment response assessment for various tumors. None of the many software tools for quantification are standardized. The ISMRM Open Science Initiative for Perfusion Imaging-Dynamic Contrast-Enhanced (OSIPI-DCE) challenge was designed to benchmark methods to better help the efforts to standardize measurement. METHODS: A framework was created to evaluate values produced by DCE-MRI analysis pipelines to enable benchmarking. The perfusion MRI community was invited to apply their pipelines for quantification in glioblastoma from clinical and synthetic patients. Submissions were required to include the entrants' values, the applied software, and a standard operating procedure. These were evaluated using the proposed score defined with accuracy, repeatability, and reproducibility components. RESULTS: Across the 10 received submissions, the score ranged from 28% to 78% with a 59% median. The accuracy, repeatability, and reproducibility scores ranged from 0.54 to 0.92, 0.64 to 0.86, and 0.65 to 1.00, respectively (0-1 = lowest-highest). Manual arterial input function selection markedly affected the reproducibility and showed greater variability in analysis than automated methods. Furthermore, provision of a detailed standard operating procedure was critical for higher reproducibility. CONCLUSIONS: This study reports results from the OSIPI-DCE challenge and highlights the high inter-software variability within estimation, providing a framework for ongoing benchmarking against the scores presented. Through this challenge, the participating teams were ranked based on the performance of their software tools in the particular setting of this challenge. In a real-world clinical setting, many of these tools may perform differently with different benchmarking methodology.

Keywords

DCE-MRI, challenge, data analysis, glioblastoma, open-science, perfusion

Publication Date

12-19-2023

Publication Title

Magnetic resonance in medicine

E-ISSN

1522-2594

PubMed ID

38115695

Digital Object Identifier (DOI)

10.1002/mrm.29909

Link to Full Text

Share

COinS