In the last few years, there has been a growing awareness of reproducibility concerns in many areas of science.
In medical imaging, the increasing complexity of the software and pipelines undermines our ability to reproduce scientific results over time and across research teams.
This tutorial is aimed at all researchers interested in (i) improving the computational reproducibility of their research and in (ii) accessing medical imaging applications as a service (i.e. online, ready-to-use and freely available software).
We will focus on computational reproducibility, which is one's ability to obtain identical results by repeating identical treatments to the same set of inputs. Reproducibility experiments will be performed on neuroimaging applications resulting from previous Brain Tumor Segmentation (BraTS) challenges at MICCAI.
This tutorial will be structured around three parts with short presentations, demonstrations or hands-on activities. The first part will introduce common reproducibility issues in Medical Imaging; and then put the spotlight on computational reproducibility. The hands-on part will allow participants to run reproducibility experiments across pipeline versions and computing environments. To that end, participants will be provided with Jupyter Notebooks running in their own web browser. The final part will provide general best practices and tools to make reproducible computations in research.
Enjoy a pause while the experiments run their course
The hands-on part is based on two Jupyter Notebooks that can be run either through Binder, Google Colab or attendees' own Jupyter execution environment.
Step-by-step guide on how to create a VIP account and generate a VIP API key.
The ReproVIP project aims at evaluating and improving the reproducibility of scientific results obtained with the Virtual Imaging Platform (VIP) in the field of medical imaging. ReproVIP focuses on a reproducibility level ensuring that the code produces the same result when executed with the same set of inputs and that an investigator is able to reobtain the published results. We investigate reproducibility at three levels: (i) the code itself, and in particular different versions of the same code, (ii) the execution environment, such as the operating system and code dependencies, parallel executions and the use of distributed infrastructures and (iii) the exploration process, from the beginning of the study and until the final published results.
VIP is a web portal developed and deployed at the CREATIS lab since 2011. By effectively leveraging the computing and storage resources of the EGI federation, VIP offers its users high-level services enabling them to easily execute medical imaging applications on a large scale computing infrastructure. In 2022, VIP counts more than 1400 registered users and about 20 applications, among which a few internationally used and renowned ones (e.g., GATE, Freesurfer, FSL). The platform is freely available to any researcher in medical imaging (regardless of their participation to the tutorial). As a matter of fact, VIP has been the computing platform for the MSSEG and MSSEG-2 MICCAI Challenges.
This tutorial is proposed by the ReproVIP team, who gathers international members with complementarity expertise from multiple research institutions in France (CREATIS lab affiliated to CNRS, INSERM, Université Lyon I, INSA Lyon and IPHC lab affiliated to CNRS and Université de Strasbourg) and Canada (Concordia University), as described below in the Team section.
CNRS Research Engineer @ CREATIS, VIP manager
Sorina Pop is a CNRS research engineer at Creatis, currently in charge of the VIP platform and the ReproVIP project. Since her Ph.D. degree in 2013, her activity has been focused on optimizing the execution of medical image processing applications on heterogeneous distributed systems. She is particularly interested in enhancing open and reproducible science through her VIP activities, but also through other projects such as the EU OpenAIRE-Connect and EGI-ACE projects and the France Life Imaging (FLI) platform.
CNRS Research Engineer @ CREATIS, VIP main developer
Axel Bonnet is a computing research engineer at Creatis where he is currently the main developer on the VIP project. He is an expert in software architecture and development. He has taken in charge numerous developments and improvements concerning the VIP platform, including its continuous integration (CI) and the implementation of the CARMIN API.
ReproVIP CNRS Post-doc @ CREATIS
Gaël Vila holds a PhD in signal and data processing and a Master's degree in biomedical engineering. His postdoctoral research at Creatis lab focuses on computational reproducibility of medical imaging applications.
INSA associate professor in artificial intelligence and computer vision @ CREATIS
Carole Frindel is associate professor at INSA Lyon. She is a co-author of 35 publications in international journals related to MRI and signal/image processing in medicine. She possesses strong skills in the development and optimization of new image processing and machine learning methods for clinical applications with a specific focus on stroke
CNRS research scientist @ CREATIS
Helene Ratiney is a Ph.D. permanent CNRS researcher, leader of the MAGICS Team of Creatis30. She has been working since 2001 on sequence development and analysing methods for MR spectroscopy and MRI, with 45 publications in international journals in these fields. She developed a time domain quantification method (cQUEST which has recently been ported to the VIP portal and is currently being tested) for MR spectroscopy and has special interest on issues related to MR quantification in general.
UCBL research Engineer @ CREATIS
Frédéric Cervenansky is head of the Creatis IT service and has an expertise on the Girder framework for data management, GPRD aspects and on evaluating applications for computational challenges.
ReproVIP CNRS Engineer @ CREATIS
Alexandre Cornier is a CNRS engineer working on the EGI ACE project, more precisely on the authentication part, support and design of neuroimaging applications. He obtained his master 2 in Bioinformatics in 2021 after having done a 6 months internship at Inserm (Magendie Neurocenter) in the Cortical plasticity team. He worked in particular on the project of Dr. Ourania Semelidou on the atypical sensory responses in Autism Spectrum Disorder to develop a full data analysis pipeline.
UCBL research engineer @ CREATIS
Claire Mouton is a research engineer at Creatis. She is interested in scientific computing, software development and deployment and more specifically in continuous integration
CNRS research engineer, technical director @ IPHC
Jérôme Pansanel is Ph.D. Research Engineer at CNRS, technical director of France Grilles and head of the Scientific Cloud Infrastructure at IPHC. He has extensive experience in distributed computing infrastructures and virtualized environments.
CNRS research Engineer @ IPHC
Emmanuel Medernach is a Ph.D. Research Engineer at IPHC with an extensive experience in scientific computing and simulation. He is working on the use of Guix for managing the distribution of the software developed in the framework of the International Linear Collider project
Post-doc @ Concordia University
Yohan Chatelain is a postdoctoral researcher at Concordia University in Montreal, working in Professor Glatard's laboratory. His primary research topics include computer arithmetic, high-performance computing, and neuroimaging. Overall, Yohan Chatelain's research goal is to democratize the use of numerical analysis tools, ultimately facilitating greater access and usability for researchers across various disciplines.
Associate Professor @ Concordia University, Computer Science and Software Engineering
Tristan Glatard is Associate Professor at Concordia University, on Big Data Infrastructures for Neuroinformatics. His research aims at designing infrastructures to enable efficient, open and reproducible neuroinformatics. In particular, he is the PI of Boutiques and he investigates the effect of infrastructural parameters on the reproducibility of neuroimaging pipelines.