Make your Results Reproducible with the Virtual Imaging Platform (Thu. October 12th, 8 AM)

Tutorial objectives

In the last few years, there has been a growing awareness of reproducibility concerns in many areas of science. In medical imaging, the increasing complexity of the software and pipelines undermines our ability to reproduce scientific results over time and across research teams. This tutorial is aimed at all researchers interested in (i) improving the computational reproducibility of their research and in (ii) accessing medical imaging applications as a service (i.e. online, ready-to-use and freely available software).
We will focus on computational reproducibility, which is one's ability to obtain identical results by repeating identical treatments to the same set of inputs. Reproducibility experiments will be performed on neuroimaging applications resulting from previous Brain Tumor Segmentation (BraTS) challenges at MICCAI.

Program

This tutorial will be structured around three parts with short presentations, demonstrations or hands-on activities. The first part will introduce common reproducibility issues in Medical Imaging; and then put the spotlight on computational reproducibility. The hands-on part will allow participants to run reproducibility experiments across pipeline versions and computing environments. To that end, participants will be provided with Jupyter Notebooks running in their own web browser. The final part will provide general best practices and tools to make reproducible computations in research.

8:00 - 9:00

Reproducibility. Where do I begin ?

Tutorial introduction
Presentation - G. Vila (20')
What triggered my interest in reproducibility ?
Discussion - speakers & attendees (20')
Zoom on computational reproducibility
Presentation - G. Vila (20')

9:00 - 10:00

Hands-on / Part 1 : Reproducibility Experiments

Setup your computing environment
Hands-on - attendees (15')
VIP Tutorial
Demo - A. Bonnet (15')
Launch reproducibility experiments using Notebook N°1
Hands-on - attendees (30')

10:00 - 10:30

Coffee Break

Enjoy a pause while the experiments run their course

10:30 - 11:00

Hands-on / Part 2: Output Analyses

Feedback on Part 1
Presentation - G. Vila (5')
Analyze your own results using Notebook N°2
Hands-on - attendees (25')

11:00 - 12:00

Best Practices and Tools for Reproducible Analyses

Hands-on summary
Presentation - G. Vila (10')
Lessons learnt from the ReproVIP project
Presentation & Demo - S. Pop & A. Bonnet (20')
Larger overview of reproducibility issues in Medical Imaging
Presentation - S. Pop (15')
Which other tools/astuces/pratiques would you like to share ?
Discussion - speakers & attendees (15')

Materials

Hands-On Part

The hands-on part is based on two Jupyter Notebooks that can be run either through Binder, Google Colab or attendees' own Jupyter execution environment.

Binder
Start the hands-on with Binder
The Binder session may be slow to start. If it does not open within 10 minutes, try it with Colab.
Colab
Start the hands-on with Google Colab
Colab requires a valid Google account. If you do not wish to use a Google account, you may download and execute the Notebooks on your own computer.
Notebooks
Download the Notebooks on your computer
You need JupyterLab of another IDE compatible with Notebooks (e.g. PyCharm, VSCode) to run these files on your computer.

VIP Account & Settings

Step-by-step guide on how to create a VIP account and generate a VIP API key.

About us

The ReproVIP project aims at evaluating and improving the reproducibility of scientific results obtained with the Virtual Imaging Platform (VIP) in the field of medical imaging. ReproVIP focuses on a reproducibility level ensuring that the code produces the same result when executed with the same set of inputs and that an investigator is able to reobtain the published results. We investigate reproducibility at three levels: (i) the code itself, and in particular different versions of the same code, (ii) the execution environment, such as the operating system and code dependencies, parallel executions and the use of distributed infrastructures and (iii) the exploration process, from the beginning of the study and until the final published results.
VIP is a web portal developed and deployed at the CREATIS lab since 2011. By effectively leveraging the computing and storage resources of the EGI federation, VIP offers its users high-level services enabling them to easily execute medical imaging applications on a large scale computing infrastructure. In 2022, VIP counts more than 1400 registered users and about 20 applications, among which a few internationally used and renowned ones (e.g., GATE, Freesurfer, FSL). The platform is freely available to any researcher in medical imaging (regardless of their participation to the tutorial). As a matter of fact, VIP has been the computing platform for the MSSEG and MSSEG-2 MICCAI Challenges.
This tutorial is proposed by the ReproVIP team, who gathers international members with complementarity expertise from multiple research institutions in France (CREATIS lab affiliated to CNRS, INSERM, Université Lyon I, INSA Lyon and IPHC lab affiliated to CNRS and Université de Strasbourg) and Canada (Concordia University), as described below in the Team section.

Team members

Sorina Pop

Sorina Pop

CNRS Research Engineer @ CREATIS, VIP manager

Sorina Pop is a CNRS research engineer at Creatis, currently in charge of the VIP platform and the ReproVIP project. Since her Ph.D. degree in 2013, her activity has been focused on optimizing the execution of medical image processing applications on heterogeneous distributed systems. She is particularly interested in enhancing open and reproducible science through her VIP activities, but also through other projects such as the EU OpenAIRE-Connect and EGI-ACE projects and the France Life Imaging (FLI) platform.

Membre de l'équipe

Axel Bonnet

CNRS Research Engineer @ CREATIS, VIP main developer

Axel Bonnet is a computing research engineer at Creatis where he is currently the main developer on the VIP project. He is an expert in software architecture and development. He has taken in charge numerous developments and improvements concerning the VIP platform, including its continuous integration (CI) and the implementation of the CARMIN API.

Membre de l'équipe

Gaël Vila

ReproVIP CNRS Post-doc @ CREATIS

Gaël Vila holds a PhD in signal and data processing and a Master's degree in biomedical engineering. His postdoctoral research at Creatis lab focuses on computational reproducibility of medical imaging applications.

Membre de l'équipe

Carole Frindel

INSA associate professor in artificial intelligence and computer vision @ CREATIS

Carole Frindel is associate professor at INSA Lyon. She is a co-author of 35 publications in international journals related to MRI and signal/image processing in medicine. She possesses strong skills in the development and optimization of new image processing and machine learning methods for clinical applications with a specific focus on stroke

Membre de l'équipe

Helene Ratiney

CNRS research scientist @ CREATIS


Helene Ratiney is a Ph.D. permanent CNRS researcher, leader of the MAGICS Team of Creatis30. She has been working since 2001 on sequence development and analysing methods for MR spectroscopy and MRI, with 45 publications in international journals in these fields. She developed a time domain quantification method (cQUEST which has recently been ported to the VIP portal and is currently being tested) for MR spectroscopy and has special interest on issues related to MR quantification in general.

Membre de l'équipe

Frédéric Cervenansky

UCBL research Engineer @ CREATIS


Frédéric Cervenansky is head of the Creatis IT service and has an expertise on the Girder framework for data management, GPRD aspects and on evaluating applications for computational challenges.

Membre de l'équipe

Alexandre Cornier

ReproVIP CNRS Engineer @ CREATIS

Alexandre Cornier is a CNRS engineer working on the EGI ACE project, more precisely on the authentication part, support and design of neuroimaging applications. He obtained his master 2 in Bioinformatics in 2021 after having done a 6 months internship at Inserm (Magendie Neurocenter) in the Cortical plasticity team. He worked in particular on the project of Dr. Ourania Semelidou on the atypical sensory responses in Autism Spectrum Disorder to develop a full data analysis pipeline.

Membre de l'équipe

Claire Mouton

UCBL research engineer @ CREATIS

Claire Mouton is a research engineer at Creatis. She is interested in scientific computing, software development and deployment and more specifically in continuous integration

Membre de l'équipe

Jérôme Pansanel

CNRS research engineer, technical director @ IPHC

Jérôme Pansanel is Ph.D. Research Engineer at CNRS, technical director of France Grilles and head of the Scientific Cloud Infrastructure at IPHC. He has extensive experience in distributed computing infrastructures and virtualized environments.

Membre de l'équipe

Emmanuel Medernach

CNRS research Engineer @ IPHC

Emmanuel Medernach is a Ph.D. Research Engineer at IPHC with an extensive experience in scientific computing and simulation. He is working on the use of Guix for managing the distribution of the software developed in the framework of the International Linear Collider project

Membre de l'équipe

Yohan Chatelain

Post-doc @ Concordia University

Yohan Chatelain is a postdoctoral researcher at Concordia University in Montreal, working in Professor Glatard's laboratory. His primary research topics include computer arithmetic, high-performance computing, and neuroimaging. Overall, Yohan Chatelain's research goal is to democratize the use of numerical analysis tools, ultimately facilitating greater access and usability for researchers across various disciplines.

Membre de l'équipe

Tristan Glatard

Associate Professor @ Concordia University, Computer Science and Software Engineering

Tristan Glatard is Associate Professor at Concordia University, on Big Data Infrastructures for Neuroinformatics. His research aims at designing infrastructures to enable efficient, open and reproducible neuroinformatics. In particular, he is the PI of Boutiques and he investigates the effect of infrastructural parameters on the reproducibility of neuroimaging pipelines.

Affiliated institutions and partners