Context

Due to the dramatic increase of longitudinal acquisitions in the past decades such as video sequences, global positioning system (GPS) tracking or medical follow-up, many applications for time-series data mining have been developed. Thus, un-supervised time-series filtering and data mining have become highly relevant with the aim to automatically denoise, detect and identify similar temporal patterns in time-series datasets.

Synthetic image sequences (left) used for quantitative assessments and a real data MRI longitudinal study.

Objectives

1- Filtering of spatiotemporal image sequences

We will focus on anisotropic diffusion filtering as it is one of the most efficient and tractable filtering approach. Considering longitudinal medical images as image sequences, we want to extend Perona and Malik’s basic conception of anisotropic diffusion including temporal dimension in the process. This will allow us to gain information from both spatial and temporal dimensions.

We may derive our filtering formulation from the Barash’s discrete form [Barash02], chosen for intuitiveness and stability, described using bandwidth matrices for the spatial, the intensity and the temporal dimensions. One drawback of anisotropic diffusion filtering being its poor performance to remove very noisy points in homogeneous regions, we also add a regularization term to the iterative procedure formulation to suppress remaining outlier values.

2- Clustering of spatiotemporal image sequences

The mean-shift technique, which has been proposed by [Fukunaga75] can be used in the context of image filtering and segmentation [Comaniciu02]. Men-shift has already been applied to spatiotemporal data filtering (Anjum and Cavallaro (2008)) and some works were published on longitudinal MRI processing with mean-shift. However, they have neither directly used the time information, reducing it to a scalar value, nor explicitly formulated how to handle the time dimension.

In contrast, we claim to explain how to extend mean-shift to spatio-temporal data by adding a constraint on the evolution of the samples over time. Only the samples sharing a similar evolution pattern of their feature will contribute to the data filtering.

Results

1- Locally Controlled Regularized Spatiotemporal Anisotropic Diffusion (RAD+t)

We have proposed a new method to filter image sequences, taking into account both spatial and temporal dimensions in the filtering process. We extended Perona and Malik's anisotropic diffusion filtering works [Perona90] using Barash's form [Barash02] while adding temporal data and a locally controlled regularized term in the process that suppresses peak noise in homogeneous regions.

We have shown that our method improves noise removal and preserves region boundaries on image sequences with a time-evolving region compared to BM3D [Dabov07].

We have also proven the efficiency of using the regularization term to improve denoising [Portejoie-15]. The use of three scale parameters to control the anisotropic diffusion allows intuitive tuning of the parameters. However our approach requires a fine tuning of 2 over 6 parameters (three scale parameters, ratio defining outliers and the number of iterations).

Input (left) and results for BM3D [Dabov07] (center) and our RAD+t (right) for 20th time frame of synthetic data (top) and 4th time frame of real MRI data (middle). Bottom presents a detail of a particular region of real data.

2- Spatio temporal Mean-Shift (STM-S)

We introduce a new formalism that takes into account the temporal evolution of a feature, by decomposing the STM-S equation in two parts in order to handle independently the spatial and the temporal dimensions. We have studied the relative importance of each scale parameter and we have shown the great importance of the range scale parameter, which embeds a temporal neighborhood restriction to ensure good convergence of the result. The validation of STM-S on synthetic data demonstrates high fidelity with respect to the ground truth, and its superiority over M-S for the filtering of noisy spatiotemporal data.

While the standard M-S framework gathers data only for one specific time point at a time, our spatio-temporal variant of M-S enables the separation of samples following different temporal paths in feature space. Consequently, taking into account the temporal dimension leads to more precise filtering results where features overlap at one or more time-points. This allows more precise discrimination of groups of pixels that would have been indistinguishable at individual time-points.

Furthermore, the need to tune only two parameters to run the STM-S procedure ensures simplicity and suggests that it should facilitate STM-S use in several contexts.

Slice of a 3D ROI containing a multiple sclerosys lesion evolving over 48 weeks. The grey level evolution of the pixels are shown at the bottom of the figure. The red, blue and green curves represent the evolutions of the red, blue and green landmarks in the top images sequence.

Slices of three different 3D ROI containing a Multiple sclerosys lesion. (b) Grey level evolution curves after STM-S filtering, (c) STM-S filtering result of (a). (d) Classification of the filtered grey level evolution of (b). The pixel class colors correspond to the colors of the filtered evolution curves in (a).

Collaborations and supports

• F. Cotton, Professeur des universités-Praticien hospitalier, Team 6 Creatis member.
• C. Guttmann, Dr. Guttmann is the Director of the Center for Neurological Imaging at Brigham and Women's Hospital and an Assistant Professor in Radiology at Harvard Medical School. His main interest is the quantitative evaluation of normal and pathological states of the brain using MRI. Among Dr. Guttmann's specific goals are the understanding of the natural course of MS and of white matter disorders in the elderly. Using MRI findings as phenotypic descriptors and elucidating the relationship between brain morphological changes and functional deficits is particularly emphasized.

• These works were funded by CNRS grant PEPS INS2LabEx PRIMES.
• These works were performed within the framework of the LABEX PRIMES (ANR-11-LABX-0063) of Université de Lyon, within the program "Investissements d’Avenir" (ANR-11-IDEX-0007) operated by the French National Research Agency (ANR).

References

[Perona90] P. Perona and J. Malik, “Scale-space and edge detection using anisotropic diffusion,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 12, no. 7,pp. 629–639, Jul 1990.

[Barash02]  D. Barash, “Fundamental relationship between bilateral filtering, adaptive smoothing, and the nonlinear diffusion equation,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 24, no. 6, pp. 844–847, Jun 2002.

[Fukunaga75] K. Fukunaga, L.D. Hostetler. “Estimation of the gradient of a density function with applications in pattern recognition”. Information Theory, IEEE Transaction on 21, 32–40, 1975.

[Comaniciu02] D. Comaniciu, P. Meer, “Mean shift: A robust approach toward feature space analysis”. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 603–619, 2002

[Dabov07] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, "Image denoising by sparse 3D transform-domain collaborative filtering," IEEE Trans. Image Process., vol. 16, no. 8, August 2007