Publications

Un réseau de neurones augmenté pour simuler l’évolution de la distribution de matière noire

Published in (preprint), 2025

Pour étudier l’Univers, il est essentiel de pouvoir simuler rapidement l’évolution de la distribution de matière noire. Ce travail implémente un réseau de neurones 3D-UNet pour prédire efficacement le résultat d’une coûteuse simulation numérique à N corps de l’évolution de cette distribution dans un volume donné. Une nouvelle stratégie pour prédire conjointement position et vitesse des particules de la simulation est proposée, avec un terme de couplage dans la fonction d’entraînement pour assurer la cohérence entre position et vitesse. Nos résultats atteignent une précision état de l’art avec un gain significatif en temps d’exécution.

Recommended citation: Jean Prost, Pierre-Antoine Thouvenin, Jenny Sorce, Pierre Chainais

LATINO-PRO: LAtent consisTency INverse sOlver with PRompt Optimization

Published in (preprint), 2025

Text-to-image latent diffusion models (LDMs) have recently emerged as powerful generative models with great potential for solving inverse problems in imaging. However, leveraging such models in a Plug & Play (PnP), zero-shot manner remains challenging because it requires identifying a suitable text prompt for the unknown image of interest. Also, existing text-to-image PnP approaches are highly computationally expensive. We herein address these challenges by proposing a novel PnP inference paradigm specifically designed for embedding generative models within stochastic inverse solvers, with special attention to Latent Consistency Models (LCMs), which distill LDMs into fast generators. We leverage our framework to propose LAtent consisTency INverse sOlver (LATINO), the first zero-shot PnP framework to solve inverse problems with priors encoded by LCMs. Our conditioning mechanism avoids automatic differentiation and reaches SOTA quality in as little as 8 neural function evaluations. As a result, LATINO delivers remarkably accurate solutions and is significantly more memory and computationally efficient than previous approaches. We then embed LATINO within an empirical Bayesian framework that automatically calibrates the text prompt from the observed measurements by marginal maximum likelihood estimation. Extensive experiments show that prompt self-calibration greatly improves estimation, allowing LATINO with PRompt Optimization to define new SOTAs in image reconstruction quality and computational efficiency.

Recommended citation: Alessio Spagnoletti, Jean Prost, Andrés Almansa, Nicolas Papadakis, Marcelo Pereyra

Efficient posterior sampling for diverse super-resolution with hierarchical VAE Prior

Published in 19th International Joint Conference on Computer Vision Theory and Applications (VISAPP2024), 2024

We investigate the problem of producing diverse solutions to an image super-resolution problem. From a probabilistic perspective, this can be done by sampling from the posterior distribution of an inverse problem, which requires the definition of a prior distribution on the high-resolution images. In this work, we propose to use a pretrained hierarchical variational autoencoder (HVAE) as a prior. We train a lightweight stochastic encoder to encode low-resolution images in the latent space of a pretrained HVAE. At inference, we combine the low-resolution encoder and the pretrained generative model to super-resolve an image. We demonstrate on the task of face super-resolution that our method provides an advantageous trade-off between the computational efficiency of conditional normalizing flows techniques and the sample quality of diffusion based methods.

Recommended citation: Jean Prost, Antoine Houdard, Nicolas Papadakis, Andrés Almansa

Plug-and-Play image restoration with Stochastic deNOising REgularization

Published in Forty-first International Conference on Machine Learning, 2024

Plug-and-Play (PnP) algorithms are a class of iterative algorithms that address image inverse problems by combining a physical model and a deep neural network for regularization. Even if they produce impressive image restoration results, these algorithms rely on a non-standard use of a denoiser on images that are less and less noisy along the iterations, which contrasts with recent algorithms based on Diffusion Models (DM), where the denoiser is applied only on re-noised images. We propose a new PnP framework, called Stochastic deNOising REgularization (SNORE), which applies the denoiser only on images with noise of the adequate level. It is based on an explicit stochastic regularization, which leads to a stochastic gradient descent algorithm to solve ill-posed inverse problems. A convergence analysis of this algorithm and its annealing extension is provided. Experimentally, we prove that SNORE is competitive with respect to state-of-the-art methods on deblurring and inpainting tasks, both quantitatively and qualitatively.

Recommended citation: Marien Renaud, Jean Prost, Arthur Leclaire, Nicolas Papadakis https://openreview.net/forum?id=byAXJTk0LH

Inverse problem regularization with hierarchical variational autoencoders

Published in International Conference on Computer Vision (ICCV 2023), 2023

In this paper, we propose to regularize ill-posed inverse problems using a deep hierarchical variational autoencoder (HVAE) as an image prior. The proposed method synthesizes the advantages of i) denoiser-based Plug \& Play approaches and ii) generative model based approaches to inverse problems. First, we exploit VAE properties to design an efficient algorithm that benefits from convergence guarantees of Plug-and-Play (PnP) methods. Second, our approach is not restricted to specialized datasets and the proposed PnP-HVAE model is able to solve image restoration problems on natural images of any size. Our experiments show that the proposed PnP-HVAE method is competitive with both SOTA denoiser-based PnP approaches, and other SOTA restoration methods based on generative models.

Recommended citation: Jean Prost, Antoine Houdard, Nicolas Papadakis, Andrés Almansa https://openaccess.thecvf.com/content/ICCV2023/papers/Prost_Inverse_Problem_Regularization_with_Hierarchical_Variational_Autoencoders_ICCV_2023_paper.pdf

SCOTCH and SODA: A Transformer Video Shadow Detection Framework

Published in Computer Vision and Pattern Recognition (CVPR 2023), 2022

Shadows in videos are difficult to detect because of the large shadow deformation between frames. In this work, we argue that accounting for shadow deformation is essen- tial when designing a video shadow detection method. To this end, we introduce the shadow deformation attention trajectory (SODA), a new type of video self-attention mod- ule, specially designed to handle the large shadow defor- mations in videos. Moreover, we present a new shadow contrastive learning mechanism (SCOTCH) which aims at guiding the network to learn a unified shadow represen- tation from massive positive shadow pairs across differ- ent videos. We demonstrate empirically the effectiveness of our two contributions in an ablation study. Furthermore, we show that SCOTCH and SODA significantly outperforms existing techniques for video shadow detection. Code is available at the project page: https://lihaoliu- cambridge.github.io/scotch_and_soda/

Recommended citation: Lihao Liu, Jean Prost, Lei Zhu, Nicolas Papadakis, Pietro Lio, Carola-Bibiane Schönlieb, Angelica I Aviles-Rivero https://openaccess.thecvf.com/content/CVPR2023/papers/Liu_SCOTCH_and_SODA_A_Transformer_Video_Shadow_Detection_Framework_CVPR_2023_paper.pdf

Learning Local Regularization for Variational Image Restoration

Published in Scale Space and Variational Methods in Computer Vision: 8th International Conference, (SSVM 2021), 2021

In this work, we propose a framework to learn a local regularization model for solving general image restoration problems. This regularizer is defined with a fully convolutional neural network that sees the image through a receptive field corresponding to small image patches. The regularizer is then learned as a critic between unpaired distributions of clean and degraded patches using a Wasserstein generative adversarial networks based energy. This yields a regularization function that can be incorporated in any image restoration problem. The efficiency of the framework is finally shown on denoising and deblurring applications.

Recommended citation: Jean Prost, Antoine Houdard, Andrés Almansa, Nicolas Papadakis https://link.springer.com/chapter/10.1007/978-3-030-75549-2_29