🌆 ProRes:
Exploring Degradation-aware Visual Prompt for Universal Image Restoration

Jiaqi Ma1,✢, Tianheng Cheng2,✢, Guoli Wang3, Xinggang Wang2, Qian Zhang3, Lefei Zhang1,📧
Wuhan University1
Huazhong University of Science & Technology2
Horizon Robotics3
Under Peer Review
✢: Equal Contribution, 📧: Corresponding Author

TL;NR: The First Visual Prompt Based All-in-one Image Restoration Framework

Code arXiv

All-in-one Image Restoration

MY ALT TEXT

Conceptual comparison with previous approaches.

Abstract

Image restoration aims to reconstruct degraded images, e.g., denoising or deblurring. Existing works focus on designing task-specific methods and there are inadequate attempts at universal methods. However, simply unifying multiple tasks into one universal architecture suffers from uncontrollable and undesired predictions. To address those issues, we explore prompt learning in universal architectures for image restoration tasks. In this paper, we present Degradation-aware Visual Prompts, which encode various types of image degradation, e.g., noise and blur, into unified visual prompts. These degradation-aware prompts provide control over image processing and allow weighted combinations for customized image restoration. We then leverage degradation-aware visual prompts to establish a controllable and universal model for image restoration, called ProRes, which is applicable to an extensive range of image restoration tasks. ProRes leverages the vanilla Vision Transformer (ViT) without any task-specific designs. Furthermore, the pre-trained ProRes can easily adapt to new tasks through efficient prompt tuning with only a few images. Without bells and whistles, ProRes achieves competitive performance compared to task-specific methods and experiments can demonstrate its ability for controllable restoration and adaptation for new tasks.

Overall Pipeline of ProRes

MY ALT TEXT

(a) Training ProRes: we add the target visual prompt to the input image and flatten the prompted image into patches. We leverage a vision transformer, i.e., ViT-Large, as the image encoder and adopt a simple pixel decoder to generate the restored image. Then we adopt pixel loss to optimize ProRes.
(b) Prompt Tuning: we freeze the weights of ProRes and randomly initialize the learnable prompts for new tasks or new datasets.

Control Ability

Adaptation on New Datasets & Task

BibTeX

@article{ma2023prores,
        title={Prores: Exploring degradation-aware visual prompt for universal image restoration},
        author={Ma, Jiaqi and Cheng, Tianheng and Wang, Guoli and Zhang, Qian and Wang, Xinggang and Zhang, Lefei},
        journal={arXiv preprint arXiv:2306.13653},
        year={2023}
      }