Not Just Streaks: Towards Ground Truth for Single Image Derainings

Abstract

We propose a large-scale dataset of real-world rainy and clean image pairs and a method to remove degradations, induced by rain streaks and rain accumulation, from the image. As there exists no real-world dataset for deraining, current state-of-the-art methods rely on synthetic data and thus are limited by the sim2real domain gap; more- over, rigorous evaluation remains a challenge due to the absence of a real paired dataset. We fill this gap by collecting the first real paired deraining dataset through meticulous control of non-rain variations. Our dataset enables paired training and quantitative evaluation for diverse real-world rain phenomena (e.g. rain streaks and rain accumulation). To learn a representation invariant to rain phenomena, we propose a deep neural network that reconstructs the underlying scene by minimizing a rain- invariant loss between rainy and clean images. Extensive experiments demonstrate that our model outperforms the state-of-the-art deraining methods on real rainy images under various conditions.

Files

Paper (link)
Code (link)
Dataset (link)
Poster (link)
Video (link)

Challenge New!

The GT-RAIN challenge invites the public to push the boundary of single image deraining for challenging real world images degraded by various degrees of rainy weather that were collected from all around the world – stretching from North America to Asia. The competition features the first large scale dataset comprised of real rainy image and ground truth image pairs captured from over 115 scenes. The challenge is sponsored by the US Army Research Laboratory (ARL) with monetary awards for the best performing teams: $1000 USD for first place, $800 USD for second place and $500 USD for third place.

Results

Top: Ours. Bottom: Previous state of the art MPRNet^[7].

Additional Comparison

_{Note EDR V4 (S)^[6] denotes the EDR model trained on SPA-Data^[1], and EDR V4 (R)^[6] denotes the EDR model
trained on Rain14000^[8].}

_{[1] Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., Lau, R.W.: Spatial attentive
single-image deraining with a high quality real rain dataset. In: Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12270–12279 (2019)

[2] Li, R., Cheong, L.F., Tan, R.T.: Heavy rain image restoration: Integrating physics
model and conditional adversarial learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1633–1642 (2019)

[3] Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., Ma, J., Jiang, J.: Multiscale progressive fusion network for single image deraining. In: Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8346–8355 (2020)

[4] Wang, H., Xie, Q., Zhao, Q., Meng, D.: A model-driven deep neural network for single image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (June 2020)

[5] Hu, X., Zhu, L., Wang, T., Fu, C.W., Heng, P.A.: Single-image real-time rain removal
based on depth-guided non-local features. IEEE Transactions on Image Processing
30, 1759–1770 (2021)

[6] Guo, Q., Sun, J., Juefei-Xu, F., Ma, L., Xie, X., Feng, W., Liu, Y., Zhao, J.: Efficientderain: Learning pixel-wise dilation filtering for high-efficiency single-image deraining. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 1487–1495 (2021)

[7] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.:
Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14821–14831 (2021)

[8] Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from
single images via a deep detail network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3855–3863 (2017)}

Citation

@inproceedings{ba2022not, 

    title={Not Just Streaks: Towards Ground Truth for Single Image Deraining}, 

    author={Ba, Yunhao and Zhang, Howard and Yang, Ethan and Suzuki, Akira and Pfahnl, Arnold and Chandrappa, Chethan Chinder and de Melo, Celso M and You, Suya and Soatto, Stefano and Wong, Alex and others}, 

    booktitle={European Conference on Computer Vision}, 

    pages={723--740}, 

    year={2022}, 

    organization={Springer} 

}

Contact

Yunhao Ba
Electrical and Computer Engineering Department
yhba@ucla.edu

Howard Zhang
Electrical and Computer Engineering Department
hwdz15508@g.ucla.edu