WeatherStream: Light Transport Automation of Single Image Deweathering

Howard Zhang*1 Yunhao Ba*1 Ethan Yang1 Varan Mehra1 Blake Gella1 Akira Suzuki1 Arnold Pfahnl1 Chethan Chinder Chandrappa1 Alex Wong2 Achuta Kadambi1

University of California, Los Angeles1 Yale University2

CVPR 2023, Vancouver, Canada

image
We propose a new dataset, publicly available,
that makes all image-based deweathering models we tested perform better.

Abstract
Today single image deweathering is arguably more sensitive to the dataset type, rather than the model. We introduce WeatherStream, an automatic pipeline capturing all real-world weather effects (rain, snow, and rain fog degradations), along with their clean image pairs. Previous state-of-the-art methods that have attempted the all-weather removal task train on synthetic pairs, and are thus limited by the Sim2Real domain gap. Recent work has attempted to manually collect time multiplexed pairs, but the use of human labor limits the scale of such a dataset. We introduce a pipeline that uses the power of light-transport physics and a model trained on a small, initial seed dataset to reject approximately 99.6% of unwanted scenes. The pipeline is able to generalize to new scenes and degradations that can, in turn, be used to train existing models just like fully human-labeled data. Training on a dataset collected through this procedure leads to significant improvements on multiple existing weather removal methods on a carefully human-collected test set of real-world weather effects.


Files


Challenge  New!
The GT-RAIN challenge invites the public to push the boundary of single image deraining for challenging real world images degraded by various degrees of rainy weather that were collected from all around the world – stretching from North America to Asia. The competition features the first large scale dataset comprised of real rainy image and ground truth image pairs captured from over 115 scenes. The challenge is sponsored by the US Army Research Laboratory (ARL) with monetary awards for the best performing teams: $1000 USD for first place, $800 USD for second place and $500 USD for third place.

The WeatherStream challenge extends this challenge to multiple weather conditions. Coming soon!


Results

In contrast to previous datasets, this dataset is not manually collected, but uses algorithms based on light transport to automatically curate the dataset. Doing this algorithmically not only improves quality of the dataset, but also the scale.


Citation

@inproceedings{zhang2023weatherstream,
  title={WeatherStream: Light Transport Automation of Single Image Deweathering},
  author={Zhang, Howard and Ba, Yunhao and Yang, Ethan and Mehra, Varan and Gella, Blake and Suzuki, Akira and Pfahnl, Arnold and Chandrappa, Chethan Chinder and Wong, Alex and Kadambi, Achuta},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2023}
}


Contact
Howard Zhang
Electrical and Computer Engineering Department
hwdz15508@g.ucla.edu
 
Yunhao Ba
Electrical and Computer Engineering Department
yhba@ucla.edu