WeatherProof: A Paired-Dataset Approach to Semantic Segmentation in Adverse Weather

Blake Gella*1 Howard Zhang*1 Rishi Upadhyay1 Tiffany Chang1 Matthew Waliman1 Yunhao Ba1 Alex Wong2 Achuta Kadambi1

University of California, Los Angeles1 Yale University2

Missing

By fine-tuning on the paired adverse and clear images of the WeatherProof dataset, the paired-training losses, and CLIP guidance, we improve InternImage’s performance on adverse weather conditions by up to 18.4%.

Abstract
The introduction of large, foundational models to computer vision has led to drastically improved performance on the task of semantic segmentation. However, these existing methods exhibit a large performance drop when testing on images degraded by weather conditions such as rain, fog, or snow. We introduce a general paired-training method that can be applied to all current foundational model architectures that leads to improved performance on images in adverse weather conditions. To this end, we create the WeatherProof dataset, the first semantic segmentation dataset with accurate clear and adverse weather image pairs, which not only enables our new training paradigm, but also improves the evaluation of the performance gap between clear and degraded segmentation. We find that training on these paired clear and adverse weather frames which share an underlying scene results in improved performance on adverse weather data. With this knowledge, we propose a training pipeline which accentuates the advantages of paired-data training using consistency losses and language guidance, which leads to performance improvements by up to 18.4% as compared to standard training procedures.

Method

Missing
By using a paired-training method with consistency losses and CLIP injection, foundational models are able to generate features that are more resilient to adverse weather conditions. During paired data training, a CLIP-Guided Injection module learns a CLIP-informed prior representing the adverse weather effect in the CLIP latent space. Clear and adverse weather images are fed into a shared weight encoder-decoder structure. Intermediate features and output segmentation maps are used in a feature consistency loss and an output consistency loss respectively to ensure an advantageous representation.



Missing
WeatherProof Dataset contains accurate clear and adverse weather image pairs with 10 semantic classes. In contrast, the ACDC dataset’s paired images have major differences in semantic information and scene structure.



Missing
Our CLIP injection layer is able to accurately predict the composition of weather effects in images. The percentage of weather effect contributions was taken by passing in these images into our CLIP injection layer and extracting the weights.


Files


Results

Missing
Our proposed paired training method outperforms standard fine-tuning on adverse images only for both InternImage and ConvNeXt. Including language guidance and consistency losses further improve our results.




Citation
@article{gella2023weatherproof,
  title={WeatherProof: A Paired-Dataset Approach to Semantic Segmentation in Adverse Weather},
  author={Gella, Blake and Zhang, Howard and Upadhyay, Rishi and Chang, Tiffany and Waliman, Matthew and Ba, Yunhao and Wong, Alex and Kadambi, Achuta},
  journal={arXiv preprint arXiv:2312.09534},
  year={2023}
}

Contact
Howard Zhang
Electrical and Computer Engineering Department
hwdz15508@ucla.edu