Fine-Tuning CLIP via Explainability Map Propagation for Boosting Image and Video Retrieval

Recent studies have highlighted the remarkable performance of CLIP for diverse downstream tasks. To understand how CLIP performs these tasks, various explainability methods have been formulated. In this paper, we reveal that the explainability maps associated with CLIP are often focused on a limited portion of the image and overlook objects that are explicitly mentioned in the text. This phenomenon may result in a high similarity score for incongruent image-text pairs, thereby potentially introducing a bias. To address this issue, we introduce a novel fine-tuning technique for CLIP that leverages a transformer explainability method. Unlike traditional approaches that generate a single heatmap using an image-text pair, our method produces multiple heatmaps directly from the image itself. We use these heatmaps both during the fine-tuning process and at inference time to highlight key visual elements, applying them to the features during the image encoding process, steering the visual encoder’s attention toward these key elements. This process guides the image encoder across different spatial regions and generates a set of visual embeddings, thereby allowing the model to consider various aspects of the image, ensuring a detailed and comprehensive understanding that surpasses the limited scope of the original CLIP model. Our method leads to a notable improvement in text, image, and video retrieval across multiple benchmarks. It also results in reduced gender bias, making our model more equitable.