Large Language Models for Inclusive Image Captioning

Abstract

The recent rapid development of artificial intelligence (AI) has opened up many new possibilities for making digital content more accessible and inclusive. One of the most exciting advancements in this area is the use of large language models (LLMs) for image captioning. Trained on vast amounts of text data, these models, among other capabilities, can generate detailed descriptions of images – this can make visual content on the Web, which often has missing or unreliable captions, more understandable for individuals with visual impairment. In this article, we investigate the possibility of using LLMs to improve image captioning of visual web content. We discuss the current capabilities of this new class of ML models, comparing several free and open-source LLMs that can be utilized for the task. Finally, we propose the architecture of a novel system that can be utilized by visually impaired web users to automatically caption visual content on the websites they visit.