The intersection of Eye Tracking and Multimodal Large Language Models (MLLMs) is an emerging research field undergoing rapid development. This paper presents a web browsing system that enables users to select webpage elements simply by looking at them. The text and graphic content of the selected elements can then be sent to an MLLM for processing (e.g., to be summarized, translated, or enriched with additional information). The goal is to offer an augmented browsing experience where gaze input naturally complements traditional modalities, like mouse and keyboard, while users seamlessly leverage MLLMs to explore web content more efficiently. The system is designed to be compatible with different web browsers and eye-tracking devices. A user study involving 30 participants yielded positive results in terms of both usability and user satisfaction.
Gaze-Enhanced MLLM-Augmented Web Browsing
Dondi, Piercarlo;Porta, Marco
2026-01-01
Abstract
The intersection of Eye Tracking and Multimodal Large Language Models (MLLMs) is an emerging research field undergoing rapid development. This paper presents a web browsing system that enables users to select webpage elements simply by looking at them. The text and graphic content of the selected elements can then be sent to an MLLM for processing (e.g., to be summarized, translated, or enriched with additional information). The goal is to offer an augmented browsing experience where gaze input naturally complements traditional modalities, like mouse and keyboard, while users seamlessly leverage MLLMs to explore web content more efficiently. The system is designed to be compatible with different web browsers and eye-tracking devices. A user study involving 30 participants yielded positive results in terms of both usability and user satisfaction.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
