Anonymous submission · Under review

A Generalist Foundation Model
for Gaze Target Detection

LookAnything unifies diverse gaze-centric tasks within a single multimodal framework. By internalizing gaze priors via large-scale cross-domain learning, the framework bypasses explicit auxiliary inputs, enabling flexible adaptation across various scenarios, task settings, and output formats.

Watch video Model HoliGaze Results

311KQ&A pairsHoliGaze corpus and benchmark

9Gaze-centric tasksFive capability dimensions

2BModel scalePoints, boxes, heatmaps, language

33VLMs outperformedOn every HoliGaze-Bench metric

This video could not be loaded.

A Unitree G1 carries LookAnything through real scenes. The reel covers social interaction, shopping assistance, a tabletop setting, point, box, heatmap, and language outputs, plus static and moving viewpoints.

01 / Overview

Unifying diverse gaze-centric tasks.

Existing methods rely on auxiliary inputs like head crops or depth maps, and are confined to isolated target detection with heatmap outputs. Benefiting from large-scale multi-task and cross-domain training, LookAnything flexibly adapts to various settings, generating diverse outputs (points, boxes, heatmaps, and language) and supporting broad gaze-centric localization, description, and reasoning.

The limitation

Bypassing external explicit inputs.

Head crops and depth maps add cascading error and extra runtime. A heatmap alone has no instance-level meaning, so the same model cannot describe the target, find who is looking, or refuse a bad question.

The approach

Gaze priors in the latent space.

We introduce a set of learnable gaze-prior queries that internalize scene depth, source-head grounding, and target spatial geometry directly into the latent space, which promotes intrinsic gaze-aware representation learning. The language head then reads those representations and emits a point, a box, a heatmap, or text.

LookAnything and HoliGaze overview: data engine, nine tasks, and benchmark highlights around the model. — HoliGaze supplies 311K question-answer pairs over nine tasks. LookAnything trains on that corpus and is evaluated both as a holistic gaze model and as a conventional gaze-target detector.

02 / Model

Instruction-conditioned autoregressive generation.

LookAnything appends learnable gaze-prior queries to multimodal tokens to learn gaze-aware representations via auxiliary geometric supervision. The language head then autoregressively decodes the answer in a unified output space. The auxiliary branches can be skipped during inference.

LookAnything architecture. An RGB image and a task prompt enter a large language model with gaze-prior queries. Auxiliary decoders predict depth, a head mask, and a gaze heatmap during training. — Built on Qwen3-VL-2B. The vision encoder is adapted with LoRA. Each query group has 9 tokens. The language head is the only path required at inference.

Depth Queries

Relative monocular depth

Depth queries reconstruct relative monocular depth with a lightweight decoder. The loss is scale-invariant, allowing the model to focus on layout geometry.

Head Queries

Source head grounding

Head queries predict a binary mask of the looking person, serving as the spatial anchor for gaze localization.

Gaze Queries

Target spatial geometry

Gaze queries predict a spatial heatmap, a normalized point, and a bounding box, enabling diverse localization output formats.

Block-wise attention

Disentangle queries into specialized groups.

Forcing a single set of tokens to capture disparate physical and geometric properties causes feature entanglement. By explicitly disentangling the queries into specialized groups (each attending only to the image, the question, and itself), LookAnything maintains clean, domain-specific representations.

Block-wise attention mask. Depth, head, and gaze query groups do not attend to each other. Answer tokens attend to every group.

03 / HoliGaze

A comprehensive resource for holistic gaze.

HoliGaze standardizes eight existing gaze datasets into a single schema, comprising 311K Q&A samples across 9 distinct tasks: 303K for training and an 8K human-checked benchmark over 6,143 images.

Nine HoliGaze tasks with example questions: gaze target and source description and localization, depth, direction, contact, presence, and ambiguous reference. — Open-ended questions cover generative localization and description. Multiple choice covers tasks that can be scored automatically. The benchmark mixes the two formats one to one.

GTL · GTD

Target detection & description

Given a source person, detect the gaze target (point or bounding box) or describe the attended object.

GSL · GSD

Source detection & description

Given a target, detect the bounding box of the source person looking at it, or describe the person.

GDO · GDR

Spatial reasoning

Reason about the spatial relationship, including 3D relative distance comparison (farther or closer) and 2D relative direction classification.

GCD

Contact detection

Determine whether the source person is making direct eye contact with the camera (mutual gaze).

ARR · PV

Answerability reasoning

Identify unanswerable queries, such as ambiguous referring expressions or out-of-frame gaze targets.

HoliGaze data engine: collection from eight datasets, annotation completion, template question generation, and human verification, with corpus and benchmark statistics. — Native train and test splits are kept, so benchmark images do not leak into the corpus. HoliGaze-Bench is checked instance by instance. The automatic drafts were edited on 0.70% of items.

1. Collect

Annotations from eight academic gaze datasets are standardized into a unified schema.

2. Complete

Dense labels are automatically augmented using state-of-the-art vision models like SAM and Depth Anything.

3. Ask

Canonical Q&A generation templates produce instruction-response pairs covering both open-ended and multiple-choice formats.

4. Check

The 8K benchmark split is rigorously reviewed and verified by human annotators.

311KQ&A pairs

171KImages

187KGaze interactions

8Source datasets

0.70%Bench edits

Samples count native gaze annotations or question-answer pairs. HoliGaze is the only set in this comparison with both referring types, point, box, and text outputs, and both question formats.
Dataset	Samples	Tasks	Text ref.	Box ref.	Point	Box	Text	MCQ	openQA
GazeFollow NeurIPS 2015	130K	1	–	Yes	Yes	–	–	–	–
VideoCoAtt CVPR 2018	138K	1	–	–	Yes	Yes	–	–	–
VAT CVPR 2020	165K	2	–	Yes	Yes	–	–	–	–
GOO CVPRW 2021	172K	2	–	Yes	Yes	Yes	–	–	–
ChildPlay ICCV 2023	258K	2	–	Yes	Yes	–	–	–	–
GazeHOI NeurIPS 2024	56K	2	–	Yes	Yes	–	Yes	–	–
VL4Gaze arXiv 2025	489K	4	Yes	–	Yes	–	Yes	–	Yes
Gaze-Co CVPR 2026	120K	1	Yes	–	Yes	Yes	–	–	–
HoliGaze This work	311K	9	Yes	Yes	Yes	Yes	Yes	Yes	Yes

04 / Results

Exceeding VLMs in breadth, matching specialists in precision.

HoliGaze-Bench tests localization, description, spatial reasoning, social contact, and refusal. To verify that task unification does not compromise conventional gaze target detection, we also evaluate LookAnything on 5 official test splits of specialist benchmarks.

HoliGaze-Bench. We compare LookAnything with leading closed-source and open-source models, including generalist and embodied models. LookAnything is the 2B model trained on HoliGaze-Corpus. It is best on every column. The strongest other model in a column is underlined. Lower is better only for GTL point L₂. GTD and GSD report multiple-choice accuracy or F1 together with open-ended ROUGE-L and BLEU-4. Click a metric to sort.

Geometric grounding is the primary gap.

While closed-source models excel on descriptive MCQs, most struggle on localization. Broad language and reasoning capability therefore does not intrinsically yield precise gaze-conditioned grounding.

Embodied pretraining does not automatically transfer.

Embodied models perform adequately on contact or depth questions, but remain weak on point and box localization, indicating that generic spatial or action-centric supervision cannot substitute for gaze-specific priors.

Broad gains at a compact scale.

Compared to its Qwen3-VL-2B backbone, LookAnything reduces GTL-T2P L₂ from 0.387 to 0.094. Even 27B–38B generalists fall far short of its localization averages, showing the gains are not from scale but from large-scale multi-task training and gaze-aware representation learning.

HoliGaze-Bench score against model size. LookAnything at 2B lies above the trend of much larger open models. — HoliGaze-Bench versus model size. The 2B LookAnything model is the point above the open-source trend.

Five conventional gaze-target benchmarks

Official test splits of GazeFollow, VideoAttentionTarget, ChildPlay, GazeHOI, and GOO-Real. LookAnything achieves the best overall performance with an average L₂ of 0.109 and average AUC of 0.942, matching or exceeding dedicated specialists on core localization.

Method	Average		VAT		GazeFollow			ChildPlay		GazeHOI			GOO-Real
Method	L₂ ↓	AUC ↑	L₂ ↓	AUC ↑	L₂ ↓	Min L₂ ↓	AUC ↑	L₂ ↓	AUC ↑	L₂ ↓	AUC ↑	Acc ↑	L₂ ↓	AUC ↑
Recasens et al.	0.253	0.809	0.242	0.789	0.227	0.155	0.857	0.292	0.754	0.259	0.817	0.371	0.245	0.829
Lian et al.	0.226	0.835	0.242	0.789	0.189	0.119	0.885	0.292	0.754	0.200	0.877	0.561	0.205	0.868
Chong et al.	0.172	0.857	0.126	0.864	0.138	0.077	0.923	0.128	0.858	0.197	0.869	0.564	0.272	0.770
HGTTR	0.224	0.825	0.266	0.816	0.169	0.109	0.847	0.204	0.859	0.216	0.824	0.528	0.265	0.780
MADGTD	0.172	0.857	0.128	0.860	0.138	0.077	0.925	0.133	0.866	0.195	0.870	0.578	0.265	0.762
Gupta et al.	0.181	0.867	0.128	0.898	0.156	0.091	0.914	0.144	0.893	0.198	0.867	0.556	0.279	0.765
Miao et al.	0.170	0.877	0.117	0.906	0.132	0.072	0.928	0.122	0.919	0.194	0.864	0.584	0.287	0.768
Tafasca et al.	0.121	0.921	0.123	0.904	0.128	0.065	0.935	0.112	0.924	—	—	—	—	—
ViTGaze	0.136	0.900	0.103	0.908	0.105	0.047	0.949	0.112	0.899	0.163	0.880	0.714	0.196	0.862
Sharingan	0.150	0.900	0.104	0.920	0.112	0.055	0.944	0.110	0.930	0.177	0.886	0.633	0.248	0.819
Gaze-LLE (ViT-B)	0.132	0.930	0.105	0.932	0.104	0.045	0.956	0.115	0.945	0.162	0.906	0.705	0.175	0.909
Gaze-LLE (ViT-L)	0.127	0.929	0.100	0.937	0.098	0.041	0.958	0.102	0.950	0.160	0.907	0.710	0.173	0.915
GazeAnywhere	0.183	0.897	0.236	0.845	0.179	0.115	0.923	0.139	0.916	0.196	0.885	0.625	0.165	0.918
LookAnything	0.109	0.942	0.098	0.941	0.091	0.048	0.945	0.094	0.956	0.130	0.940	0.771	0.132	0.927

Bold is best and underline is second best, including a tie on VAT L₂. Tafasca et al. do not report GazeHOI or GOO-Real. AUC uses the query-based heatmap.

05 / Real world

Deployment on a Unitree G1 robot.

To evaluate out-of-domain generalization, we deploy the model in real-world physical environments. From the robot’s egocentric perspective, LookAnything accurately grounds the source person, reliably recognizes social gaze signals, and precisely tracks gaze targets under both static and moving perspectives.

Scenarios

Three settings, one policy. The corner flag is gaze-contact with the robot. The blue overlay is the predicted source head.

This clip could not be loaded.

Viewpoint

The same grounding holds when the camera is fixed and when the robot is walking.

This clip could not be loaded.

Output format

The question decides the output. The same scene family can be answered with a point, a box, a heatmap, or a selection among described items.

This clip could not be loaded.

A Generalist Foundation Model for Gaze Target Detection

Unifying diverse gaze-centric tasks.

Bypassing external explicit inputs.

Gaze priors in the latent space.

Instruction-conditioned autoregressive generation.

Relative monocular depth

Source head grounding

Target spatial geometry

Disentangle queries into specialized groups.

A comprehensive resource for holistic gaze.

Target detection & description

Source detection & description

Spatial reasoning

Contact detection

Answerability reasoning

Exceeding VLMs in breadth, matching specialists in precision.

Geometric grounding is the primary gap.

Embodied pretraining does not automatically transfer.

Broad gains at a compact scale.

Five conventional gaze-target benchmarks

Deployment on a Unitree G1 robot.

Scenarios

Viewpoint

Output format

A Generalist Foundation Model
for Gaze Target Detection