Gaze2HOI Contact Us

Gaze2HOI: Gaze-Aware Intention Disambiguation for Egocentric Hand-Object Interaction Forecasting

Chuanyang Ma1, Chenye Wang2, Yixuan Wang1, Huiling Zhou1, Xingqun Qi3, Muyi Sun1, Man Zhang1, Caifeng Shan4, Zhenan Sun5

1 Beijing University of Posts and Telecommunications2 The Hong Kong University of Science and Technology3 North China Electric Power University (Beijing)4 Nanjing University5 Institute of Automation, Chinese Academy of Sciences
Paper Draft arXiv Code & Model Dataset

Demo samples are available now; the full EgoAIM dataset will be released upon paper acceptance.

The idea

A future interaction begins
with an intention.

Similar observation histories can precede futures with different intentions, contact configurations, and trajectories. Gaze2HOI reduces ambiguity hierarchically: gaze helps infer intentions, contact grounding provides spatial guidance, and diffusion generates coordinated motion.

Figure 2 follows a U-shaped path from observations at the lower left, through gaze-aware intention inference at the upper left, geometric ambiguity and contact grounding across the top, to joint hand-object forecasting at the lower right. Original source elements appear progressively and remain together in the completed diagram.

Observed historyOne second of observations.
Input:Observed video frames V minus 29 through V zero
First observed RGB frame Intermediate observed RGB frame Last observed RGB frame
···
···
Challenge 1Semantic ambiguity
What to interact with?
Moka pot Phone Eraser Marker Cube Mug
Gaze cues
Gaze cues
Ego-RGBs
Temporal Gaze
Saliency Modulation
Temporal gaze saliency on the observed frame
Qwen vision-language model
Qwen-VL
L:<No intention>R:Reach and grasp,Moka pot, Handle
Per-hand intentions
Challenge 2Geometric ambiguity
Where to make contact?
Illustrative contact configuration 1 Illustrative contact configuration 2 Illustrative contact configuration 3 Illustrative contact configuration 4
Intention Text
HOI History
Point Cloud
Anticipatory
Contact Grounding
Contact likelihood concentrated near the moka pot handle
Relative contact likelihood
Low
High
Gaze2HOI: Hierarchical Intention Disambiguation
Output:Predicted future states one through thirty
Gaze2HOI · Figure 2

Scroll to explore the complete diagram.
01 / Intention

What to do next?

Infer structured intentions for each hand from observations and gaze.

02 / Contact

Where to make contact?

Localize likely contact regions using object geometry and the interaction history.

03 / Motion

How will it unfold?

Jointly forecast 3D articulated motion of both hands and 6-DoF object trajectories.

EGOAIM BENCHMARK

One benchmark.
Four interaction phases.

Fine-grained future intentions paired with
continuous 3D hand-object motion.

7,429Forecasting windows
33Object instances
9,038Non-null per-hand intentions
30 + 30Observed + future frames at 30 fps
EgoAIM dataset overview and structured hand-object intention annotations.
Built on HOT3D-Aria recordings, EgoAIM pairs intention annotations with continuous hand-object motion. Training, validation, and test splits contain 5,132, 976, and 1,321 windows.
01

Establish Contact

Reach toward the intended object and part.

02

Move

Transport the object toward a target.

03

Manipulate

Articulate the hand for the ongoing action.

04

Put Down

Place the object on a supporting surface.

Gaze2HOI framework

From gaze to interaction.

Semantic and geometric disambiguation
for coordinated motion forecasting.

Gaze2HOI: Gaze-Aware Intention Disambiguation and HOI Forecasting. Complete framework overview above the original three architecture panels: gaze-aware vision-language model, anticipatory contact grounding, and conditional motion diffusion.
Gaze-Aware Vision-Language Model

Reading the gaze.

Temporal Gaze Saliency Modulation (TGSM) integrates current and historical gaze into VLM visual features to infer the structured future intention for each hand.

Anticipatory contact grounding

Finding the contact.

Anticipatory Contact Grounding (ACG) combines contact priors with observed interactions and hand trajectories to predict contact fields, with support and placement fields for Put Down.

Conditional motion diffusion

Forecasting the motion.

A diffusion model jointly generates hand and object motion under semantic and geometric conditions, while retaining the observed history.

Experimental results

Better intentions.
More accurate interactions.

Evaluation on the EgoAIM test set.
See the paper for the complete protocol.

Intention inference82.29%

Sample EM@100px

+9.62 percentage points over No Gaze
Hand motion4.16cm

Hand average displacement error

Valid future wrist positions
Establish contact92.43%

Reach success rate

On the Establish Contact subset

Qualitative Results

Select a sample row to play its observation and forecast. The selected row plays all methods together, then advances to the next sample. Use the arrow keys to change samples.

Observations GT ObjectForesight CAHMP Transformer Ours
Sample 1: Observations
Sample 1: GT
Sample 1: ObjectForesight
Sample 1: CAHMP
Sample 1: Transformer
Sample 1: Ours
Sample 2: Observations
Sample 2: GT
Sample 2: ObjectForesight
Sample 2: CAHMP
Sample 2: Transformer
Sample 2: Ours
Sample 3: Observations
Sample 3: GT
Sample 3: ObjectForesight
Sample 3: CAHMP
Sample 3: Transformer
Sample 3: Ours
Sample 4: Observations
Sample 4: GT
Sample 4: ObjectForesight
Sample 4: CAHMP
Sample 4: Transformer
Sample 4: Ours

The visualized results show raw model outputs, without additional temporal smoothing or trajectory refinement.

Failure Cases

Use the three dots or arrow keys to select a case. Observations play first, both intentions are highlighted, and ground-truth and predicted motions then play together. Prediction is in the middle and ground truth is on the right, with each intention above its motion.

Observations

Egocentric observations of both hands holding a wooden bowl.
Prediction
Ground Truth

L:Grasp, Wooden bowl, Rim
Place, Dark wooden square table,
[567, 656]

R:Grasp, Wooden bowl, Rim
Place, Dark wooden square table,
[567, 656]

L:Grasp, Wooden bowl, Rim
Shake to mix

R:Grasp, Wooden bowl, Rim
Shake to mix

Predicted future hand-object motion.
Ground-truth future hand-object motion.

These examples illustrate three failure modes: purpose confusion, contact-part confusion, and hand assignment errors. Such errors can lead to forecasts that deviate from the intended interaction, as when placement is predicted instead of shaking. At the same time, a different contact part or active hand may still offer a viable way to complete the same task. Disagreement with a single recorded future therefore does not necessarily imply failure to capture the underlying interaction goal.

The effect of intention

Structured intentions change
the forecast.

Predicted intentions improve motion accuracy and interaction success over intention-free conditioning, bringing contact establishment close to oracle performance. A substantial gap remains in precise placement, highlighting the importance of accurate intentions and spatial goals.

Motion forecasting across intention conditions
IntentionH-ADE (cm) ↓O-ADE (cm) ↓Reach-SR (%) ↑Place-Hit (%) ↑
None6.491.0556.9118.94
Predicted4.160.7292.4344.93
Oracle3.650.5596.2295.59

Reference

Build on this work.

Gaze2HOI, arXiv:2610.27877, 2026.

Read the paper
BibTeX
@article{ma2026gaze2hoi,
    title   = {{Gaze2HOI}: Gaze-Aware Intention Disambiguation for Egocentric Hand-Object Interaction Forecasting},
    author  = {Ma, Chuanyang and Wang, Chenye and Wang, Yixuan and Zhou, Huiling and Qi, Xingqun and Sun, Muyi and Zhang, Man and Shan, Caifeng and Sun, Zhenan},
    journal = {arXiv preprint arXiv:2610.27877},
    year    = {2026}
}