VACI: Visual Action-Consequence Inference for VLMs

Open Problem

All evaluated VLMs show significant performance drops when textual environment feedback is removed (Wang et al., VisGym 2026). This work demonstrates that the feedback gap is closable: Difference-Augmented Prompting (DAP) recovers 103.4% of text-feedback performance on validity accuracy by providing explicit visual difference maps as auxiliary input.

103.4%

Feedback Gap Recovery

DAP exceeds text feedback baseline

0.879

DAP Validity Accuracy

+0.228 vs Naive (0.651)

73.1%

Probe Accuracy

Bottleneck is in reasoning

800

VACI-Bench Transitions

4 envs x 200 each

Naive Baseline

Present pre/post frames directly to VLM. Recovers only 76.6% of text-feedback performance. Struggles with subtle pixel changes.

Difference-Augmented Prompting (DAP)

Compute explicit pixel difference map, apply noise suppression and morphological closing. Transforms comparison task into description task. Achieves 1.000 on Maze2D and Sliding Block.

Visual Chain-of-State (VCoS)

Decompose into 4 steps: State Description, Change Detection, Action Matching, Consequence Derivation. Matches DAP validity (0.879) with greater interpretability.

Method Comparison: All Metrics

Per-Environment Validity Accuracy

Feedback Gap Recovery Ratio

Per-Outcome Accuracy

Difficulty Analysis

Contrastive Probe Training

Main Results Table

Method	Validity Acc.	Outcome Acc.	Change Detection F1	Feedback Gap Ratio
Naive Baseline	0.651	0.579	0.710	0.766
DAP	0.879	0.879	0.918	1.034
VCoS+DAP	0.879	0.828	0.918	1.034
Text Feedback	0.850			1.000

Per-Environment Breakdown

Environment	Total	Success	Blocked	No Effect	Naive	DAP	VCoS+DAP
Maze 2D	200	107	93	0	0.580	1.000	1.000
Sliding Block	200	142	58	0	0.925	1.000	1.000
Matchstick	200	120	0	80	0.470	0.600	0.600
Maze 3D	200	191	9	0	0.630	0.915	0.915

Visual Action-Consequence Inference (VACI)