AVERT-VLN

Vision-and-language navigation

AVERT-VLN

Abstention-aware Visual Error Recovery and Training
for Vision-and-Language Navigation

Minrui Liu1,† Jingke Wang1,† Yuehao Huang1 Hao Su1 Jiajun Lü1 Yukai Ma1,‡ Yong Liu1,∗

1 Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou, China.

† Equal contribution ‡ Project Lead ∗ Corresponding author
Detect a deviation. Recover the task. Learn from the correction.

How it works

From recognizing errors to learning better decisions.

A shared vision-language Monitor connects a runtime recovery loop with an offline learning loop, without generating the navigation actions itself.

Loop 01 · Online recovery

Asynchronous Sidecar Monitoring

The Monitor reads the instruction, visual history, and current observation alongside ongoing navigation. An accepted Lost verdict cancels pending autonomous actions and requests a pixel goal or a local turn from a human. Navigation resumes after correction.

MonitorAbstainGuideResume

Loop 02 · Offline improvement

Trajectory-Anchored Preference Learning

Trace a deviation back to the nearest preceding Pass boundary. Under the same decision context, pair the original decision with a correction-derived preferred decision. Decision-focused DPO targets corrected direction-selection and visual-grounding tokens.

Locate anchorPair decisionsUpdate policy

See the two loops

AVERT-VLN in action.

01 Runtime recovery

From a Lost verdict to guidance and resumed navigation.

Runtime monitoring & recovery

Reference-guided replay, not live human intervention. Snapshot restoration is schematic, not a physical rollback; resumption alone does not establish recovery success.

02 Preference learning

Autonomous navigation before and after DPO, shown sequentially.

Finding the bathroom

Selected examples, with monitoring and human guidance disabled.

Citation

@misc{liu2026avertvlnabstentionawarevisualerror,
      title={AVERT-VLN: Abstention-aware Visual Error Recovery and Training for Vision-and-Language Navigation},
      author={Minrui Liu and Jingke Wang and Yuehao Huang and Hao Su and Jiajun Lv and Yukai Ma and Yong Liu},
      year={2026},
      eprint={2609.39579},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2609.39579},
}