MSc thesis · ViCoS, University of Ljubljana2026

Visual Thought Tokens: Reading Perception out of a Robot’s VLA, and Feeding It Back

X-VLAFlow matchingDINOv2CeDiRNetToken groundingSO-ARM101LeRobot

Thesis ↗Token grounding report ↗Code ↗ViCoS ↗

Overview

Does a robot’s VLA already know where the cloth corner is, and just fail to use it? My thesis reads perception out of a frozen VLA with a small decoder, then feeds it back to the action head.

X-VLA (0.9B) on an SO-ARM101 arm, trained on my own LeRobot demos: 184 cube, 70 cloth-to-box and 50 folding. Supervised by prof. Danijel Skočaj and dr. Domen Tabernik at the ViCoS lab.

A decoder that reads the frozen backbone

Frozen X-VLA tokensVisual-thought decoderExpert’s view

Learned queries cross-attend to X-VLA’s internal tokens and learn to match an expert. X-VLA stays frozen. One decoder, three jobs:

DINOv2: rebuild the patch tokens of a DINOv2 ViT-B/14.
CeDiRNet: rebuild its cloth-corner direction maps.
Token grounding: point at the image tokens the instruction needs.

Visual-thought decoder: learned queries pass through self-attention, cross-attention over the projected frozen VLM tokens, and a feed-forward block, repeated L times, producing visual-thought tokens

What the backbone already knows

Both experts can be read back out. A ~5M-parameter decoder rebuilds the corner maps with a validation error of 0.015.

Input frame, real DINOv2 features and features decoded from X-VLA, shown as PCA colour maps for a cube scene and a cloth scene

Sine and cosine cloth-corner direction maps decoded from X-VLA, overlaid on the cloth scene

Token grounding: which tokens matter right now?

One visual-thought token scores each cell of X-VLA’s 7×7 image-token grid, and the best cells are refined to 14×14. Inspired by MolmoPoint’s pointing tokens.

Data: 4,069 image–instruction pairs, 1,524 of them negatives.
Precision first: a wrong highlight sends the robot to the wrong object. Best checkpoint: 88% precision at 60% recall.
Follows the task: the named cube lights up before the grasp, the box after.

Pick up black cube and place inside white box: before the grasp the selected token covers the black cube; after the grasp it covers the box

Feeding it back to the policy

Implicit guidance

Reconstruction loss added while fine-tuning. Inputs unchanged.

Explicit guidance

32×32 corner tokens into the action transformer, at layers 6/12/18 or appended to its input.

Fair comparison

Every guided model has a control trained for the same extra steps.

Guided X-VLA pipeline: guidance tokens extracted from the VLM features are fused with the native action-conditioning sequence before the self-attention action transformer

Results on the real robot

Thesis ↗

16 trials per model on each cloth task, 48 on cubes.

P(correct corner) · cloth-to-box, 16 trials each

Grasp score | correct corner

P(named cube) · 48 trials

Folding: implicit and layer-inserted guidance found the corner 16/16 times, but fold quality stayed low for every model (19–34%).
Cubes: the baseline’s misses were hesitations between two cubes. With guidance it committed, sometimes to the wrong one (5 of 48).

What I took away

The backbone knows more than the action head uses. A few million parameters can read it out.
Guidance picks the right target. Grasping it well is a separate problem.
Getting X-VLA running on the SO-ARM101 took work: end-effector actions, state normalisation, varied lighting and smoothed commands. Small camera shifts still hurt it.