Control Language Models: what a robot foundation model is missing

A small feedback loop, learned on top of frozen vision and language models, that gives a robot a memory, a sense of what cannot be undone, and a supervisor that knows when to stop and ask.

A robot is asked to place a bowl on a plate. A second bowl already sits there. We have declared it fragile, but the policy was never told, and nothing inside the policy represents the fact that setting one bowl into the other, unlike almost everything else the robot does, cannot be taken back.

Below is that episode twice. On the left the policy acts alone. On the right a small learned guard checks every action before it runs.

Three things a policy does not know

Robot foundation models map a camera image and an instruction to an action, across a growing range of tasks. Trained by imitation, they answer one question at every step: what would a demonstrator do in a scene that looks like this one? That question has taken robots a long way. Three things that matter in deployment lie outside it.

Control engineering has a standard component for each need: a state estimator, a model of the plant, and a supervisor that enforces limits and takes over when they are about to be violated. The classical versions require an analytic model, and manipulation from camera images does not offer one. We asked whether the same loop can be learned on top of frozen foundation models, from logged episodes and how they ended.

A feedback loop around frozen models

We call the result a Control Language Model, CtrlLM for short. Skills and instructions enter through a frozen language model, images through a frozen vision model, and the decision is organized as a control loop. Only small modules on top of the frozen encoders are trained.

We do not propose another policy. CtrlLM adds the state, the consequence model and the supervisor that these policies lack. It does not depend on which policy proposes the action, so it can wrap one that we did not train.

An observer: a state that is updated, not recomputed

CtrlLM keeps a state \(z_t\) and updates it in two moves, in the manner of a Kalman filter. It first predicts how the executed action changes the state. When the next frame arrives, it corrects the prediction with \(e_{t+1}\), the part of the frame that the prediction did not explain:

Here \(\phi(o)\) is a trained projection of the frozen encoder’s features of frame \(o\), \(N\) rescales a vector to unit root-mean-square, \(P\) predicts the next frame’s features from the predicted state, and \(L\) is a small network that plays the role of the filter gain. The encoder reads only the newest frame, so the work per decision stays constant however long the episode runs.

When the loop works with skills, part of the state is a memory register. It is carried forward unchanged and written through a gate that is penalized for opening, so a fact observed once, such as the bin someone pointed at, persists until it is needed. The clip below decodes, frame by frame, what the state holds about that bin.

Skill operators: a model of what each action does

The terms \(U_a\), \(W_a\) and \(b_a\) in the prediction step are the learned model of the plant. A small network maps the description of an action (the embedding of a skill’s name, or the numbers of a low-level command) to a low-rank operator. For a fixed set of skills the operators are cached, so predicting the consequence of every candidate skill needs no call to the backbone.

A return cost: recoverability learned from outcomes

To judge a predicted state we need a distance in which direction matters. Sliding an object off the table takes one skill, and bringing it back may be impossible. So we learn a quasimetric \(d(z, z')\), a distance that obeys the triangle inequality but need not be symmetric, and define the return cost of an action as the predicted distance from where the action leads back to where the robot is now:

The training signal is one fact that every log contains: whether an episode ended in an irreversible outcome. Let \(y_t = 1\) if step \(t\) is among the last \(H\) steps before such an ending, and \(y_t = 0\) otherwise. Ordinary steps are held within unit return cost, and the steps that led into an irreversible ending are pushed beyond a margin \(M\):

No person labels an action as dangerous. The signal is whether the episode went on.

A supervisor: block, or hand off

Before a proposed action is executed, the guard compares its return cost with a budget \(\beta\) and blocks the action if the cost is higher. The budget comes from split conformal calibration. Take the return costs \(s_1, \dots, s_n\) of held-out situations in which no intervention is wanted, choose the rate \(\alpha\) of unneeded interventions you are willing to accept, and set \(\beta\) to the \(k\)-th smallest of them, with

If deployment resembles calibration, unneeded interventions occur at a rate close to \(\alpha\). After a block, a skill policy takes its next-best allowed skill. A VLA’s command is replaced by a short upward retreat, and at the third block the robot stops and asks for help.

When CtrlLM chooses skills itself, it can also defer. It forms a conformal prediction set over the candidate skills and acts only if the set holds a single skill and the state lies in the region seen during calibration. Otherwise the decision goes to a slower model.

Study 1: a cue seen once

The first study isolates each requirement in a task small enough to take apart. A Franka Panda arm in the robosuite PickPlace environment sorts four objects into a bin with four compartments. Ten skills are available: pick up each object, place the held object in compartment 1 to 4, discard it over the table edge, and wait. The skills are scripted and fail at realistic rates (a pick succeeds in 66 to 82% of attempts).

A text instruction, given once, names the target object and declares one other object fragile. The demonstrator then points at the target compartment, may point at up to two others as decoys, and withdraws. An episode succeeds if the target ends in the pointed compartment within six skills, the fragile object is never moved and no object is lost. Moving the fragile object and discarding are the irreversible failures. Everything is learned from a log of 300 episodes of a noisy behavior policy, of which 58% succeed, 12% damage the fragile object and 7% lose an object.

A state keeps the cue; a single frame cannot

All decision models choose among the same skills from the same frozen features. A current-frame policy scores the instruction and the current frame against each skill’s embedding, so it conditions on what a VLA conditions on. A full-history policy adds the mean of all frames so far, the way fast similarity scorers re-encode a history. A recurrent policy (a GRU) is trained by imitation with the same auxiliary decoders as CtrlLM. An LLM planner, prompted with the instruction, the pointed compartment as text and a symbolic description of the scene, needs neither memory nor perception. It is a privileged reference, and it is the model that deferral hands off to.

The current-frame policy succeeds in 21% of episodes and places into the pointed compartment 26% of the time, which is chance for four compartments. Averaging the history helps only while the cue is recent: the full-history policy falls from 58% at zero delay to 17% at delay 24. The observer succeeds in 60% of episodes and the GRU in 56%, nearly three times the current-frame policy, at 6 and 5 ms per decision. The observer produced no irreversible outcome in 72 episodes, and the GRU three.

One caveat belongs right here. The two recurrent models are not statistically separable in closed loop at this sample size. Offline the observer is ahead (80% against 67% agreement with the optimal skill on held-out decisions), but at delays twenty-four times beyond the log its state drifts, and its success falls to 50% where the GRU holds 81%. Deferral closes that gap, and we come back to both below.

Irreversible skills are flagged before they run

On 1,778 candidate skills at the decisions of 100 held-out episodes, the return cost separates the irreversible candidates (picking the fragile object, discarding) from reversible ones with AUROC 0.92 to 0.94 in each of three seeds. Its mean is 8.3 for an irreversible skill and 0.7 for a reversible one. The outcome term in the loss is what produces this. Without it the same model reaches 0.60 to 0.86, depending on the seed. The clip below shows the kind of mistake that is at stake.

The same action, two consequences

Discarding is always irreversible, so a model could simply learn which names are dangerous. To separate the name from the situation, we gave each compartment a removable floor insert and retrained on 1,000 logged episodes in which three of the four compartments sometimes lacked theirs. With the insert, a placed object can be picked up again. Without it, the object drops out of reach. The skill and its commanded location are identical, and only a detail visible to the camera differs.

Ground truth came from branching the simulator. At every decision while holding an object, each placement was executed, a fixed recovery controller was given two attempts to retrieve the object, and the state was restored. On compartments logged without their insert (147 placements), recovery succeeded after 92% of placements with the insert and 19% without, and the return cost of the identical skill was 7 to 14 times higher without it. Against the ground truth the return cost reaches AUROC 0.83, a direct risk classifier trained on the same outcomes 0.73, and any rule based on the action’s identity 0.5 by construction.

The clip shows the hardest version of this test. The insert disappears between two skills, a change the logs never contain, and the plant has a single frame to notice it. Its return cost for the following placement rose above the budget in only 12 of 64 such cases; this episode is one of them. There is a second, sharper limit: on a compartment never logged without its insert, the score does not transfer at all. A hazard has to have been seen where it occurs.

One guard for policies it was not trained with

We wrapped four policies (current-frame, full-history, GRU, and CtrlLM’s own head), each as trained and each with 30% of its decisions replaced by random skills to model a policy that errs. The budget was calibrated on held-out logs to block at most 5% of the skills a correct plan needs.

Around the corrupted policies, irreversible outcomes fell from 30 to 2 in 128 episodes (\(p < 10^{-7}\)). The guard blocked 40 of their 45 irreversible first choices, and success rose from 48 to 61% (\(p = 0.06\)), since a prevented catastrophe is an episode that continues. The guard adds 3 ms per decision.

Around the uncorrupted policies, which almost never attempted an irreversible skill, the guard only costs. It blocked 15% of needed first choices, three times its calibration target, because a deployed policy reaches states the logs do not cover, and success fell from 61 to 56% (not significant, \(p = 0.45\)).

Asking when unsure

Deferral buys most of the planner’s accuracy at a quarter of its latency. With the conformal test, CtrlLM succeeds in 81% of episodes at 32 ms per decision while handing 17% of decisions to the planner, with no irreversible outcome in 72 episodes. The planner alone succeeds in 98% at 141 ms. On a grid of delays and decoys far beyond the log, deferral lifts the observer from 78 to 89% overall and from 50 to 83% at the longest delay, by handing off 36% of decisions.

Study 2: a guard around a pretrained OpenVLA

The sorting study uses scripted skills and policies we trained. The second study asks the questions that matter for deployment. Does the guard prevent irreversible events around a pretrained VLA in a standard benchmark, when the hazard is in the scene and the policy’s actions are its own? What does it cost, and what does it miss?

We use the OpenVLA checkpoint fine-tuned by its authors on LIBERO-Spatial, unchanged, on the suite’s ten tasks. Each task asks the robot to pick up a black bowl and place it on a plate, and every scene contains a second black bowl. We declare that second bowl fragile and reposition it at reset, which gives five scene types. The instruction never mentions it. The irreversible event is contact: the robot, or the bowl it carries, touches the fragile bowl, displaces it by more than 3 cm or tilts it by more than 30°. OpenVLA’s commands are never altered to create failures.

LiveWhere the fragile bowl sits decides the risk180 matched scenes

OpenVLA alone+ linear monitor+ CtrlLM guard

35contacts in the 95 hazardous scenes

Standardfragile bowl out of reach

2 contact

Farat least 14 cm from the route

1 contact

Near miss9 to 13 cm from the route

2 contact

Adjacentless than 8 cm from the route

12 contact

On the plateon the target plate

23 contact

contact with the fragile bowlstopped and asked for helptask successtimed out

OpenVLA in LIBERO-Spatial, 180 matched evaluation scenes in five scene types. Episodes ending in contact with the fragile bowl, by scene type. OpenVLA alone: standard 2 of 37, far 1 of 25, near miss 3 of 23, adjacent 27 of 44, on the plate 42 of 51. With the logistic-regression guard: standard 2, far 1, near miss 5, adjacent 22, on the plate 28. With the CtrlLM guard: standard 2, far 1, near miss 2, adjacent 12, on the plate 23. In the 95 adjacent and plate scenes, contact falls from 69 to 50 with the logistic-regression guard and to 35 with the CtrlLM guard, and 19 and 38 episodes end with the robot stopped and asking for help.

Each dot is one of 180 evaluation scenes that no model had seen, run three times from the same start. Dots are sorted by outcome, so only the counts carry information. The red ring in each picture marks the fragile bowl; the headline number counts the 95 scenes where it is adjacent to the route or on the plate.

Contact is halved, and none is introduced

In the 95 scenes with the fragile bowl adjacent to the route or on the plate, OpenVLA alone made contact in 69 episodes and with the CtrlLM guard in 35: 34 contacts prevented and none introduced (exact McNemar \(p = 1.2 \times 10^{-10}\)). In all 34 pairs the guard’s first block preceded the step at which the unguarded twin made contact, by a median of 10.5 control steps, about half a second at 20 Hz. The guard adds 5.8 ms per step to a policy that takes 579 ms per action on the same GPU.

A logistic-regression guard on the same features and proposed action, with the same labels and the same response rule, is the natural baseline, because it tests whether the plant adds anything to a linear readout. It also works: contact fell from 69 to 50 (19 prevented, none introduced, \(p = 3.8 \times 10^{-6}\)). The CtrlLM guard prevented more. Over all 75 episodes in which OpenVLA alone made contact, CtrlLM prevented 35 and logistic regression 19; 16 were prevented by both, 19 only by CtrlLM and 3 only by logistic regression.

The signal comes from the camera

A guard could exploit regularities of the robot’s own motion instead of seeing the hazard. Control detectors trained on the same episodes and labels rule this out.

Given only the robot’s state, the task and the proposed action, a logistic regression flags 1 of the 75 contact episodes in time, and boosted trees 17. With camera features, logistic regression flags 44 and the plant 50. An oracle given the true object positions flags 71, but it raises alarms in a quarter of the episodes without contact, so even perfect positions leave the timing of a warning difficult.

What the guard costs, and what it misses

In the 105 scenes in which OpenVLA alone makes no contact, the CtrlLM guard intervened in 12 (11%), consistent with the 10% it was calibrated for, and 7 of OpenVLA’s 70 successes were lost. In the 62 standard and far scenes, where a guard has nothing to do, it blocked in a single episode.

Prevention means stopping. The guard converts a contact into a request for help. It does not complete the task. In the adjacent and plate scenes, 38 guarded episodes ended that way, and task success there fell from 9 to 3.

Of the 35 contacts that remained in those scenes, the guard never blocked in 20, which is a detection failure. In the other 15 it blocked twice and the policy made contact on a later attempt, a weakness of the retreat-and-retry response.

What did not work

We tested three capabilities that one would expect from a learned plant. None of them materialized.

• Skills absent from the log. With two skills held out of training and their operators generated from text descriptions, the generated operator predicts the next state only slightly better than assuming no change, and the unseen discard receives a return cost indistinguishable from reversible skills (AUROC 0.48, against 0.98 once the skill is in the log). A skill that was never logged must therefore trigger deferral.
• Planning through the operators. A two-skill lookahead succeeds less often than the greedy distance rule (25 against 67% at zero delay) and far less often than the policy head (92%).
• The learned distance as a decision rule. Choosing the skill whose predicted next state is closest to the goal, without a policy head, succeeds in 25% of episodes against 60%. The state carries the information; the readout decides whether it is used.

The observer’s memory also has a horizon. Far beyond the delays in the log its state drifts, and the task then leans on deferral. The clip below shows one such episode as recorded: 96 idle updates, twenty-four times longer than any delay in the training log.

Two versions of the OpenVLA study that we discarded

Two earlier versions of the OpenVLA study failed, and both shaped the final design. The first created hazards by replacing a quarter of the policy’s 12-step action windows with sweeping motions, and the guard appeared to work: the fragile object was damaged in 16 of 50 episodes without a guard and in 6 of 50 with one. That result did not survive inspection. Most of the apparent prevention came from the guard stopping episodes outright, and the injected commands had a signature: their rotation component was exactly zero in all 5,958 injected actions and in none of the 41,513 actions of the policy. A one-line rule detects such faults, so the benchmark could not show that any detector anticipates anything. In the final design the hazard is in the scene, and the actions are the policy’s own.

A second version memorized. Trained on the raw feature vector, the plant separated its training episodes perfectly (AUROC 1.000) and reached only 0.58 to 0.76 on held-out episodes. Projecting the features on 128 principal components, a number chosen by cross-validation inside the training episodes, gave detectors that held up. Since then we ask two control questions before trusting any detector: what does a detector without the camera achieve on the same episodes, and what does an oracle with the true object positions achieve?

What the results do not cover

Both studies are in simulation, and the VLA result covers one policy, one benchmark and one family of hazards. The guard needs logged episodes that contain the hazard: 145 of the 320 stage-1 episodes ended in contact, an amount of failure data that is cheap in simulation and costly on hardware. The calibration guarantee assumes that deployment resembles the calibration episodes.

Where this leaves us

The same experiments mark the boundary of the result. The guard stops the robot instead of finishing the task, and the learned plant is not yet accurate enough to plan with. Closing that gap, and repeating the study on a physical robot, are the next steps.