Why I expect robot control to be hybrid

Astra can already produce end-effector (EEF) actions accurate enough for simple manipulation. In Robocurve’s YAM-arm experiments, it commands gripper poses and opening, with fresh camera images and robot state between moves. I see this as a striking expression of multimodal intelligence: understanding a scene, reasoning spatially, and turning an instruction into useful motion.

The same report records 19/20 successful block-into-bowl trials, but only 2/20 puzzle insertions, under a 20-call budget and a 25% speed cap. These results describe one system and setup; they do not isolate why insertion fails. But they make the gap between successful pick-and-place and reliable contact-rich execution concrete.

Phillip Isola’s Robot-Use Agents argues that improving agents such as Astra and Fable could spread robot capabilities much faster than bespoke development. He also acknowledges current command latency and reliability limitations. I share his excitement about that possibility. I am less convinced that a general-purpose LLM agent alone will supply all the control competence it requires.

The missing piece, in my view, is robust physical intelligence: understanding contact and dynamics well enough to regulate forces, adapt motion, and recover while an interaction is unfolding. Astra’s results demonstrate useful spatial action and feedback; they do not yet establish fast, precise control under changing contact conditions. That is the capability boundary I want to examine.

Robot policies approach this from the other direction. Vision-language-action models (VLAs) and world-action models (WAMs) learn execution from robot experience. π0.5 co-trains on robot experience and multimodal data to generate continuous actions and generalize to new environments; our Flex-π jointly models future visual representations and actions, including on precise gripper-repair tasks. I see these policies as a stronger foundation for low-level execution within the tasks and conditions they have learned.

Their remaining gap is the breadth of generalization and open-ended reasoning that makes general-purpose agents useful. This is a relative limitation: VLAs already transfer semantic knowledge, and WAMs can generalize beyond their training scenes. Yet those gains do not establish the ability to handle arbitrary requests, unfamiliar physical situations, or new strategies without further robot learning. Neither family has solved reliable general-purpose control.

That is why I expect a hybrid or hierarchical future: general reasoning and physical policies contributing complementary capabilities. The next step, to me, is learning both a physically capable low-level policy and an interface through which high-level reasoning can reliably steer it.

The hard part is often executing how to do it robustly under contact, uncertainty, and fast-changing dynamics.

The following illustrations focus on friction as a physical property and dynamic adaptation to disturbances.

1. Contact changes what an action can accomplish

Both grippers reach the ball. Whether they hold it depends on friction.

The goal and initial flight are identical. The difference appears when the robot has to exchange forces with the object. An accurate prediction of where to intercept it does not guarantee that the contacts can arrest its motion and retain it. The available friction limits the tangential forces the fingers can exert.

For an LLM agent, even choosing an accurate interception pose solves only part of the problem. The complete system needs actions that work for the actual contact conditions. That understanding could be explicit, learned from physical experience, or inferred during execution. It still has to affect what the robot does.

The failed run also shows why feedback alone is not a universal remedy. Here, the same feedback controller fails when friction is low enough. Robustness requires useful observations, an adequate response, and physically feasible contacts.

2. The world can invalidate a reasonable plan

The same nudge tests whether the robot can revise its motion in time.

Self-repair adds precision: the tool must align with a small socket and recover if a disturbance breaks contact.

Nothing about the instruction has changed. New information arrives while the task is unfolding, and the appropriate action changes with it. In manipulation, that information might be a visible slip, an unexpected force, or a tool touching the edge of a socket instead of seating inside it.

Astra already uses feedback between commands. The question here is whether the complete observation-to-action loop can respond quickly and precisely enough for the interaction in front of it. A good explanation of the error, delivered after the object has fallen, is different from a correction that arrives in time.

This is also how I would assess a Rubik’s Cube demonstration. A solved cube shows task completion, but the task name alone does not establish dexterity. I want to see the contact changes, object motion within the grasp, and recovery from errors. For me, the relevant capability lies in how the gripper manages the interaction, rather than the number of fingers.

Why I expect a hierarchy

A general agent brings broad knowledge and can revise a strategy. A physical policy brings experience of how actions unfold on a robot. I want a system that combines both: the agent can steer execution as its understanding changes, while the policy retains the ability to respond to contact without waiting for another round of deliberation.

Hi Robot demonstrates useful hierarchical instruction following with high-level reasoning and low-level actions. I see this as evidence that the division can be productive. In the near term, I expect an off-the-shelf agent such as Astra to provide the high-level reasoning. The learning problem is then to train the physical policy and its interface to work with that agent, without jointly training the agent itself.

Autonomous driving offers an interesting thought experiment. I see its comparatively constrained action interface as a useful test case, even though perception, interaction with other road users, and safety make the overall problem extremely difficult.

If general-purpose LLMs are sufficient for physical control, would we expect a future Tesla FSD to replace its specialized driving model with an LLM? Or would language-based reasoning become part of a system that retains dedicated perception, prediction, and action?

I mean this as a test of the broader claim, not a prediction about Tesla’s architecture or plans. Driving and dexterous manipulation have different constraints; neither outcome would settle robotics by itself.

I would change my view in response to reliable execution across unfamiliar materials, disturbances, and tasks, with control interfaces and failure cases made clear. Current demonstrations make me optimistic about agents using robots, while leaving the strongest claims about physical control open.

The next step: physical intelligence and steering

How do we train a low-level policy with enough physical intelligence for reliable execution, and an interface through which an off-the-shelf agent can steer it? These are coupled problems. The policy needs to learn how to handle contact, changing dynamics, and recovery. The interface needs to turn revised goals and constraints into changes in behavior without sacrificing that physical competence.

I do not think language or intermediate visual representations provide a good interface for this. We need a more effective way to connect high-level intent with the policy’s physical skills.

Train the physical policy and its interface to an off-the-shelf agent.

The policy should respond to new intent while retaining fast local feedback, and communicate what is feasible or where execution is failing. What representations, training data, and objectives make this possible remains open. I see this as a next question for robotics; this essay focuses on the capability gap that motivates it.