The Robotics Breakthrough Everyone Has Been Waiting For

The development of foundation language models marked a significant milestone, not just for their improved conversational abilities or math-solving skills, but for their capacity to generalize to entirely new tasks from context alone after sufficient pre-training. Language is an ideal medium for this due to its compressed and informative nature, allowing a few words to specify goals, describe objects, explain procedures, or provide task examples. This led to the context window becoming a temporary learning mechanism for language models. The next question was whether this behavior could emerge in other modalities. While image, video, and voice generation models later demonstrated similar in-context learning, its emergence in robotics was highly anticipated despite being a much harder problem. Robotics faces unique challenges: information is not neatly tokenized, robots must continuously reason about vision, position, contact, force, motion, and action consequences. Slight variations, like an object moving a centimeter or a grasp slipping, can significantly alter task execution. The data requirements are substantially higher than for language or video generation, and success is harder to define than predicting the next token. Consequently, many expected generalization in robotics to take much longer. However, in August 2026, General AI released Gen 1.5, showcasing impressive robotics results, particularly in generalization or in-context learning. This model suggests a future where robots can learn new behaviors by simply observing a demonstration, rather than requiring training from scratch. After extensive pre-training on physical interaction data, Generalist AI's robot exhibited behavior strikingly similar to in-context learning in large language models (LLMs). For robotics, in-context learning involves showing a robot a completely new task for a few seconds, inserting this demonstration into its context window, and then the robot immediately attempts the task without any weight changes. Gen 1.5 possesses about 30 seconds of video memory, alongside proprioceptive (body position, movement, force) and language inputs. A 3 to 12-second video demonstration can be inserted as a sensory-motor sequence, including what the robot saw and the actions taken (at 100 Hz). The model then infers and reproduces the physical behavior. Generalist AI tested Gen 1.5 across ten manipulation tasks, none of which were included in the training data. The robot successfully replicated tasks like twisting a lid off a jar, unzipping a pencil pouch, brushing a cube into a bowl, removing a vacuum pad, pouring bolts into a cup, and zipping a zipper. The "magic" of in-context learning is that these actions were never explicitly trained, meaning the robot performs them "on the fly." The model also demonstrates adaptability to unexpected situations. In an example where a robot knocked over a cup while attempting to put a marker in it, it dropped the marker, righted the cup, and then placed the marker inside. This shows an ability to recover and adjust. Furthermore, Gen 1.5's generalization extends beyond human demonstrations with robot claws. It can learn from demonstrations performed entirely in simulation, even though it was pre-trained with zero simulation data (no rendered footage, no simulated physics). The simulated trajectory is placed in the context window as a physical prompt, and the real robot performs the task in the real world. This is distinct from traditional sim-to-real transfer, where a policy is trained in simulation and then deployed. Here, the simulation is merely an example in the context window, and the model maps the simulated action to its own physical body, even accommodating changes in hand usage or object position/size. Pushing this further, the researchers demonstrated that Gen 1.5 can learn from bare-hand human demonstrations directly in front of its cameras. Despite the physical differences between a human hand and a robot gripper, the model can extract the abstract goal and translate it into actions for its own robotic body. This "next action prediction" capability is highly impressive, as it requires understanding the intent rather than exact motion replication. In-context learning experiments also showed that physical prompts can be composed. By placing two independently recorded demonstrations (e.g., unzipping a pouch and retrieving money from it) into Gen 1.5's context window, the robot seamlessly connected these actions into one coherent movement. Despite these impressive demonstrations, the initial success rate for in-context learning without gradient updates or task-specific fine-tuning across all ten tasks was 59% on average. However, fine-tuning significantly improves performance. Normally, adapting a robot policy to a new task requires tens of thousands of gradient updates. With Gen 1.5, Generalist AI collected only about 5 minutes of demonstrations (roughly 50 examples) per task and fine-tuned the model for just 10 gradient steps. This increased the average success rate from 59% to 83%. Some tasks saw massive improvements: sweeping with a brush went from 37% to 99%, opening a jar from 60% to 94.5%, and unzipping a pencil pouch from 55.5% to 86%. The researchers noted that this adaptation procedure was not heavily optimized, using pre-training hyperparameters. This suggests the model isn't learning from scratch but being "nudged" towards something it nearly already knows. Evidence for this is that after 10 gradient steps, the model's weights changed by less than 0.15% on held-out tasks, indicating minimal underlying model alteration. This fast adaptation emerged after over 8 months of Gen 1.5's pre-training, during which task adaptation requirements decreased from hundreds to tens to one, and eventually to zero gradient steps (i.e., in-context learning). This highlights the crucial role of extensive pre-training data. Another benefit of large-scale pre-training is physical generalization. After adapting Gen 1.5 to a task, it doesn't just replay the exact demonstration but can generalize to new objects, environments, grasps, hands, obstacles, and even entirely different physical strategies to achieve the same goal. For example, if fine-tuned to rotate a jar lid with one hand, it might spontaneously use its second hand and a different contact/motion strategy during some rollouts. Models fine-tuned for 10 gradient steps on 5 minutes of data for twisting off a jar lid can generalize to unseen cups and bottles, demonstrating an understanding of the goal beyond the specific demonstrated behavior. A striking example of physical generalization involved a sweeping task. After fine-tuning on 5 minutes of demonstrations using a brush to sweep a cube into a bowl, the researchers replaced the brush with various random objects, including a banana. The robot, tasked with sweeping the cube into the bowl, decided the banana could function as a makeshift sweeping tool. Even more remarkably, when given a dustpan, Gen 1.5 didn't force the same sweeping motion but changed strategies entirely: it scooped the cube onto the dustpan, lifted it, and dumped it into the bowl. According to researchers, neither the fine-tuning nor pre-training data (to their knowledge) contained a dustpan used in this manner for the task. This suggests the model generated a completely new contact sequence without explicit language instructions about the tool. Further emergent behaviors include "physical common sense." A model fine-tuned to put one block into one bowl sometimes went beyond the task, sorting multiple blocks by color or category. This wasn't taught or prompted but seemed to be a broader organizing behavior pulled from pre-training. Emergent improvisation was also observed. When manipulating Lego, if a brick got stuck on a gripper fingertip, Gen 1.5, instead of failing, used its other hand to remove the stuck Lego and then resumed the original task without supervision. This demonstrates an ability to recognize and address unexpected problems dynamically. In another instance, a model fine-tuned for one gradient step to put a block into a bowl, when faced with the bowl covered by paper (unseen in training data), understood the paper was an obstacle. It moved the paper aside, placed the block in the bowl, and sometimes even replaced the paper afterward. An additional fascinating observation shared by researchers was the robot's spontaneous interaction with an unknown object. When simply given a wallet (not in pre-training data) without any prompt, the robot fidgeted with it, flipping and closing it, demonstrating some form of object understanding. Later, when prompted to "take money out of the pouch" (with the wallet still being an unknown object), it proceeded directly to the task. This suggests it likely saw similar objects like money pouches during pre-training and generalized the concept. The progress in robotics, exemplified by Gen 1.5, appears to be accelerating faster than anticipated. While many robots perform complex tasks, they often lack true generalizability. Gen 1.5 represents a significant milestone, proving robots capable of in-context learning, and its results are captivating.