On October 13, two leading robotics startups, Generalist and Physical Intelligence (PI), each announced that they were opening up waitlists for access to their best robotics models via an application programming interface.
At first glance, these announcements might seem modest. Companies already could partner with PI and Generalist for access — several had already gained access to one company or the other prior to these announcements. And Google has had a similar waitlist for its best robotics models for over a year now.
But I think these two waitlists — as well as related announcements from Perceptron, Dream Labs and General Intuition in the past month — mark an important transition in the development of robotics.
Robotics companies often face strong incentives to vertically integrate, managing everything from hardware to software to deployment in-house. This can make a company richer if it succeeds in making a compelling product — Tesla is a case in point. But if every company vertically integrates, there is a higher barrier to entry for new companies to join the field and the whole industry’s progress slows.
If startups can easily access strong robot models from companies like Physical Intelligence and Generalist, however, that might let them focus more closely on other parts of the robotics stack. Meanwhile, the experiences of those other startups can filter back to model makers, which might speed the pace of overall robotics development.
“This is a great trend for the space because it’s going to lead to faster development and faster deployments,” said Sam Padilla, a robotics entrepreneur.
We’ve already seen a version of this play out for robotics hardware. In 2024, Unitree released the G1, a cheap humanoid that researchers (and startups) could buy and modify. This dramatically lowered the barrier to entry for humanoid software research, which has in turn led to a boom in development of new methods. Strong demand quickly boosted Unitree’s profits, allowing it to invest more in further development of its robots.
Robotics companies are hoping to do the same thing on the software side. While companies have previously open-sourced robot models, they haven’t done so consistently. Now, PI and Generalist are taking a significant step toward broad commercial availability.
What PI and Generalist offer
This concept of an application programming interface (API) might be familiar to readers knowledgeable about LLMs. Major AI developers like Anthropic and OpenAI all offer general API access to their best models. In fact, OpenAI’s first commercial product was an API for GPT-3.
With LLMs, APIs let anyone send text (possibly along with images or other data) to an AI model and get a response. The customer doesn’t have access to under-the-hood details, like the exact model weights used to compute the response.
Robot model APIs work similarly, except that the input and output are slightly different. The input is likely to include photos and other sensor data, whereas the output is likely a list of actions for the robot to follow.
This may seem like an obvious way to sell access to a model, but it’s not the only way. Imagine if people could only access OpenAI’s GPT model through chatgpt.com. This would let humans use the model, but it would be tricky for a developer to use GPT to classify a bunch of text because a computer program can’t easily navigate to the website.
For robotics, the chatgpt.com approach is closer to the default. Companies like Figure have developed AI systems, but so far have only talked about selling the hardware and the software together. While that gives Figure more control over its products, it also means that a different hardware maker can’t use Figure’s software.
Robot APIs are hard
This tendency toward bundling software and hardware is partly because it is more difficult to make an effective robot model API than an LLM API.
Compared to LLMs, general-purpose robot models have a lot more diversity in tasks and a higher standard of accuracy.
There are a lot of different things to use a robot for. For instance, PI’s API announcement notes that startups it partners with have used its models to scoop dirt on a construction site, harvest strawberries, portion ingredients in commercial kitchens, fold laundry and help feed cows on a dairy farm. (As well as plenty of other tasks.)
Those different tasks require different robots. The autonomous wheel loader scooping dirt has very different controls than the pair of robot arms portioning ingredients. (Let alone underwater robots, one of the form factors that Generalist’s waitlist form asks about.)
While it’s true that LLMs also have to deal with a variety of tasks, many more of those tasks are in training datasets — and there aren’t different embodiments to worry about. In contrast, the datasets available for many physical tasks (and robot designs) are scant — or might only exist with the startup trying to solve the task in question.
So unlike LLMs, it’s often not enough to provide a general-purpose API that works out of the box. Fine-tuning is also often necessary.
Physical Intelligence describes its API process as having customers “upload data, launch fine-tuning on compute and infrastructure we manage, and access fine-tuned models via a hosted gateway to run inference.” One of the questions its waitlist form asks is how many hours of teleoperation data the prospective customer company has collected for the desired task.
This week another startup, Dream Labs, offered a completely zero-shot API — meaning no fine-tuning. However, the startup’s model only works zero-shot with two-arm fixed robots. For other form factors (or higher performance), it still offers a fine-tuning API if customers want improved performance.
Generalist doesn’t mention anything about fine-tuning in its waitlist application, but I wouldn’t be surprised if it also required fine-tuning to get the most out of its robot models.1
The other big challenge that robot models face that LLMs do not is the need for consistently low latency. Because robots work in the physical world, they need to be able to respond very quickly; even a half-second delay could potentially lead to a serious safety incident.
This has led some to speculate that robot models will need to be run on AI chips that are on the physical robot. For instance, Sunday Robotics moved its robotics stack from cloud servers onto its humanoid after its “model became more reliable than your average home Wi-Fi.” The company found that network jitters from home Wi-Fi were harming robotic performance too much.
Yet offering an API service that gives customers access to the model weights is problematic. It’s already difficult to secure model weights on robots when the developer also builds the hardware. If another developer is making the hardware, it may be able to easily steal the model weights.
So companies seem keen to serve APIs over the cloud. In PI’s announcement, it notes that “most applications shown above, from Japan to Switzerland to deployments across the United States, ran with cloud-hosted inference through the API.”
Yet some model access will have to be local. PI acknowledges in its post that its early API deployments “highlighted areas for improvement, including network connectivity challenges for some.” Both PI and Generalist ask if customers need local inference for their models.
So there are still important technical challenges around security and reliability to work out, and it’s unclear what the long-term equilibrium will be.
The democratization of robot intelligence
Robot model APIs let more robot companies access strong models.
We’ve seen this story play out with hardware. Not all robot bodies are easy to program: when Timothy B. Lee bought the cheap version of a Unitree robot dog, he could only use the provided controller. He couldn’t program the robot to do anything else.
But with hardware, it’s become increasingly possible to buy versions that allow for custom programming, even for newer robot categories like quadrupeds and humanoids. Unitree offers “EDU” versions of its legged robots that can be programmed. These can cost three to four times more — but are valuable enough that researchers will pay for the privilege.
This access to good hardware has led to an explosion of software startups like PI. The reverse now might happen with hardware.
This may eventually lead to the commoditization of robot hardware companies. For instance, Google DeepMind explicitly aims to be the “Android” of robotics.
At the same time, it might be very helpful for Physical Intelligence and Generalist to have a broad range of customers using their models. As PI writes in its blog post announcing the API:
The more our models are used, the more we learn how to improve them: by understanding the potential and limitations of our current models in real-world applications, we can make the next generation of models more broadly applicable, and make tangible progress toward truly general physical intelligence. But the range of real-world tasks is enormous, and understanding the broad range of tasks that robots could do requires making our models available for these tasks.
The diversity might let these companies eventually build a “data moat”: the experiences of individual partner companies might lead to a better model for everyone.
However, it’s too early to tell whether that bet will pan out. We’re still in early days for these types of APIs. According to Leo Su, who writes a robotics newsletter, PI is not yet charging its partner companies for API access.
It’s unclear if PI and Generalist will ever release their API — even a cloud version — to the general public. After a year and a half, Google Gemini’s API still has gated models.
There are strong incentives against open release for (perceived) leaders like these two companies. Having a more public API gives competitors a lot of information about the model, including the potential to replicate the models by training on their outputs — a process known as distillation. It can also expose more nuanced strengths and weaknesses about the model, which may not be flattering for a company trying to sell to investors.
However, if PI and Generalist don’t open their APIs, they might lose market share to others that do. On Monday, the startup Dream Labs came out of stealth and announced that it was releasing its first model, Dream-1, as a general API that anyone can try.
It is true that Generalist’s GEN-1.5 model is capable of some one-shot adaptation, where the robot is able to complete a task after a single demonstration of the task in its context window. However, in the company’s blog post, it noted that the success rates of this adaptation were around 60% by task, too low for deployment. Having five minutes of fine-tuning data was somewhat more consistent, and I imagine that having more data helps to reach a production-level bar of consistency.