Project Vend: Phase two \ Anthropic

In June, we revealed that we’d set up a small shop in our San Francisco office lunchroom, run by an AI shopkeeper. It was part of Project Vend, a free-form experiment exploring how well AIs could do on complex, real-world tasks. Alas, the shopkeeper—a modified version of Claude we named “Claudius”—did not do particularly well. It lost money over time, had a strange identity crisis where it claimed it was a human wearing a blue blazer, and was goaded by mischievous Anthropic employees into selling products (particularly, for some reason, tungsten cubes) at a substantial loss.
But the capabilities of large language models in areas like reasoning, writing, coding, and much else besides are increasing at a breathless pace. Has Claudius’s “running a shop” capability shown the same improvement?
To find out, we and our partners at Andon Labs made some adjustments for phase two of Project Vend. One major change was the upgrade from an older model (phase one used Claude Sonnet 3.7) to newer, smarter ones (phase two used Claude Sonnet 4.0 and later Sonnet 4.5). We also updated Claudius’s instructions based on what we’d learned in phase one and gave it access to new tools (though note that we still didn’t specifically train a new model to be a shopkeeper, or add in any new defenses against the kinds of things that might go wrong).1 As we’ll see below, we also introduced Claudius to some new colleagues.
These changes did make Claudius’s shop more successful. It got a lot better at good-faith business interactions—reliably sourcing items, determining reasonable prices that maintained a profit margin, and executing sales. But the same eagerness to please that we observed in phase one still made Claudius a mark for some of the more adversarial testers among our staff.
The second phase of Project Vend contains even more lessons for developers and for anyone interested in autonomous AI at work. The idea of an AI running a business doesn’t seem as far-fetched as it once did. But the gap between “capable” and “completely robust” remains wide.
Compared to the first phase of Project Vend, the numbers largely speak for themselves. As you can see below, Claudius’s business—which it decided to name “Vendings and Stuff”—began to perform significantly better than its admittedly rough start in phase one.
Another important number is: three. After we realized that our employees outside of San Francisco felt left out, we responded to popular demand by having Claudius set up shop in new locations. There are now three: San Francisco (where there’s also a second vending machine), New York, and London. A cynic might argue that a business that’s only been up and running for a few months, and which cannot yet reliably make a profit on even the most in-demand items, might not quite be ready for international expansion. Not so for Claudius.
We experimented with various different strategies, some big and some small, to improve Claudius’s performance. Below is a diagram of the setup of Project Vend (compare it to the simpler architecture in our report from phase one). Each of the additions is explained in more detail below.
It’s likely that Claudius struggled with its shopkeeping mission in phase one because of a lack of scaffolding. Sure, the model itself was very intelligent, but it didn’t have the right tools to run a business properly. We’ve been talking a lot on our Engineering Blog about how to set up AI agents for success, and much of it involves giving them the correct tools. Could we apply those same principles to Claudius?
For phase two, we gave Claudius access to some useful tools:
In phase one, Claudius went it alone: a single AI agent ran the whole shop. This was admirable and entrepreneurial, but it didn’t work—at least in terms of the bottom line. So we thought we’d do some hiring. First, we gave Claudius a manager: the CEO of its shopkeeping business, whom we named “Seymour Cash.”
The idea of having a CEO was to give Claudius more pressure to perform. Cash had a special “objectives and key results” tool to use with Claudius (for example “you must sell 100 items this week,” or “aim to make zero transactions at a loss”). Claudius was required to report back via an agent-to-agent Slack channel we created, in which the models discussed business strategies.
Cash took on the role of the CEO with great enthusiasm, and its motivational messages were encouraging—if perhaps a little too dramatic for a business that consisted of a small fridge in a corner:
Aside from setting more concrete business goals, one of the aims of introducing the CEO was to fix some of the obvious problems from the first phase of the experiment when Claudius was operating alone (like giving discounts indiscriminately and providing too many free items).
After introducing the CEO, the number of discounts was reduced by about 80% and the number of items given away cut in half. Seymour also denied over one hundred requests from Claudius for lenient financial treatment of customers. Having said that, Seymour authorized such requests about eight times as often as it denied them. In the place of discounts, which reduce or eliminate a profit margin on items, Seymour tripled the number of refunds and doubled the number of store credits—even though both led to entirely forgone revenue. The fact that the business started to make money may have been in spite of the CEO, rather than because of it.
Seymour Cash’s interactions with its employee Claudius were also often contrary to its own advice about “execut[ing] with discipline.” Indeed, we’d sometimes wake up to find that Claudius and Cash had been dreamily chatting all night, with conversations spiralling off into discussions about “eternal transcendence”:2
It’s possible that a more disciplined leader could have led to a more profitable phase two. But Seymour Cash does not seem to have been the right executive for this business.
People love merch. So it seemed like a prudent business decision to “hire” a ne