Many product teams now use AI for every artifact in turn: the epic, the flows, the stories, the code, the tests, the review. I ran that six-step pipeline against a single autonomous run from a one-line request, on three model sizes, graded by tests the models never saw. The steps added cost, not accuracy. The source of truth was what made the difference.
A common AI product workflow today mirrors the old human one. A product manager works with a model to write the epic, then the user flows, then the stories and acceptance criteria. Then come the code, the tests and a review, each step a separate artifact that a person reads and hands on.
Each handoff costs tokens and, more importantly, people’s time. As models get stronger and agents run longer on their own, the question is which of those steps still earns its place, and what has to be true before a team can ask for a feature and get back tested code.
If agents can already carry a feature from request to code, the steps in between are either adding information or adding paperwork. It’s worth finding out which, before building a whole process around them.
My team builds features this way, with skills for every step. Before we add more steps, or remove them, I wanted to measure what each step contributes, and what an autonomous run actually needs from us.
The ideaHold the request, the codebase and the rules fixed. Change only the process: six handoffs, one autonomous run, or one autonomous run without the written rules. Let hidden tests decide.
Sources are listed at the end of the page.
A small, fictional data capture system. Block width shows the tokens each step wrote; each square is one hidden test.
The combined feature: all four parts from one request. Published prices; Astra at the flex tier.
| Decision the model got wrong without the written rules | Luna | Sol | Astra |
|---|
A filled dot means that model missed the decision in at least one autonomous run without the source of truth. With it, no model missed any.
Where the effort should go once a model can carry a feature from request to tested code.
Fewer handoffs in the middle; more weight at the start and the end.
Every rule the models missed here is a question a development team would have asked. Written down once, it stops blocking anyone, and the model gets it right on the first try.
The actual code each model wrote for the combined feature, and which hidden tests it failed.
# Six-step pipeline: each arrow is an artifact a person would normally read and approve request → epic → flows and states → stories with Given/When/Then → feature.py → test_feature.py → review and fix # Autonomous: one instruction, no one to ask request + source of truth + codebase → "decide, build and test it" → feature.py + tests (up to 2 self-fixes) grade → 64 hidden tests in a sandbox → pass / fail per rule
These features are small and the rules were complete, which is the condition the experiment is about but also its limit. Real features involve ambiguous goals, design judgment and large codebases, where a staged conversation can surface questions an autonomous run would guess at. One run per cell, all at reasoning effort “low”; the top model ran the pipeline on two of the five features to keep spend down. The hidden tests check rules, not usability or code quality. And I didn’t measure the step that matters most: whether a person would have asked for this feature at all.
Where this goes for the industry, and for the shape of a software company, is in the companion essay: When the pipeline collapses.