We ran the test exactly the way you’re supposed to.

Same documents. Same prompt. Everyone runs it, we compare outputs, we find the variation, we tighten the prompt, we run it again. Each iteration, the outputs converge a little more. Add parameters. Add guardrails. Specify the formatting, the columns, the tabs, the template. Watch the variance shrink.

It felt rigorous. It felt like science. And it was measuring the wrong thing.

Here’s the trap most enterprise AI pilots are walking into right now: we test the artifact instead of the outcome. My team spent iterations debating how many line items the file should have, how many columns, which tabs. Every round of prompt tuning bought us a prettier file. None of it answered the only question that matters: does this make a project manager faster in real life?

That’s the reframe I keep bringing back to the room. Before you test the prompt, define the success criteria. Tie it to time savings, and be specific. If the goal is a work breakdown structure, the question isn’t “does the file look right.” It’s “how quickly does this produce a project plan a PM can take into a room and execute?” Then ask the harder question: what is the bare minimum output that gets us there? Everything beyond that is theater. It feels like progress. It isn’t.

There’s a deeper reason the variation-chasing fails, and it’s the nature of the tool. LLMs are not deterministic. There will always be variation in what they produce, and sanding that down past a point has vicious diminishing returns. The teams treating the model like a file generator will keep polishing prompts forever. The ones who win will treat it like a thought partner.

That opens the testing frontiers most teams haven’t gotten to yet. Start with varied inputs, because inputs change everything. A prompt run against a mature set of scoping documents, a project charter, and real requirements will produce something entirely different from the same prompt run against thin documentation. Output quality tracks input quality, drastically. Yet most testing holds the inputs constant and grades the prompt, which is backwards.

Then there’s the measurement that might matter most, and it’s the one almost nobody is tracking yet. When a PM gets an AI-generated plan, how fast do they get acquainted with the project? How quickly do they understand the structure, see which conversations need to happen, which bridges need crossing, and start managing against reality instead of a template? That comprehension dividend compounds over the entire life of the project. It’s an acceptance criterion. Nobody’s measuring it.

So here’s where I land on the maturity curve. Level one is education: what is AI, how does it work. Level two is prompt fundamentals: how to write a good prompt, how to engineer the interaction. Both necessary. But level three is where the value lives: strategic thought partnership. Not “give me the file faster” but “help me think through the work.” Most organizations are stuck optimizing level two and calling it transformation.

The question I keep coming back to, in every room: what are we actually measuring? Answer that first. Everything else is formatting.

This is the first post in The Maturity Curve, a series on enterprise AI adoption. More to come.