The question has shifted. Two years ago, every executive conversation about AI started with “are your developers using it?” Today it starts with “what did we get for the money?”
Most engineering organizations are not set up to answer the second one.
Recently I sat down with Kamal Syal, VP and Chief Enterprise Architect at Eaton, and Michael Tkachuk, Head of Operations at Cisco, to work through this. Both have been on this journey since the early days of GitHub Copilot and have moved well past adoption into something harder: building the structures that turn AI spend into demonstrable business value.
Adoption Was the Easy Part
When Cisco first launched AI tools, one of the primary success metrics was token consumption. The assumption: more usage meant more value. Teams optimized for burning tokens because that was the visible signal of engagement.
“We all know that large companies such as Amazon and many others have stopped using that metric because it’s totally misleading. How do you actually drive that towards an outcome and where do you start to see efficiency gains?”
— Michael Tkachuk, Head of Operations, Cisco
Kamal described the same starting point at Eaton. Early reporting centered on active Copilot users. That number grew but whether it was changing how software shipped stayed unanswered.
Both teams moved toward metrics with actual signal: productivity and velocity, PR cycle time and lead time, acceptance rates tied to shipped features, security and quality outcomes. As agentic adoption grows, Kamal added another layer worth tracking: in an end-to-end workflow, how much is agent-driven, and whether an agent is even the right tool for each step.
The framework underneath this is the inner/outer loop distinction. Inner loop is building the product. Outer loop is shipping it. Coding assistants have made the inner loop faster. The harder problem is connecting those gains to outer-loop outcomes such as shipped features, deployment frequency, change failure rate, time to recovery. These are the DORA measures organizations already track. AI needs to connect to that foundation and show whether the numbers moved.
Faster Code Creation Is Shifting the Burden Downstream
Opsera’s own benchmark data makes this concrete. AI consistently reduces time-to-PR by 48 to 58 percent but AI-generated PRs wait 4.6 times longer for review. The inner loop got faster, but the outer loop got slower. The work did not disappear; it moved downstream and accumulated there.
AI tools are designed for parallel execution, capable of operating with or without human supervision, often running agents and loops in the background. On the other hand, SDLC controls (the approvals, the review gates, the security checks) are designed for sequential, human-paced delivery. When an AI-speed inner loop runs into a human-speed outer loop without rethinking the handoff, the volume of work waiting for human review grows faster than the team can process it.
Kamal grounded this in the governance reality. Agentic tools now execute multi-file changes, architectural decisions, and deployments before any human reviews a step. That pace is what opens the governance gap behind them. “Guardrails are not friction to be eliminated in the name of velocity. They are the reason things do not break in production.”
When Amazon discussed the exposure of 12 terabytes of source code, the takeaway was not to slow AI adoption. It was that PR approvals and review steps cannot be bypassed in the process.
Every Query Has a Cost and Most Organizations Are Not Tracking It
When pipelines slow down at review, agents regenerate, loop, and retry. Each iteration runs at cost. Token spend compounds in exactly the places where governance is weakest.
Kamal traced a pattern that has appeared before. Twenty years ago, APIs arrived and organizations built them without fully accounting for data movement costs. Cloud followed the same arc. Everything migrated before teams worked out what should and should not live there. AI is running the same playbook.
“If you could just do a Google search, why would you want to run a Copilot query on the same thing? There is a cost behind it. When you introduce an agent or any AI component, you have to understand how frequently you’ll use it and whether there is a true value coming out on the other side.”
— Kamal Syal, VP and Chief Enterprise Architect, Eaton
Michael’s formative years in finance shape how he looks at AI spend. Unmonitored AI spend crowds out other budget lines. Cisco currently tracks token cost by LLM vendor. The next step Michael is pushing toward is tracking at the project and product level. How many tokens did it take to ship this feature? That turns spend into a learning instrument as investment in specific technologies becomes visible and comparable over time.
Kamal put the longer trajectory as AI FinOps, maturing the same way cloud FinOps did. Benefit realization is incremental. Year one looks different from year two as the foundation settles and automation expands. The organizations with cloud cost discipline already have a head start.
Context Loss Is Where Tokens Go Quietly
Token spend has a visible side. There is also a quieter drain most teams have not named yet: re-contexting. When an agent starts without sufficient context, it hallucinates, regenerates, and loops back through the problem. Every iteration carries a cost. The bill grows not because the team is doing too much, but because the agent keeps redoing work it should have gotten right the first time.
Kamal connected this to something familiar. A well-designed prompt is equivalent to a well-written requirements document. The quality of the input determines the quality of the output and how many passes it takes. Twenty years ago, poorly written requirements created rework in the sprint. With AI, the same dynamic shows up in token spend and regeneration loops.
Taking time to plan what you actually need before prompting anything is where human contribution matters most. Thinking, judgment, and strategy have not moved to the machine.
“The biggest bottleneck I’m seeing with AI is over-reliance on it. Someone pastes a call transcript into an AI tool, converts it to slides, and ships it straight to executives. You need a human in the loop to orchestrate the dialogue and make decisions above and beyond what you get from AI.”
— Michael Tkachuk, Head of Operations, Cisco
Reducing re-contexting costs comes down to a few things: spec-driven development, proper context carryover across sessions, and routing validation and testing tasks to lighter models rather than defaulting to frontier models for everything. These are the same cost and quality disciplines that have always applied to engineering. AI makes ignoring them more expensive, faster.
What Actually Changed at Eaton and Cisco
Both teams anchored AI measurement to DORA metrics rather than building a new framework. Michael’s analogy: when electric vehicles arrived, the industry did not invent a new speed metric. Miles per hour still worked. Deployment frequency, lead time, change failure rate, mean time to recovery: these are the baseline. If AI investment is not moving those numbers, that is the finding worth acting on.
Both started with a cohort of 100 to 500 developers rather than opening licenses to the full organization. A large rollout produces a token bill with no behavioral baseline to learn from. A smaller cohort over two to three months creates a feedback loop. Uage patterns surface, problem areas appear before they become expensive, and lessons carry forward before the next group joins.
Cisco gave developers tool choice rather than mandating a platform. Usage and feedback narrowed the field over time.
When token data looked unflattering, both shared it anyway. Hiding the data does not change the underlying behavior. It removes the one instrument available for improving it.
Kamal’s closing point: sometimes a problem that has persisted for ten years is solved by changing a process, with no AI component required. Asking whether technology is the right answer at all is not a constraint on ambition. It is what keeps deployment from becoming a spending exercise.
Final Thoughts
The most useful thing any engineering leader can do right now is pause and look honestly at where the last three years of AI investment actually landed. Most organizations have not done that yet. The ones that do will have a much clearer sense of what to scale and what to stop.
For full conversation and deep insights, watch the full webinar here:
