When AI Usage Becomes a KPI, Waste Becomes the Outcome

There is a difference between measuring AI adoption and rewarding AI usage.

The first tells you whether people are using a new capability.

The second can quietly change how people work.

An organization starts with a reasonable question:

Are our teams actually using the AI tools we invested in?

So it adds a few numbers to a dashboard:

• active AI users

• prompts submitted

• tokens consumed

• agent runs

• generated code

• AI-assisted tasks

• percentage of employees using AI

At first, these metrics are useful. They reveal adoption patterns, identify teams that may need support, and show whether the technology is being incorporated into real workflows.

The problem begins when those same numbers become targets.

When leaders expect adoption to rise every month, teams learn what success looks like on the dashboard.

And then something predictable happens:

People optimize for the dashboard.

Not necessarily for the business outcome.

That is where AI measurement becomes an incentive-design problem.

The Problem Is Not Measuring AI Usage

Usage metrics are not inherently bad.

Google’s guidance on generative AI measurement explicitly includes usage and adoption metrics as part of understanding how people interact with AI systems. But it also separates those signals from system performance and business impact—the areas that ultimately determine whether an AI investment is producing useful results.

That distinction matters.

Knowing that 80% of an engineering team used an AI assistant this month tells you something.

It does not tell you:

• whether better software shipped,

• whether review time improved,

• whether rework decreased,

• whether incidents declined,

• whether cost per completed task improved,

• whether customers received more value.

Usage is evidence of activity.

It is not evidence of value.

The same distinction applies to agentic AI. Microsoft’s current AI agent planning guidance recommends connecting agent initiatives to concrete business outcomes such as lower cost, faster workflows, better decisions, or improved customer experiences rather than treating deployment itself as the objective.

The measurement problem therefore starts with a simple question:

Are we observing this metric, or are we using it to define success?

Those are very different things.

Goodhart’s Law Has Entered the AI Dashboard

A useful way to understand this problem is through Goodhart’s Law.

The principle is usually summarized as:

When a measure becomes a target, it becomes less useful as a measure.

The idea originated in economic policy, but later research has shown that the same pattern appears whenever people or systems are strongly optimized against proxy metrics. Researchers studying variants of Goodhart’s Law describe several ways optimization can make a previously useful metric ineffective or even harmful.

Enterprise AI creates almost ideal conditions for this failure.

Why?

Because AI systems generate an enormous amount of measurable activity.

We can count:

• tokens

• conversations

• API calls

• generated files

• accepted suggestions

• agent sessions

• workflow executions

• model requests

• tool calls

• code completions

That abundance of telemetry makes it tempting to equate measurable activity with progress.

But consider what happens when each metric becomes a target.

Target: More prompts

People use AI for tasks where a simpler tool—or no AI at all—would have been better.

Target: More tokens consumed

Long prompts, oversized context windows, and unnecessarily expensive models can look like healthy adoption.

Target: More AI-generated code

Teams can generate more code while increasing review burden, maintenance cost, and technical debt.

Target: More agent runs

Automations may execute frequently without reducing meaningful work.

Target: Higher AI adoption

Employees may feel pressure to demonstrate AI usage even when the task does not benefit from it.

The organization intended to measure adoption.

Instead, it created an incentive to manufacture adoption.

The Most Dangerous AI Metric May Be the One That Looks Successful

Bad metrics do not always produce obviously bad dashboards.

Sometimes they produce excellent dashboards.

Imagine an executive report showing:

AI adoption: +38%

Token usage: +62%

Agent executions: +44%

AI-assisted engineering tasks: +51%

Everything is moving up and to the right.

But underneath those numbers:

• inference spend doubled,

• review queues grew,

• teams switched simple tasks to expensive frontier models,

• agent workflows repeated unnecessary context retrieval,

• generated work created more cleanup,

• nobody can attribute the extra spend to a business outcome.

The dashboard still looks successful because the organization selected metrics that were almost guaranteed to increase as usage increased.

This is why AI programs need to separate three layers of measurement:

1. Adoption

Are people using the capability?

2. Operational efficiency

Is the capability performing economically and reliably?

3. Business impact

Is it improving an outcome the organization actually cares about?

Google uses a similar distinction in its current guidance, separating model quality, system quality, user engagement, operational efficiency, and financial or business impact rather than relying on a single adoption number.

AI adoption can be successful at Layer 1 and disappointing at Layers 2 and 3.

That is not a contradiction.

It means adoption worked.

Value realization did not.

Token Consumption Is a Cost Signal, Not a Success Metric

Tokens deserve special attention because they are both easy to measure and directly tied to cost in many AI systems.

A team may genuinely need more tokens as AI usage expands.

But token consumption should be interpreted like infrastructure consumption:

as something that needs attribution and optimization not applause.

Google Cloud’s AI/ML cost-optimization guidance recommends estimating project costs, tracking actual costs throughout the lifecycle, applying FinOps practices, and assigning clear ownership for both costs and benefits.

That is a much healthier model.

Instead of asking:

How many tokens did the team consume?

ask:

What useful outcome did those tokens help produce?

This leads to more meaningful measurements:

• cost per resolved case

• cost per accepted code change

• cost per successful agent workflow

• cost per qualified lead

• cost per validated document

• cost per completed investigation

• cost per production-ready artifact

Now cost is connected to an output.

That changes behavior.

A team using a smaller model, shorter context, and fewer calls to achieve the same result should look better, not less mature.

AI FinOps Starts With Attribution

Traditional cloud FinOps became necessary because organizations eventually discovered that aggregate cloud spend was not enough.

They needed to know:

• which team created the cost,

• which workload consumed it,

• why the resource existed,

• whether utilization justified the spend,

• who was accountable for optimization.

AI requires the same discipline.

Google’s FinOps guidance for AI specifically emphasizes cost allocation, model optimization, pricing structures, and reporting cost alongside value so business, finance, and technology teams can make better scaling decisions.

For enterprise AI, useful attribution might include:

By team

Which group generated the spend?

By use case

Was the model supporting customer service, coding, research, sales, or internal operations?

By workflow

Which agent or automation consumed the resources?

By model

Did the task really require the most expensive model?

By outcome

What completed result can be associated with the spend?

Without attribution, organizations tend to manage AI cost with blunt instruments:

• organization-wide token limits,

• arbitrary usage caps,

• model bans,

• broad budget reductions.

Those controls may reduce spend.

They do not necessarily reduce waste.

Good AI FinOps should help teams distinguish between:

expensive work that creates value

and

expensive activity that merely looks productive.

Incentives Shape AI Architecture

Metrics do more than change employee behavior.

They can change technical architecture.

Suppose an AI platform team is rewarded for increasing agent adoption.

What architecture does that incentive encourage?

Possibly:

• more automated workflows,

• more model calls,

• more generalized agents,

• broader integration,

• larger context windows,

• more persistent AI presence.

Now suppose the same team is measured on:

• cost per completed workflow,

• failure rate,

• human intervention rate,

• rework,

• outcome quality,

• business impact.

The architecture may look very different.

The team may choose:

• smaller models for routine classification,

• deterministic code instead of an agent,

• retrieval only when necessary,

• shorter context,

• fewer tool calls,

• stricter workflow boundaries,

• human review only for meaningful exceptions.

The metric has influenced the system design.

This is why measurement cannot be treated as an analytics task that happens after deployment.

Metric design is part of architecture.

If the organization rewards AI activity, engineers will naturally build systems that produce AI activity.

If it rewards useful outcomes, engineers gain permission to choose the least complicated solution that works.

Sometimes that solution will involve AI.

Sometimes it will not.

That is healthy engineering.

The Hidden Cost of “AI Everywhere”

One of the easiest ways to create waste is to make AI usage itself a cultural expectation.

When every team is asked:

“How are you using AI?”

the implicit message may become:

You should be using AI somewhere.

That reverses the decision process.

The right sequence is:

Business or Engineering Problem
↓
Desired Outcome
↓
Constraints
↓
Candidate Solutions
↓
Is AI the best option?
↓
If yes, which AI approach?

A weaker sequence is:

We bought AI
↓
Where can we use it?

The second approach creates solutions in search of problems.

It also makes teams reluctant to remove AI from a workflow once it has been publicly counted as an adoption success.

That matters because mature AI programs should be able to conclude:

This workflow does not need an LLM.

or:

This agent should be replaced with deterministic automation.

or:

The cheaper model produces equivalent business results.

Those should be considered optimization wins.

Not adoption failures.

Measure the Cost of Verification, Not Just Generation

AI operating cost is not limited to inference.

One of the most overlooked expenses is human verification.

Consider an AI coding workflow.

The visible cost may include:

• model tokens,

• API requests,

• agent infrastructure.

But the actual economic cost also includes:

• engineer review time,

• debugging,

• test execution,

• correction,

• security validation,

• rollback,

• maintenance.

The same applies outside engineering.

A customer-service agent may generate answers cheaply but create expensive escalations.

A research agent may retrieve information quickly but require extensive fact-checking.

A document-generation system may produce content at scale but increase editorial review.

That means organizations need a broader unit of measurement:

Cost Per Validated Outcome

Not:

cost per generation

but:

cost to produce something the organization is actually willing to accept and use.

This metric forces AI programs to account for both machine cost and downstream human cost.

It also makes comparisons between workflows much more useful.

A Better AI Measurement Stack

Instead of choosing one heroic KPI, organizations should use a layered measurement model.

Layer 1 Adoption Signals

Useful for understanding behavior:

• active users

• workflow penetration

• feature usage

• task categories

These should usually remain diagnostic, not performance targets.

Layer 2 Cost and Efficiency

Measure:

• cost per workflow

• cost per successful result

• tokens per validated outcome

• model utilization

• retry rate

• unnecessary tool calls

• human intervention

Microsoft and Google both provide current guidance that connects AI management to cost efficiency and business-value measurement rather than usage alone.

Layer 3 Quality

Measure:

• acceptance rate

• factual or functional correctness

• rework

• defect rate

• escalation

• reviewer confidence

• policy compliance

Layer 4 Operational Outcome

Depending on the use case:

• resolution time

• lead time

• deployment success

• customer response time

• analyst throughput

• incident recovery

• workflow completion

Layer 5 Business Outcome

Ultimately:

• revenue

• margin

• customer retention

• conversion

• cost reduction

• risk reduction

• capacity created

Not every AI initiative needs to improve revenue directly.

But every production AI initiative should eventually connect to a meaningful organizational outcome.

Four Questions to Ask Before Adding an AI KPI

Before placing an AI number on an executive dashboard, ask four questions.

1. What behavior will this metric encourage?

Assume people will optimize for it.

Because eventually they will.

2. Can this metric increase while the business outcome gets worse?

If yes, it is probably a proxy rather than a success metric.

3. Does the metric account for cost?

A workflow that improves output by 5% while increasing cost by 400% may not be an improvement.

4. Does it include downstream work?

Generation without verification, integration, and correction provides an incomplete picture.

These questions do not eliminate the need for AI telemetry.

They make telemetry useful.

What a Healthy AI Dashboard Should Tell You

A good dashboard should make it possible to answer questions such as:

• Where is AI producing measurable value?

• Which use cases are becoming more efficient?

• Which workflows are becoming more expensive?

• Where is human review increasing?

• Which model is overqualified for the task?

• Which agent loops are consuming resources without improving outcomes?

• Which workflows should be simplified?

• Which AI use cases should be discontinued?

That last question is important.

A mature AI program should be able to stop an AI initiative because the economics do not work.

If every AI project must eventually be declared successful, the measurement system has already failed.

Conclusion

AI adoption is worth measuring.

But it should not become the destination.

Prompt counts, token consumption, agent runs, generated code, and active users can help organizations understand how AI is being used.

They cannot tell the organization whether that usage is valuable.

Once those activity metrics become targets, they begin shaping behavior:

• more prompts,

• larger models,

• more agent executions,

• more generated output,

• more AI where AI may not be necessary.

The result can look like adoption while functioning like waste.

A better measurement system connects AI activity to:

• cost,

• quality,

• verification,

• operational outcomes,

• and business value.

The goal should not be to maximize AI usage.

It should be to maximize the value produced per unit of AI usage.

That distinction may determine which organizations scale AI economically and which simply scale the bill.