How we measure value & effectiveness
The honest version. "Value" (hours saved / $) can't be captured directly from tokens — anyone who promises a hard number is bluffing. Here's the ladder: what's real today, what needs Roper's data, and what's only a model.
The AI productivity paradox (2025–26): adoption hit ~93% but independently-measured delivery gains stalled near ~10% — individual output (PRs, commits) soars while org delivery stays flat, because speed just shifts the bottleneck to review, QA & rework. So adoption ≠ value, and even PR volume ≠ value unless the work merges and ships.
| Tier | What it measures | Kind | Source | Status | The catch |
|---|
| A · Usage | active users ÷ seats, frequency, active time, acceptance rate | effectiveness | Anthropic analytics + TokenTrail | have now | Seats ≠ adoption; acceptance rate is a vanity metric |
| B · Output / velocity | PRs merged, commits, time-to-first-commit, throughput | value | Claude Code analytics + Git | have (partial) | PRs raised but not merged = review inventory, not value |
| C · Delivery — DORA | cycle time, lead time, deploy freq, change-failure, rework/revert | real value / impact | Roper Git/Jira/CI + pre-AI baseline | needs Roper data | The honest ceiling — segment AI-vs-human & baseline |
| D · Business $ / ROI | hours saved × rate, cost-per-useful-outcome, ROI % | $ model | a factor Roper sets | assumption | Never assert; always a labeled, adjustable model |
| E · DevEx / SPACE | satisfaction, flow, well-being | human signal | short surveys | optional | Catches what activity misses; survey-based |
Our definition (defensible end-to-end):
Effectiveness = adoption — are they using it well (Tier A).
Value = output that ships — PRs merged, throughput (Tier B → C).
Dollar ROI = a transparent, Roper-set model (Tier D) — never a number we invent.
Today vs. built with Roper:
Today (real): cost per company/user/model + usage + AI-attributed PRs/commits.
Phase 2 (joint): DORA outcomes on Roper's Git/Jira with a pre-AI baseline — where effectiveness becomes proven impact.
Sources: DORA 2025/26 · SPACE (GitHub/Microsoft) · DevEx (DX) · independent AI-coding ROI studies. Cost = measured (Bedrock). Value = this ladder. $ = Roper-owned model.