Measuring AI coding assistants
What Copilot, Claude Code, Codex and Gemini Code Assist figures are good for, how to value the time they save, and the pitfalls that make them mislead.
Answered from these docs only, by a model that cannot see your account. Check the pages it cites.
Checked against the product on November 9, 2026
GitHub Copilot, Claude Code, OpenAI's Codex and Gemini Code Assist each report how much developers used them, such as suggestions offered and accepted, lines of code, and how many people used the tool on an average day. Not every tool reports every one of these. This page covers what those figures can tell you, how Roiva turns them into a value, and where they mislead.
What the figures are good for
- Valuing the time an assistant saves. The AI Coding Assistant Rollout template values it as accepted suggestions × minutes saved per acceptance × your fully loaded developer hourly rate. For an assistant that counts no acceptances, such as Codex, it uses lines of code kept × minutes saved per line instead. An initiative measuring one assistant gets one of the two, never both, because both measure the same saved hour. With more than one assistant reporting, each gets a formula of its own, in its own unit.
- Valuing what developers report, where the assistant reports nothing. See When the assistant reports nothing.
- Following adoption. Active users and accepted suggestions, month to month, show whether a rollout is taking hold, slowing, or concentrated in a few people.
- Comparing a tool with itself. The same assistant's figures this quarter against last quarter answer most of what a team wants to know.
The value is freed capacity, not cash: developer hours priced at what they cost. It's reported beside cash value and counts in no ROI until someone shows, with a document, that the time became cash, such as a contractor not renewed or a hire not made. See How Roiva calculates value.
The units aren't the same across tools
Each assistant counts in its own unit, and they don't convert:
- Copilot counts acceptance activities. One activity can bundle several completions.
- Claude Code counts accepted tool actions, one edit or write each, whether it reports through an Anthropic Console organization or across Claude Enterprise seats.
- Codex reports no count of accepted suggestions at all, so Roiva reads the lines of code it added that developers kept.
- Gemini Code Assist counts the code suggestions developers accepted in the IDE and the Gemini CLI, as Google records them in Cloud Monitoring for the project it reports to. Built from Google's published reference. If Google's answer doesn't match that reference, or leaves out a day it reported before, Roiva records nothing from it and says so on the connection, rather than count no one coding.
Codex comes through the ChatGPT Enterprise connection. A key made only for Codex analytics stops working on December 15, 2026, when OpenAI retires that endpoint; a key with Enterprise analytics usage keeps reading Codex after that. See Measuring a workplace AI assistant.
So Roiva reads coding assistants from one source at a time, the primary, and never adds two tools' counts together. Organization Settings → Metric Sources shows which one the account reads. Where a team runs more than one assistant and they report, the template gives each a formula of its own, so one that goes quiet holds only its own month. Copilot and Claude Code each have their own minutes figure. Claude Code on Console and on Claude Enterprise share one, and Gemini Code Assist and Codex use the general figures for an acceptance and a line of code.
If an assistant reports work that no formula on an initiative reads, the initiative's Health tab says so, and its value leaves that assistant out until you count it. Count adds the assistant's formula from the month after the last month these formulas have an entry for, so no month is priced twice, and an earlier month with no entry stays unbooked. With no entries yet, it books past months too. A formula that reads only the primary source is replaced at that point: its booked months stay, and entries still waiting on it are dropped. The row is advice and doesn't count against Health, so you can leave an assistant out. If you archive an assistant's formula, it stays out and the row goes away.
Lines of code differ too. Copilot and Gemini Code Assist count the lines developers accepted, Claude Code the lines it added, and Codex the lines it added that developers kept. Don't add them up or compare one tool's lines with another's.
When the assistant reports nothing
Some assistants report nothing Roiva can read. Claude Code signed in with a claude.ai plan (Pro, Max or Team) reports to that plan, and no API reads it. Connected through an Anthropic organization whose developers work that way, Claude Code reports no acceptances or lines at all. With GitHub connected, the template values the extra merged pull requests instead, and only the share you credit to the initiative. Without it, the formula books $0.
Either way, the formula's page offers Developer Capacity Freed by AI (hours saved, self-reported) instead. It values developers who answered a survey × the hours a week they say the assistant saved them × the weeks in the period × your developer hourly rate. DX's AI measurement framework tracks the same figure, time saved per developer per week. Missing Inputs asks for both from each month's survey, once the month has ended, and a month waits until both are in. Only a person switches to it: Roiva never picks it from your connected tools.
This figure is what developers report, not a measurement. In METR's 2025 randomized trial, experienced developers believed AI tools made them 20% faster, but they took 19% longer. So every month this formula books says Self-reported and waits for a person to approve it, even with auto-approve on. Before you approve, check the hours against something your team already watches, such as merged pull requests or cycle time. Ask the same question the same way each month, and count only the developers who answered.
Best practices
- Set the minutes yourself. Roiva ships a default for minutes saved per acceptance and per line so a figure appears on day one, but no published study measures either one, so each is a convention rather than a measurement. A time study can't set them either: it times whole tasks, and no assistant reports how many acceptances or lines one task took. Replace each default with a figure your engineering leads will stand behind, and say why in its note. Ground it in more than how much faster people feel: in METR's 2025 randomized trial, experienced developers took 19% longer with AI tools while believing they had been 20% faster. The defaults themselves are in Assumption defaults.
- Price Copilot and Claude Code separately. An accepted Claude Code edit is a smaller unit than a Copilot acceptance, so one minutes figure for both overvalues one tool or undervalues the other.
- Use a fully loaded rate. The developer hourly rate should include benefits and overhead, not salary alone, or the saving reads low.
- Read counts beside active users. A month with twice the acceptances might mean twice the developers rather than twice the help per developer.
- Prefer acceptances to lines. An accepted suggestion is a decision a developer made. A line is output, and says nothing about how hard it was to write.
- Look for the saving downstream. If time is really being saved, it tends to show up somewhere a team already watches, such as merged pull requests or cycle time, which GitHub reports too. Those figures have many causes, so treat them as a check, not as the value. The exception is an assistant that reports no accepted suggestions or lines at all: then the formula library values the extra merged pull requests, and only the share you credit to the initiative.
Pitfalls
- Accepted isn't shipped. An accepted suggestion can be rewritten or deleted an hour later. The count says a developer took the code, not that it survived review.
- Usage isn't value. Acceptance counts rise as people try a tool, and stay high when the tool is used for work that wouldn't have taken long anyway. The minutes figure is where that's accounted for, which is why it's worth setting yourself.
- Seats aren't users. Licenses bought and people using the tool on an average day are different numbers. Cost follows seats, value follows use.
- Two Claude Code populations. If some developers use Claude Code through an Anthropic Console organization and others on Claude Enterprise seats, a new initiative gets a formula for each. One already on a formula counting Claude Code from one of them leaves the other out, and its Health tab says so.
- Agent-sized pull requests. The merged pull request formula prices each extra pull request at the developer hours one took before the assistant, 9.3 by default, worked out from teams merging 3.8 a week per engineer. Where agents open and merge many small pull requests, that figure doesn't fit them. When GitHub's count of a month's pull requests says who opened them, a month whose extra pull requests, at that figure, come to more hours than those people work can't be approved, whatever share you credit, and doesn't count as waiting for approval. A count typed by hand, or a month GitHub synced before Roiva could tell people from bots, doesn't say who opened them, so that month is marked Hours not checked and waits for you to check it. Set your own hours per merged pull request for pull requests that size, or measure the initiative another way.
- A baseline from after the rollout. Roiva works the baseline out from the months before the initiative began, where GitHub has them. If the team already used the assistant then, the difference says nothing about what the assistant changed. Record a baseline from a month before anyone used it, or measure the initiative another way.
- The API bill isn't the assistant's bill. An Anthropic connection reads the Console organization's whole API bill, which also carries anything else built on it, such as your product's own features. Linking Anthropic as a coding assistant initiative's cost source, or editing it, asks for the workspaces Claude Code bills to; Anthropic names the one it creates "Claude Code". Claude Code on a claude.ai plan isn't on that bill at all: its seats come from your books, or a recurring cost if they aren't there.
Each of these figures has its own entry in the Metric registry, with what to watch out for when you read it.