Nobody Agrees What AI Can Actually Do Anymore, and the Gap Is Growing Fast
In this article
Bottom line: Developers now report opposite experiences with the same models, and the gap is widening because capability depends on task, harness, and verification habits, not just the model.
METR's 2025 randomized trial found experienced open-source developers were 19% slower with AI tools while believing they were 20% faster.
Treat every "AI can/can't do X" claim as unverified until it names the task, the setup, and how the output was checked. Run your own task-level evals.
Last month, two engineers on my team used the same model, Claude 4.6, on the same codebase, in the same week. One of them told me it had replaced about half of her ticket work.
The other told me, with a straight face, that it was "a fancy autocomplete that lies."
They're both competent, and neither of them is wrong. That's what's bothering me.
The Argument I Keep Losing
I've had some version of this conversation about forty times this year.
Someone says a model can do a thing, someone else says it can't, and we spend twenty minutes trading anecdotes that can't be compared. There's no shared ruler, so nobody wins and nobody updates.
For a long time I assumed the skeptics were just bad at prompting, or the enthusiasts were just bad at checking. That was lazy of me. I was treating a measurement problem like a character flaw.
Here's what changed my mind. I sat down with the skeptic, whom I'll call Dan, and watched him work for an hour. Dan wasn't doing anything dumb.
He was working in a 14-year-old Rails monolith with undocumented conventions, asking the model to make changes that touched six services, and pasting in a few files by hand.
My colleague who loves it was building greenfield CRUD endpoints in a clean TypeScript service with tests. Same model, same week. Completely different jobs.
Capability Isn't a Property of the Model
We talk about "what the model can do" as if it were a fact about the model, like a car's top speed. It isn't.
It's a fact about the model, the task, the context you give it, the tooling around it, and how you verify the result.
Change any one of those and the answer flips. Here's the framework I've started using to sort out why two people can honestly disagree:
1. Task shape. Is the problem well-specified, local, and easy to check? Or is it vague, cross-cutting, and dependent on tribal knowledge?
2. Harness. Is the model running in an agent loop with file access, a shell, and tests? Or is it a chat window where you paste snippets?
3. Context quality. Does it see the conventions, the failing test, the real error? Or a trimmed-down guess at them?
4. Verification cost. Can you tell in thirty seconds if the answer is right? Or does checking take longer than writing it yourself?
I call this the four-dial model. Two people can stand at opposite ends of those dials and report opposite truths about the same weights.
The harness gap is the biggest one
Most of the disagreement I see is really harness disagreement.
A person using ChatGPT 5 in a browser tab and a person running an agentic coding tool against a repo with a test suite are not using the same product, even if the underlying model family is similar.
The agent loop is where a lot of the gains come from. It lets the model run the tests, read the failure, and try again.
That feedback loop turns a mediocre first draft into a working change, and it's invisible if all you've ever done is paste code into a chat box.
The Perception Problem Makes It Worse
This is the part that humbled me. In 2025, METR ran a randomized controlled trial with 16 experienced open-source developers working on their own mature repositories.
When they were allowed to use AI tools, they took about 19% longer to finish tasks. Before starting they'd predicted a speedup, and afterward they still believed they'd been about 20% faster.
I'm not citing that to say "AI makes you slower." The study was small, specific, and about early-2025 tools on large, familiar codebases, and the authors were careful about its limits.
I'm citing it because it shows our felt sense of productivity is a poor instrument.
That breaks the usual way we settle arguments.
If "it feels fast" isn't evidence and "it feels useless" isn't evidence either, then most of what gets posted on Hacker News on any given day is two groups reporting their vibes at each other.
And I'm guilty of this. I spent most of last winter convinced I was shipping 3x faster.
When I finally went back through my own merged PRs and review cycles, the honest number was closer to 1.3x, with a fatter tail of rework.
That's still good, but it's nowhere near what I'd been telling people.
Why the Gap Is Growing, Not Shrinking
You'd expect better models to bring everyone toward agreement. I think the opposite is happening, for three reasons.
First, the ceiling keeps moving while the floor stays put. A well-configured agent on a well-tested codebase can now do work that was out of reach a year ago.
A person pasting snippets into a chat window gets roughly what they got in 2024. The best-case and typical-case experiences are diverging.
Second, the tooling is a moving target. The tools I use today didn't exist in this form eighteen months ago.
When someone says "I tried it and it was bad," I have no idea if they tried it in March or last week, with which harness, on which plan, with which context window.
Their experience may be real and also obsolete.
Third, the cost of verification is unevenly distributed. Code with tests is cheap to verify. Legal summaries, architecture decisions, and security reviews are expensive to verify.
The model can look equally confident in both cases, so people in verification-cheap domains see a miracle and people in verification-expensive domains see a liability.
That last one matters more than people admit. A tool that's right 90% of the time is a huge win if you can check it instantly and a hazard if you can't.
The Reality Check
I want to be fair to both camps, because both have a point.
The enthusiasts are right that these systems do real, economically meaningful work now. I've watched Claude 4.6 untangle a gnarly dependency upgrade that would've cost me an afternoon.
Waving that away isn't skepticism, it's denial.
The skeptics are right that fluent output is not correct output, and that models still fail in ways that are hard to predict.
I've had a model confidently invent a config flag that didn't exist and then "fix" the resulting error by inventing a second one. The failures are quiet, and quiet failures are the expensive kind.
What neither camp has is a shared, boring, task-level measurement. Benchmarks don't fill that gap for most working developers.
They're mostly someone else's tasks, often partly contaminated, and they say little about your repo or your review process.
So my verdict is this: "Can AI do X?" is the wrong question. It has no answer.
The answerable question is "Can this model, in this harness, with this context, do this task, at a verification cost I can afford?"
What I'd Actually Do
Here's the workflow I've landed on after being wrong in public one too many times.
Build a personal eval set
Keep a folder of 10 to 20 real tasks from your own work, with the context and the known-good outcome. A flaky test you fixed, a migration you wrote, a bug report with the actual root cause.
When a new model drops, run it against that folder before you form an opinion.
This takes an afternoon and it's the single most useful thing I've done this year. It replaced "I think the new one is better" with "it passed 11 of 15 where the last one passed 8."
Report claims with four fields
Whenever I say something about AI capability now, I try to include the task, the model and harness, how I verified it, and how long ago I tried.
It sounds pedantic, but it turns an unfalsifiable vibe into something a colleague can reproduce or dispute.
Time your own work, once
Pick a week and track a few comparable tasks with and without the tool. You don't need a statistically perfect study, you need to catch the gap between how fast it felt and how fast it was.
I was off by more than 2x, and I'd bet most of us are off by something.
Match the tool to verification cost
Use the model aggressively where checking is cheap: tests, type-checked code, scripts with visible output.
Use it cautiously where checking is expensive, and make the verification step explicit instead of trusting your read of the answer.
The Part I Can't Resolve
I used to think the end state here was consensus: a few more model releases and everyone would agree on what these things are. I don't believe that anymore.
I think we're heading toward a world where "what AI can do" is as personal and situational as "how good is this programming language." The honest answer will be that it depends, and the people who thrive will be the ones who can say precisely what it depends on.
That's less satisfying than a headline. But Dan and my other colleague eventually stopped arguing once we put their tasks side by side, and both of them changed how they work.
That's the only kind of progress I've seen come out of this debate.
What does your own best and worst AI experience have in common, and could you write down the task, the setup, and how you checked it well enough for someone else to reproduce it?
