AI has access to a vastly larger working memory than the human brain
In this article
**Marcus Webb** — Infrastructure engineer turned tech writer. Writes about AI, DevOps, and security.
> **Bottom line:** Human working memory holds about four chunks of information at a time — a number cognitive scientist Nelson Cowan nailed down in a landmark 2001 paper that replaced the old "magic number seven." Modern LLMs now ship with context windows in the hundreds of thousands to low millions of tokens — Gemini 1.5 Pro launched with 1 million tokens back in 2024, and Anthropic put Claude Sonnet into a 1-million-token beta in 2025.
That's not a marginal edge, it's a difference of roughly five to six orders of magnitude.
But bigger context isn't the same as better reasoning, and I learned that the expensive way, watching a production RAG pipeline choke on its own "generous" context window.
I spent a Tuesday night in March staring at a Grafana dashboard that made no sense.
Latency on our retrieval service had tripled, but nothing in the infra had changed — no new deploys, no traffic spike, no noisy neighbor on the cluster.
The culprit turned out to be embarrassingly simple: someone had swapped our chunking strategy for "just dump the whole document into context, the model can handle it now."
The model could handle it. Our p99 latency and our AWS bill could not.
The Four-Item Ceiling Nobody Talks About
Here's the thing that should humble every engineer who's ever felt smug about their memory: **your brain can hold about four things in working memory at once.** Not seven, despite what you learned in that psych 101 lecture — that was George Miller's 1956 estimate, and it's been mostly superseded.
Cowan's 2001 review in *Behavioral and Brain Sciences* pinned the real number closer to four chunks, and follow-up work has mostly held up.
Four.
That's your RAM for active thought — the phone number you're repeating before you type it in, the three variables you're tracking while debugging a race condition, the mental model of a call stack before you lose the thread and have to re-read the trace.
Now compare that to what sits on the other side of your API calls.
Claude models handle context windows that started at 100K tokens back in 2023 and now run to 200K standard, with 1M-token access rolling out through 2025 and into this year.
Gemini blew past that even earlier — Google shipped a 1-million-token window in Gemini 1.5 Pro in early 2024, with experimental runs pushing toward 2 million.
A million tokens is somewhere around 700,000 words. That's not "hold a phone number in your head." That's "hold the entire Lord of the Rings trilogy, twice, plus commentary."
What "Working Memory" Actually Means for a Machine
I want to be precise here, because this comparison gets sloppy fast in the discourse, and sloppy comparisons are how you end up with a Hacker News thread arguing past itself for 400 comments.
**Human working memory is active and volatile.** It's the stuff your brain is manipulating *right now* — comparing, reordering, holding in tension. It decays in about 15-30 seconds without rehearsal.
It's not your long-term memory, and it's definitely not your entire knowledge base.
**An LLM's context window is closer to a scratchpad than to memory in the human sense.** Every token in that window gets attended to on every forward pass — the model isn't "recalling" it, it's re-reading the whole thing each time, weighted by attention.
That's actually a big part of why it scales the way it does: it's brute-force re-derivation, not storage-and-retrieval like your hippocampus doing memory consolidation overnight.
So when I say AI has a "vastly larger working memory," I mean something specific: **the amount of task-relevant information a model can hold in active, attendable state during a single reasoning pass dwarfs what a human can hold during a single conscious thought.** That's the real claim, and it's true, and it's also not the whole story — which is where the hype train usually jumps the rails.
The Cost Nobody Puts in the Headline
Here's the part that matters if you actually run this stuff in production instead of just marveling at it in a demo.
Attention in transformer architectures scales roughly quadratically with sequence length in the naive implementation — optimizations like FlashAttention and sliding-window variants have chipped away at that, but the underlying math still means a 1M-token context isn't "free" the way a human glancing at a longer document feels free.
I've watched teams burn budget stuffing entire codebases into context because "the window's big enough now" — and get worse retrieval accuracy than a well-tuned RAG setup with a 4K window, because **needle-in-a-haystack performance degrades in the middle of long contexts.** This is documented, not anecdotal — it's often called "lost in the middle," and it shows up reliably across model families.
A bigger window doesn't mean uniform attention across that window.
Where the Comparison Breaks Down
This is where I have to talk you out of the version of this story that gets 10,000 claps and is also wrong.
The "AI has bigger working memory, therefore AI thinks better" framing skips over what working memory is actually *for* in humans: it's the workspace where we compress, abstract, and generalize.
Cowan's four chunks aren't a bug — they're a forcing function. You're bad at holding twelve variables in your head, so you build abstractions, name things, write comments, draw diagrams.
**Constraint drives compression, and compression is most of what good engineering is.**
A model with a million-token window doesn't need to compress anything. It can just... keep everything.
And you can watch what happens when it does: verbose, unfocused outputs, degraded signal on the specific fact you needed, higher latency, higher cost.
I've seen this firsthand rebuilding a support-ticket triage system — the version that retrieved 3 tightly relevant documents into an 8K context outperformed the version that dumped 200K tokens of ticket history at the model, both on accuracy and on cost per query, by a wide margin.
So the honest version of the headline isn't "AI's memory makes yours look pathetic." It's: **AI has a bigger scratchpad, but it hasn't solved the problem your four-chunk limit forces you to solve — figuring out what actually matters.**
What This Means for How You Build
If you're an engineer shipping anything on top of these models this year, here's where I'd put my attention instead of the raw context-window number on the pricing page:
- **Test retrieval quality at the context lengths you'll actually use**, not the max the vendor advertises. Run your own needle-in-a-haystack eval on your data, not a benchmark paper's.
- **Don't treat a bigger window as a replacement for RAG or good chunking.** Use the extra headroom for genuinely long-range tasks — full-codebase refactors, long-document synthesis — not as an excuse to skip retrieval engineering.
- **Watch your cost curve, not just your capability curve.** Token costs and latency both climb with context length even when accuracy plateaus or drops.
I budget context like I budget database queries — the fact that I *can* fetch everything doesn't mean I should.
- **Remember the model isn't compressing for you.** If you want an abstraction, a summary, a decision — ask for it explicitly.
The model will happily hold a million tokens of raw noise without ever telling you what mattered.
I ended up reverting that RAG pipeline back to targeted retrieval with a much smaller context window. Latency dropped back to baseline, accuracy went *up*, and the AWS bill stopped setting off alerts.
The bigger scratchpad was available. It just wasn't the right tool for what we were actually trying to do.
Cowan's four chunks have survived 25 years of cognitive science because they describe something true about how thinking works, not just a hardware limit we're waiting to route around.
The models don't have that limit anymore. **They also haven't figured out what to do with the freedom yet — and neither, frankly, have most of the teams shipping them into production.**
Have you actually measured whether a bigger context window helped your retrieval accuracy, or did you just assume it would and move on?
I'd genuinely like to know what you found — drop it in the comments.
---
