Stop Trusting Cloud AI With Your Data — Google Is Closing The Gap On That

> **Bottom line:** Google's HEIR project — an open-source compiler stack for fully homomorphic encryption (FHE), built with MLIR and backed by contributors from Google, Zama, and academia — is closing the gap between "encrypt everything, always" and "AI that actually runs at usable speed." FHE lets a model compute on encrypted data without ever decrypting it, which means a cloud provider can run inference and never see your prompt, your document, or your output in plaintext.

It's still 10x to 1000x slower than plaintext inference depending on the workload, down from the 100,000x slowdowns FHE was infamous for a decade ago.

If you've been pasting client contracts, patient notes, or proprietary code into ChatGPT and hoping for the best, this is the first real technical alternative — not a policy promise, a mathematical one.

I pasted a client's term sheet into Claude last spring to get a plain-English summary of an indemnification clause.

Then I stopped, stared at the screen, and thought: *I just sent someone else's confidential financial terms to a server I don't control, run by a company whose retention policy I have not personally audited.*

Nothing bad happened. But that half-second of dread is one every developer working with sensitive data has felt at least once.

And for a long time, the honest answer to "how do I stop that" was: don't use cloud AI for sensitive data.

Full stop. Run something local, smaller, worse.

That answer is starting to change, and it's not because a vendor promised to try harder.

The Setup: Why "Just Trust Us" Was Never Good Enough

I spend a chunk of my week building internal tooling for teams handling data they legally can't leak — health records, financial filings, source code under NDA.

The standard advice has always been some flavor of: encrypt in transit, encrypt at rest, sign a DPA, hope the vendor's SOC 2 report means what it claims.

That's encryption *around* the computation, not *during* it.

The second a cloud model needs to actually read your prompt to generate a response, it's sitting in plaintext in memory somewhere you don't control.

Confidential computing (enclaves like AWS Nitro or Azure's confidential VMs) narrowed that window, but it still depends on trusting the hardware vendor's attestation and hoping nobody's found a side-channel this month.

Researchers have found side-channels in enclaves before. Repeatedly.

What I wanted — what a lot of us have wanted since the FHE concept became mainstream knowledge around 2020 — was math that made the trust question moot. Compute on ciphertext. Never decrypt.

Get an encrypted answer back that only I can open.

That math has existed since Craig Gentry's 2009 breakthrough. It's just been too slow to matter for anything beyond toy demos.

The Core Insight: HEIR Is the Boring Infrastructure That Makes FHE Real

Here's the thing about fully homomorphic encryption that gets lost in the hype cycle: the crypto primitive was never really the bottleneck. The bottleneck was tooling.

Writing an FHE program by hand meant manually managing "noise budgets" (every homomorphic operation adds noise to the ciphertext, and too much noise corrupts the result), choosing bootstrapping strategies, and hand-tuning polynomial parameters — work that required a PhD in lattice cryptography, not just a TensorFlow tutorial.

Google's HEIR project (Homomorphic Encryption Intermediate Representation) attacks that problem the way compiler engineers attack every hard problem: with an intermediate representation and a bunch of optimization passes.

It's built on MLIR, the same compiler infrastructure underpinning modern ML compilation for TPUs and XLA.

That's not a coincidence — Google is treating "compile this neural net to run on encrypted data" as the same category of problem as "compile this neural net to run on a TPU."

What Changed Technically

A few things stack together here, and it's worth naming them because "Google fixed encryption" is doing a lot of hand-waving otherwise:

- **Automated noise management.** Instead of a human tracking ciphertext noise levels by hand, the compiler inserts bootstrapping operations only where the noise budget actually requires it — trimming a huge source of unnecessary slowdown.

- **Scheme flexibility.** HEIR supports multiple FHE schemes (CKKS for approximate arithmetic on real numbers, BGV/BFV for exact integer math), and picks the right one per operation instead of forcing an entire model into one scheme's tradeoffs.

- **Hardware targeting.** Because it's MLIR-based, the same encrypted-computation graph can eventually be lowered to different backends — CPU today, with accelerator support the obvious next step, the same way XLA targets both CPUs and TPUs.

None of this makes FHE fast in the way plaintext inference is fast.

A matrix multiply that takes microseconds in plaintext can take milliseconds to seconds encrypted, depending on model size and scheme.

But "seconds instead of unusable" is the difference between a research curiosity and something you'd actually put behind an API.

What This Looks Like in Practice

The workflow I've been testing looks roughly like this: encrypt a small feature vector or embedding client-side, send only ciphertext to the inference endpoint, get back an encrypted result, decrypt locally.

The server hosting the model never has access to the plaintext input or output at any point in the pipeline — not in a "we promise not to look" sense, but in a "there is no key on that machine that could decrypt it" sense.

For full transformer inference — the "summarize this document" use case I actually want — we're not there yet at production latency for most workloads.

Where FHE is genuinely production-ready right now is smaller, well-defined computations: encrypted similarity search over embeddings, private set intersection, logistic regression scoring, secure aggregation for federated learning.

Think fraud scoring, private genomic matching, encrypted recommendation lookups — not "chat with your encrypted PDF" yet.

The Reality Check: This Is Not "Cloud AI, But Private" Tomorrow

I want to be straight with you because I've watched too many people read "Google" and "encryption breakthrough" in the same sentence and assume ChatGPT-level private inference ships next quarter.

It doesn't. FHE inference on a full-size LLM, end to end, encrypted the whole way through, at conversational latency — that's still a research target, not a product.

The overhead for deep, nonlinear networks (attention layers, softmax, activation functions) is still brutal, because FHE handles addition and multiplication natively but approximates everything else, and every approximation costs precision and speed.

What's real today is narrower: encrypted inference on smaller models, encrypted preprocessing and feature extraction feeding into a plaintext model behind a trust boundary you do control, and encrypted search/matching primitives.

That's genuinely useful — a hospital can now run a matching algorithm against a national encrypted registry without either party seeing the other's raw records — but it's not "paste your term sheet into an encrypted Claude session" yet.

I'd also flag: **open-source compiler infrastructure is not the same as a shipped Google product.** HEIR is a toolchain other engineers can build on, including Google's own teams, but there's a real distance between "Google contributes to an open compiler stack" and "Google Cloud sells you encrypted-inference-as-a-service with an SLA." Watch that gap.

Vendors are very good at letting research momentum imply product maturity.

The Practical Takeaway: What to Actually Do This Month

If you're building anything that touches regulated or sensitive data, here's where I'd put my energy right now, in order:

1. **Audit what's actually leaving your boundary today.** Most teams don't have a clear map of which prompts contain real customer data versus synthetic examples.

Fix that first — it's free and it's overdue.

2. **For search/matching/scoring workloads, look at FHE now.** Libraries like Microsoft SEAL, OpenFHE, and Zama's Concrete are usable today for exactly the narrow use cases described above.

HEIR is worth watching as the compiler layer matures, especially if you're already in the MLIR/XLA ecosystem.

3.

**For general LLM use, keep using confidential computing and strict data minimization as your real-world floor.** It's not mathematically bulletproof, but it's what's actually deployable at chat latency right now.

4. **Don't wait for a vendor to announce "private AI" and assume it means full-model FHE.** Ask specifically what's encrypted, when, and what the latency tradeoff is.

If they can't answer the noise-budget question, they're selling you confidential computing with better marketing.

The honest version of this story isn't "Google solved privacy." It's "the compiler tooling that makes FHE usable for real workloads finally exists, and it's open source, and that's how these things actually go from paper to product — slowly, boringly, one intermediate representation at a time."

That's less exciting than a headline. It's also the version that's true.

Have you actually stopped and thought about what happens to the data in your prompts once it leaves your terminal — or is that a problem for future-you? Tell me what you've found.

I'm collecting workarounds.

---