Think about the last few things you pasted into a chatbot. A draft email to a client. A spreadsheet with names and numbers in it. A contract clause, an error log with hostnames in it, a medical test result, an idea you hadn't told anyone yet. Every one of those made a copy of itself, travelled across the internet, and landed on a machine owned by somebody else — where it now sits under policies you never read, can't negotiate, and can't revoke.

We got here because cloud AI is genuinely good and genuinely easy. But "easy" and "safe" are different words. This post is the honest ledger: what actually happens to your data at the big inference providers, and what changes when the model runs on hardware you own — what I've taken to calling private AI.

CLOUD AI your prompt leaves the building Your data the internet Provider’s servers ! Copied and retained — often 30+ days ! May be used to train their next model ! Readable by subprocessors & legal process ! Their outage becomes your outage ! Per-token pricing, rate limits, model swaps ! Retired or filtered on their schedule PRIVATE AI your data never leaves your machine Your data localhost — nothing crosses the wire Your model, your hardware Zero copies on anyone else’s server One-time hardware cost, then ~free per query Runs offline — your uptime, not theirs Compliance-ready: nothing crosses the wire Your documents indexed and searched locally Same model and rules, year over year
Cloud AI vs. private AI: the difference is which direction your data travels.

What Actually Happens to Your Prompt

When you hit send on a cloud AI service, your prompt stops being solely yours. It is transmitted, then copied — load balancers, inference nodes, request logs, backups. Copies are the whole business model of the cloud, and each copy is a place your data now lives.

Retention. Most providers keep prompt and response text for a period measured in weeks at minimum, sometimes indefinitely for "abuse monitoring" or "service improvement." Consumer terms have historically defaulted retention on, with the opt-out buried several layers into settings — and the opt-out is usually prospective only. The thing you pasted last month is still wherever it went.

Training. Provider terms have repeatedly moved the goalposts here: what was "we don't train on your data" at launch became "we may, unless you opt out," and free tiers and "free model access through ad-supported products" have generally carried training rights baked in. Even with opt-outs honored perfectly, the burden of proof sits entirely on their side of the API.

Humans and subprocessors. Big AI companies use third-party contractors to label and review model outputs, and several have openly confirmed that reviewers see real prompts. Beyond that, your data is subject to the legal process of whatever country the servers sit in — which is precisely why "the data stays in the EU/Canada region" became a compliance checkbox: the concern isn't hackers, it's subpoenas and lawful access.

None of this requires the provider to be evil. It's the normal, disclosed, legal operation of the service. The point is simpler: you have outsourced the confidentiality of your most sensitive material to a party whose incentives are storage, analysis, and growth — and you can't audit them.

It's Not Just Privacy — It's Fragility

Even if you personally don't care about the privacy trade, cloud inference gives you a system you don't control:

  • The model changes under you. Providers silently swap model versions; the prompt that worked last month behaves differently today. Prompt tweaks get retired or "improved" without notice.
  • The price and the limits change under you. Per-token billing, rate limits, and API tiering are business decisions made in someone else's quarterly interest.
  • Their outage is your outage. A status-page incident at a big provider separates millions of people from their own workflows. A home or office internet blip does the same thing instantly.
  • The vendor can just stop. Models get deprecated, products get shut down, features get re-priced. If your workflow was built on their API, your workflow is on their roadmap — not yours.

What Private AI Actually Means

"Private AI" is not a product category with a logo — it's an architectural choice: the model weights run on hardware you own, and the data path never leaves it. Open-weight models (the Llama, Qwen, Mistral families) are now good enough that a 20–30-billion-parameter model on a single workstation handles the everyday work — drafting, summarizing, coding questions, analysis — that most people use cloud chatbots for.

The stack is boring in the best way: a GPU box, a serving tool like Ollama or vLLM, and a chat interface like Open WebUI in front of it. Add a small local embedding model and your document collection gets chunked, indexed, and searched on the same box — "chat with your files" where the files never left. I wrote about building exactly that setup in My Adventures Running Large Language Models Locally.

What You Get Back

Sovereignty over your own words

The most important property is also the quietest: nothing happens. No copy lands anywhere. No retention clock starts. No policy change six months from now can retroactively expose material you processed last year, because there is nothing out there to expose. Privacy stops being a promise you trust and becomes a physical fact you can point at.

Cost that stops being a meter

Cloud inference is a utility bill: every token metered, forever. Local inference is a capital purchase: you buy the hardware once and the marginal cost of a query is a few cents of electricity. For anyone who uses AI heavily — daily drafting, batch processing, document pipelines — the crossover isn't years away, and it comes with no rate-limit anxiety.

Availability and consistency

Your model works during internet outages, during provider incidents, and at 2 a.m. exactly the way it worked at noon. The weights don't change unless you decide to change them. That stability alone is worth it for any workflow you depend on professionally.

Compliance without gymnastics

If you handle anything regulated — client records, health information, employer data under a collective agreement, government material subject to data-residency rules — "the data never leaves the building" is not a marketing line, it's the shortest possible answer to every due-diligence questionnaire. Many organizations now flatly prohibit pasting regulated content into consumer AI tools. Private AI is how you keep the capability without the violation.

The Honest Trade-Offs

I'd be selling you something if I skipped this part. Private AI is not free and not equal:

  • You pay in patience. A local 27B model answers in seconds, not milliseconds, and long documents take their time. Frontier cloud models are faster and, at the very top end, still smarter.
  • There's a capability ceiling. For cutting-edge reasoning on the hardest problems, the biggest hosted models still lead. The honest framing: local models are now good enough for the 90% of tasks that involve your data — and the cloud ceiling tasks rarely involve sensitive data anyway.
  • You own the ops. If it breaks at midnight, the status page is you. That said, a machine that runs the same software forever is far more predictable than a cloud service that ships changes under you.

My actual posture is a split: anything about me, my family, my clients, my employer, or my documents goes to the local model — full stop. Public, generic questions can go to a hosted API when I want the extra horsepower. The line isn't paranoia; it's just data classification with a router made of common sense.

The Takeaway

Cloud AI made intelligence convenient, and it will keep getting better. But convenience bought with copies of your life is a specific kind of debt — invisible, deferred, and priced by someone else. Running the model yourself costs hardware and patience, and buys the one thing you can't get back once it's gone: the certainty that when you typed it, it stayed here.

— Kris