The goal

Build the financial and architectural case for a sub-$200 user-owned hardware box that runs a complete sovereign data stack — mail server, document store, local LLM inference, on-device differential-privacy aggregation for opt-in data exchanges — at residential power budgets. Today the comparable Jetson-based bill of materials lands at ~$400 (personal-ai-computer-economics.md). The thesis here is that the same capability ships at half the price within 24 months and the next-tier capability ships at the same price within 36, driven by three compounding curves: model compression, NPU integration in budget SBCs, and on-device DP libraries reaching production maturity.

The decision this article informs: how aggressively to plan for an on-device sovereignty endpoint vs. continuing to invest in cloud-hosted infrastructure for the same workloads. The recommendation: design new infrastructure as if the on-device endpoint is real in 2027 and load-bearing by 2028.

Why this matters now

A separate analysis (data-sovereign-email.md) shows that Google's Limited Use Policy categorically blocks third-party transfer of Gmail-derived data — even aggregated, even with explicit consent, even with revenue share. That structural blocker forces Claims Genie's data-brokerage program (Terms §10 / Privacy §10a, ~90% revenue share to users) into a hybrid Paid-Question architecture with manual per-query approval for any Gmail-derived contribution. The architecture is correct given the constraint. Removing the constraint requires removing the dependency on Gmail — which means the user owns the mailbox, which means hardware they own runs the stack. The economics of that hardware are the gating question.

Today's baseline (April 2026)

Three things to notice in this table that change everything in the next 24 months:

1. The genuine sub-$200 tier is already here, just barely. Orange Pi 5 Plus at $130 with 16GB RAM and a 6 TOPS NPU runs Phi-3.5-mini (3.8B) at usable speeds. Pi 5 + Hailo-10H at $210 brings 40 TOPS INT4 to the door but is ~5% over the budget.
2. The Jetson Orin Nano Super at $249 is the price-performance leader by a wide margin — 67 TOPS, 21 t/s on a 7B model at 15W. The bare module at $199 in volume drops it firmly under $200 for an OEM build.
3. The Apple ceiling matters as a quality reference. $599 for an M4 Mac mini gets you an 80–120 t/s 7B inference machine in a sealed appliance form factor. That's the upper bound the budget-tier path is trying to approach.

Sources: SBC LLM benchmark paper, arxiv 2511.07425; NVIDIA Jetson AI Lab; Hailo-10H launch; Tom's Hardware Jetson Super coverage; Compute Market M4 review.

The compression curve (the load-bearing argument)

A claim that a $100–$200 box in 2027–2029 can run "useful" workloads requires evidence that "useful" is getting cheaper to produce per parameter and per byte of memory. Three curves compound here.

Quality per parameter (distillation + better training)

A 4B model in April 2026 matches an 8B–13B model from 2023 on standardized benchmarks. That's roughly 3–4x compression of quality-per-parameter every 12 months, sustained over 32 months. Sources: Qwen3 Technical Report arxiv 2505.09388; Phi-4 Mini benchmark guide; Microsoft Phi-3.5-mini HF model card.

Bits per parameter (quantization breakthroughs)

Q4_K_M is the workaday baseline (≤1 pt MMLU drop vs FP16). The big jump is BitNet b1.58 ternary: the official microsoft/bitnet-b1.58-2B-4T model hits MMLU 52.1 — matching Gemma 2B (51.8) and beating INT4-quantized Qwen2.5-1.5B — using ~0.4 GB of memory and running on CPU at hundreds of tokens/second. AQLM, AWQ, GPTQ, EXL2 are production-ready in the Q2–Q3 range. Sources: BitNet arxiv 2504.12285; Microsoft BitNet GitHub; HF 1.58-bit blog.

That's another 4–8x reduction in memory per parameter on top of the quality gains.

Active parameter ratio (Mixture-of-Experts)

DeepSeek-V3 (671B total / 37B active = 5.5%) and DeepSeek-V2 (236B / 21B active) prove that MoE delivers near-dense quality at a fraction of active parameters at inference time. Mamba and Mamba-2 give 4–5x higher inference throughput than transformers of equivalent size with linear-time long-context scaling and competitive quality up to ~3B. Sources: DeepSeek-V3 arxiv 2412.19437; Mamba arxiv 2312.00752.

Compounded against the 2026 baseline: - Quality per parameter: 3–4x/yr observed → over 3 years, ~30–60x conservatively - Bits per parameter: 4–8x via BitNet ternary - Active-param ratio: 4–18x via MoE for compatible workloads - Inference engine speedups: MLX vs llama.cpp = 1.2–1.9x; llamafile vs Ollama = 3–4x with 30–40% lower power on Pi 5

A 100x compression claim over 3 years is conservative on quality-per-parameter alone (it implies just 4.6x/yr against the observed 2023–2026 trendline). 1000x is plausible only if BitNet-style ternary becomes the default training regime and MoE-2B-active-300M architectures ship at scale. Both are achievable by 2028.

Tokens per joule and cost at home

Residential rate $0.15/kWh, Q4 7B-class workloads:

Crossover analysis:

Local on Jetson at $0.03/Mtok is already cheaper than every hosted API for input-heavy workloads. Even the cheapest hosted option (Gemini Flash-Lite at $0.10 input) loses to a Jetson if the duty cycle exceeds ~1 hour/day. The bigger crossover is opportunity cost: a 2026 7B local model is roughly Claude Haiku 4.5 quality on classification and structured-extraction tasks, so the cost-per-decision is comparable. For email triage at ~30 inferences/day per user, hosted is ~$0.001/user/day; local amortizes against a one-time hardware cost on a roughly 12–18 month payback window for a $200 box at expected savings.

Sources: pricepertoken.com Claude Haiku 4.5; IntuitionLabs API comparison 2026; Strix Halo guide.

The on-device differential privacy workload

This is the workload that matters for the data-brokerage thesis. A law firm asks a question like "roughly how many people in California purchased Tom's toothpaste in 2024?" The Claims Genie hybrid architecture today routes that through a privacy filter that aggregates user contributions with k-anonymity, suppression, and bucketing. The sovereign architecture pushes that further: each user's box contributes a differentially-private noised count locally, and a secure-aggregation protocol combines them so no individual contribution is recoverable.

Libraries usable on a $200 box today: - Tumult Analytics (joined OpenDP umbrella Oct 2025, production at IRS / Census / Wikimedia) — the high-level PySpark-backed query API. - OpenDP (Harvard/Microsoft) — the Rust+Python core. Use directly if no Spark. - Google Differential Privacy library — C++/Go/Java; PyDP wraps it. - Apple pfl-research — open-source local DP for federated analytics; the same primitives Apple deploys at hundreds-of-millions scale for emoji popularity, Safari trends, Apple Intelligence aggregate analytics.

Compute cost: DP aggregation queries (sum, count, histogram, quantile) are microseconds-to-milliseconds on any modern CPU. Laplace/Gaussian noise generation and clipping are trivially cheap. The expensive primitive is secure aggregation (additive secret sharing across users), which is bandwidth-bound, not compute-bound. Apple's Samplable Anonymous Aggregation primitive (paper) shows the whole pipeline runs on iPhone-class hardware once per day with milliwatts of compute.

Hardware confidential compute in the sub-$200 range: ARM TrustZone is on every Cortex-A board (Pi 5, RK3588, Jetson). NVIDIA Jetson confidential compute is AGX-class only. Apple Secure Enclave is in every M-series Mac mini ($599+). Intel SGX is deprecated; AMD SEV-SNP is server-only. Practical answer: TrustZone on RK3588/Jetson is the sub-$200 root of trust — enough for sealed-key DP randomness, not enough for full attested enclaves. Users who need attestation pay the Mac-mini-or-up tax.

The implication for the budget tier: DP queries are essentially free in compute terms. They don't change the box's BOM or power budget. The constraint is bandwidth (for secure aggregation, ~kB/query/user/day) and operational maturity of the libraries — both of which are 2026-ready.

Storage economics

Tom's Hardware April 2026 SSD pricing shows 1TB NVMe at $142 cheapest, $194 median, $235 Gen5 average — reflecting an AI-driven NAND shortage that pushed Crucial P310 from $69 to $107–138 over 6 months. M.2 2230 (the Pi 5 / Jetson dev kit form factor) carries a ~30% premium.

dm-crypt overhead is 5–10% sequential, 10–20% random; ZFS encryption is similar. A 5-year mailbox + document store (10K emails, 1K PDFs, embeddings, model weights) lands at 50–150 GB — leaving 850 GB on a 1TB drive for vector indices, snapshots, and the activity log. Storage is not the binding constraint. Even at the AI-shortage prices, a 512GB NVMe at ~$80 covers the use case for the lifetime of the device.

The 3-year extrapolation

Compounding the curves above against the 2026 baseline, the realistic capability of a $100–$200 box per year:

2026 (today):
  Hardware: Orange Pi 5 Plus 16GB ($130) or Pi 5 + Hailo-10H ($210, ~5% over)
  RAM: 8–16 GB
  Models that fit comfortably: Phi-3.5-mini (3.8B Q4), Qwen2.5-3B Q4
  Practical workloads:
    - Email triage, classification, merchant/amount/date extraction
    - Receipt OCR (with layout-aware preprocessing)
    - Basic conversation (chat with your data)
    - Whisper-small STT (near-realtime)
    - Piper TTS (realtime)
    - DP aggregation queries (trivial)
  Limits:
    - Document understanding for EOBs, complex tax forms (3B floor unreliable)
    - Multi-step reasoning (tax optimization, dispute strategy)

2027:
  Hardware: RK3688-class or Pi 5 + AI-HAT successors at $130–180
  RAM: 8–16 GB; LPDDR5X starts displacing LPDDR4X at this tier
  Models: 4B Q4 at 10–20 t/s OR 2B BitNet at 60+ t/s
  Practical additions:
    - Reliable structured-JSON extraction from EOBs
    - Calendar parsing, deadline tracking
    - Two-document compare-and-reconcile
    - Federated DP queries with secure aggregation pilots

2028:
  Hardware: 16–24 GB unified memory; 12–15 TOPS NPUs in budget tier
  Models: 8B BitNet OR 8B-MoE-active-1.5B at 15–25 t/s
  Practical additions:
    - Full document understanding (EOBs grade)
    - On-device DP aggregation deployable for 100K-user cohorts
    - Whisper-medium local
    - Voice cloning STT/TTS
    - Multi-document tax-form reasoning

2029:
  Hardware: 24–48 GB RAM; 30–60 TOPS NPUs in the budget tier
  Models: A 14B-2026-equivalent fits in 2 GB at BitNet quality
  Practical additions:
    - Tax preparation end-to-end
    - Claim form auto-fill across state databases
    - Multi-document reasoning over 5-year archive
    - Agentic email handling (compose, reply, schedule, dispute)
    - Federated DP brokerage queries at the millions-of-users scale
  Capability ceiling: A 2029 4B model should reach Claude Haiku 4.5 / GPT-4o-mini
  territory on most non-frontier tasks. Drawing 8–15W. On a $150 box.

The recommendation

Build new sovereignty-relevant infrastructure as if the on-device endpoint is real in 2027 and load-bearing by 2028. Concretely:

1. Treat hosted AI inference as the bridge, not the destination. Every new pipeline that ships in 2026 should be designed against an interface that can be served by either a hosted API (today) or a local Ollama / MLX endpoint (within 12–18 months). No proprietary cloud features that don't have a local-runtime equivalent.

2. Invest in DP libraries now even though the brokerage is small. Tumult Analytics + OpenDP are production-grade today and shape the design of every aggregate query the platform answers — the on-device version uses the same primitives. Any aggregate analytics built without DP-readiness now will need to be rebuilt later. The compute cost of running DP locally is negligible.

3. Pick mail and storage stacks that already run on edge ARM hardware. Stalwart, Mox, Postfix all run on Jetson and RK3588 today. SQLite + flat-file storage (the Claims Genie pattern) trivially moves to a home box. Avoid Postgres-only or cloud-managed-state designs for any new entity that should eventually live on the user's hardware.

4. Plan a "Year 2 hybrid → Year 3 sovereign" migration path now. Year 1 (now): cloud relay + custom domain + KYC quality score. Year 2: user-owned mailstore (small box at home for IMAP storage) with our cloud relay. Year 3: full sovereignty with our cloud as optional update repo and outbound-relay-of-last-resort. This is detailed in the companion build direction document (agentic-tasks/sovereign-data-stack-direction-2026-04-22.md when it lands in the claims-genie repo).

5. Avoid the Helm trap. Don't bundle a per-user infrastructure subscription you can't sustain over the device's lifetime. The right shape is closer to Yunohost or Start9 — open-source OS, hardware-agnostic, user owns the relay relationship — than a vertically-integrated appliance. Hardware margins on consumer electronics are brutal; the business has to be in the value the box generates, not the box itself.

Open questions

1. Will BitNet-style training go mainstream? Microsoft's BitNet is a strong proof-of-concept, but the model zoo at 2B BitNet quality is still small in April 2026. The 1000x compression projection depends on this. Worth tracking new BitNet-trained model releases monthly through 2026–2027.
2. NPU-LLM toolchain maturity for RK3588 and successors. RKLLM is workable but not as polished as CUDA on Jetson. If RK3576/RK3688 deliver good quantized inference and the toolchain matures, the Orange Pi tier becomes the dominant budget answer ahead of schedule.
3. Residential bandwidth for federated DP secure aggregation. Apple does this on cellular phones at hundreds of millions of devices, so the protocol works. Whether a residential ISP terms-of-service stomach a million-node aggregation cohort is an open practical question.
4. The price floor for KYC and identity attestation. Stripe Identity at $1.50/verification (first 50 free) and Persona at quote-based ~$0.30–$1 at volume mean per-user attestation is materially cheaper than the device. Worth confirming this scales as the user base grows beyond the cheap tier.
5. Apple Mac mini M4 as a parallel path. At $599, the M4 Mac mini is 3x the budget but ~4x the performance, runs MLX, has Secure Enclave for attestation, and is the easiest "appliance" form factor available. Some users will pay the premium for the polish — and the 2027 M5 base may push the price down. The sovereign direction probably wants both a "cheap budget tier" path and a "premium appliance" path, not just the budget one.

Sources and prior research

• personal-ai-computer-economics.md — the $400 Jetson-today BOM analysis this article extends.
• memex-privacy-vs-availability.md — encrypted-relay and personal-edge-device strategies that compose with the sovereign-mail stack.
• data-sovereign-email.md — companion article on the deliverability/quality-attested-sender layer.
• BitNet b1.58 arxiv 2504.12285, Microsoft BitNet GitHub
• Qwen3 Technical Report arxiv 2505.09388
• DeepSeek-V3 arxiv 2412.19437
• Mamba arxiv 2312.00752
• Apple Differential Privacy overview, Samplable Anonymous Aggregation, pfl-research
• Tumult Analytics → OpenDP, OpenDP tools, Google DP library
• Tom's Hardware SSD pricing tracker 2026
• LLM API pricing comparison 2026 (IntuitionLabs), Claude Haiku 4.5 pricing
• Raspberry Pi AI HAT+ 2 launch, Hailo-10H LLM blog
• NVIDIA Jetson AI Lab benchmarks, Tom's Hardware Jetson Super coverage
• SBC LLM benchmark paper arxiv 2511.07425, LLM-at-edge sustained-load study arxiv 2603.23640
• MLX vs llama.cpp benchmarks, Compute Market M4 Mac mini 2026 review