Building a voice-native OS for the blind
Replacing the screen reader paradigm with a true dialogue.
The problem
Screen readers were designed in 1989 and have barely changed since. They read text aloud as the user tabs through a visual interface — an experience optimized for sighted users who happen to also want audio. The result for blind users is a constant mismatch between what they hear and what they want to do.
We asked: what if the computer had no screen at all? What if the only interface was conversation? What would that OS look like, and could it be more productive than the screen-and-screen-reader combo for daily tasks?
This is a 10-month, ongoing open-source project. It is the hardest engineering challenge we have ever taken on, and the one we are most proud of.
Approach
- ▸Wrote the conversational shell in Rust for sub-100ms response latency
- ▸Two-stage intent router: a 4B Nemotron-3 Nano classifies every utterance, then dispatches to either the 8B Llama 3.1 reasoning model, a deterministic system handler, or a fallback skill
- ▸Self-hosted AetherFS semantic file index (gRPC + Qdrant + FTS5) so "find the invoice from Acme last Tuesday" works without the user opening Finder first
- ▸Spatial audio mixer with HRTF for natural-sounding TTS — a notification from the right side of the screen arrives from the right headphone
- ▸Barge-in support: the user can interrupt the assistant mid-sentence without waiting
Key technical decisions
On-device inference, not the cloud
A blind user should not lose their computer when their internet drops, and should not have their private conversations sent to a third party. We sized everything to fit on a single consumer GPU (GTX 1080 + 1650S + CPU).
Rust for the hot path, Python for the orchestration
The audio capture, VAD, intent routing, and audio output are all in Rust (Tokio async). Python handles higher-level skills where the ecosystem is richer (browser automation, email, calendar). The IPC is gRPC over a Unix socket.
Two-stage intent classification
We tried a single monolithic router first. It was slow and confused. Splitting into "classify intent" → "dispatch" gave us a 3x speedup and made the system debuggable. The classifier is a 4B parameter model that runs in <50ms.
Conversation anchors for retrieval
When AetherFS returns search hits, each result includes a synthesized "spoken anchor" — a one-sentence summary tailored to the user's context. So instead of "PDF file, 2.3MB, modified Tuesday", the user hears "Invoice from Acme, last modified Tuesday evening during your active hours."