Shipping a child-safe LLM at scale
Shipping an LLM to children requires re-thinking the safety stack from scratch.
The problem
The client came to us with a working LLM demo and a 6-month-old safety review document. Their first attempt at a kids' product failed an internal review: the model could be tricked into discussing violence with age-inappropriate framing, and the parent dashboard was a buried afterthought.
We were hired to redesign the entire stack — model guardrails, conversation storage, parent controls, and the kid-facing UX — and ship a version that could pass both COPPA and GDPR-K audits while still feeling delightful to a 7-year-old.
Approach
- ▸Built a layered guardrail system: a fast classifier screens every turn before it reaches the LLM, a topic-blocklist layer catches known-bad patterns, and a secondary model reviews the LLM's output before it reaches the child
- ▸Designed a parent onboarding flow that explains the tradeoffs honestly — what Mia will and won't do, what data is stored, how to delete it
- ▸Shipped a parent dashboard with conversation history, daily time budgets, and topic boundaries configurable per child
- ▸Implemented end-to-end encryption for conversation history with parent-held keys
Key technical decisions
Layered safety, not a single gate
A single classifier with 99% accuracy is the wrong architecture for child safety — that 1% becomes 1 in every 100 turns. We run 3 independent layers, each with different failure modes, and any one can block a turn.
Topic boundaries configurable per child
A 7-year-old and a 12-year-old need different guardrails. Parents set per-child policies; the model is told which policies are active at the start of each conversation via a system prompt.
Encryption with parent-held keys
COPPA wants minimal data collection. We store less by encrypting conversation history with a key only the parent's device holds. The servers see ciphertext; even a server breach leaks nothing readable.