Even with content principles in place and system prompts written to operationalize them, there was no scalable way to verify that the assistant's responses consistently met those standards before its beta launch. Prompt improvements could address individual problems, but nothing caught inconsistent quality at scale.
I recognized the gap and proposed closing it myself, leading development of the accuracy dimension within the assistant's LLM judge framework. I identified the failure modes that mattered most for a business-facing assistant — hallucinated numbers, recommendations disconnected from real account data, self-contradictions, and misdiagnosed issues — and translated them directly into evaluation criteria. To make sure the judge itself was reliable, I helped establish a golden dataset of labeled examples and validated the accuracy judge against that benchmark rather than relying on ad hoc manual review.
The same quality standards that guided generation also powered evaluation — creating a closed feedback loop between design and measurement, instead of two disciplines operating independently.