Case Study — Conversational AI

Content Principles & Quality Evaluation for Meta's AI Business Assistant

Role

Content Designer — content principles, prompts, LLM judges

Skills

Prompt engineering · LLM evaluation · Cross-functional leadership

Context

Meta's AI business assistant needed to meet a minimum quality bar before its beta launch, with no established process for AI quality evaluation to fall back on. Before this work, responses were inconsistent, there was no way to catch bad output at scale, and trust in the assistant's recommendations was a real concern.

Approach

I defined the content principles that set quality standards for the assistant's responses, then wrote the system prompts that put those principles into practice. I led the accuracy dimension of the assistant's LLM judge system — building a judge that caught hallucinated numbers, recommendations that didn't match real account data, self-contradictions, and misdiagnosed issues, validated against a golden dataset I helped establish.

Key insight

Defining a quality principle and enforcing it are two different disciplines. The same practitioner who wrote the standard also built the mechanism that checks whether the model actually met it — closing a gap that usually falls between separate teams.

Key decisions

Proactively built an internal playbook when none existed — judge-building was new to the company, and without a documented process, every team would waste time rediscovering the same steps. The playbook became the blueprint other teams used to build their own judges, well beyond the original project.
Resolved real ownership ambiguity as part of that playbook, establishing content as a clear owner of judge-building alongside defined roles for product management, engineering, and data science.

Who I influenced

Cross-functional peers (product management, engineering, data science)

Led the discussions that resolved competing claims over who should own LLM judge-building, then formalized the outcome in the playbook.

Outcome

The beta launched ahead of schedule.
The playbook became the company-wide standard for LLM judge-building.
Improved response quality contributed to increased entry points and user adoption.