Case Study — Process Leadership

LLM Judge Development Playbook

Role

Content Designer / Content Engineer

Skills

Process design · Cross-functional leadership · Documentation

Context

An internal playbook for building LLM judges already existed, but it was scrappy, incomplete, and outdated — not something teams could actually follow. As more teams started building judges to evaluate their own AI products, each one risked rediscovering the same steps from scratch, and it wasn't clear who should own the work: content, product management, engineering, or data science all had a stake.

Approach

I spotted the gap early and proactively overhauled the existing playbook into a clear, efficient, end-to-end resource: identifying common response issues, defining quality principles, building evaluation rubrics, training labelers, and establishing a golden dataset. As part of that overhaul, I led the discussions that resolved real ownership ambiguity across content, product management, engineering, and data science — winning content a clear leadership role in judge-building.

Key insight

If we didn't fix the process, every new team would hit the same walls. Overhauling the playbook gave our team — and every team after — a solid foundation to build upon to quickly ensure a high-quality customer experience.

Key decisions

Proactively overhauled the existing playbook, recognizing that a scrappy, outdated process would slow down every future team that needed to build a judge.
Put roles and responsibilities in writing across content, product management, engineering, and data science, resolving competing claims over ownership rather than leaving it ambiguous.
Framed content's role in judge-building as a leadership position, not a supporting one — a distinction that shaped how future teams staffed similar work.

Roles & responsibilities (representative)

Judge-building phase	Content	Product mgmt	Engineering	Data science
Identify common response issues	R	C	C	C
Define quality principles	R	C	—	—
Build evaluation rubric	R	—	—	C
Train labelers & label responses	R	—	—	C
Establish golden dataset	C	—	—	R
Build & validate judge	C	—	C	R
Launch & monitor	C	C	R	—

R Owner C Contributor

Representative structure recreated for this portfolio — not a screenshot of the internal playbook.

Who I influenced

Cross-functional peers (product management, engineering, user research, data science)

Led the discussions that resolved competing claims over who should own LLM judge-building, then formalized the outcome in the playbook.

Outcome

The playbook became the company-wide standard for LLM judge-building.
It laid the foundation for other teams across the company building into the business assistant.
Removing the ownership ambiguity sped up the process end to end, letting teams launch judges faster and improve quality for the customer sooner.