Teaching an LLM to classify customers' AI assistant conversations

 

Background

MAVEN is the AI assistant inside The Martec, an employer branding platform. Customers use it to analyse their data, create briefs for employee-generated content and turn existing stories into posts.

Context

We were positioning MAVEN with enterprise customers using only anecdotes. The only way to know what customers did was to read conversation logs one at a time. Rather than wait for engineering capacity, I built a classifier myself.

Process and actions

I worked in Claude Code with Superpowers skills, from brainstorm to spec, plan and test-driven build, landing as two pull requests reviewed by our engineers.

  • Define the categories. Drawing on close customer knowledge and spot reviews of past conversations, I defined three categories:

    • Intelligence: insight, analysis or planning from the customer's own data

    • Collection: preparing new questions for employees to answer as part of employee-generated content

    • Creation: making publish-ready content from material that already exists

  • Create a human answer key. A classifier is only as good as what you check it against. I built a labelling interface as a Claude artifact and hand-labelled 40 real conversations.

 

A mock up of the labelling interface. Conversation text never left the browser during labelling.

 
  • Calibrate the inputs and definitions. A category passed with 90%+ agreement against my 40 labels and no more than 10% "unsure" scores, so the model couldn't pass by hedging. I stopped at 90%, because chasing 100% on 40 conversations would overfit to this sample.

    • The category definitions: where the model disagreed with me, I found out why and rewrote the definition. For example, it initially sorted on surface words like "story", so I added one deciding question: does the source material already exist (Creation), or is material being asked for (Collection)? In one round, Collection rose from <80% to 95%.

    • What the model reads: the whole conversation, including MAVEN's replies, because short requests were ambiguous on their own.

  • Choose the model and the output structure. My first pick, OpenAI's Decisions API, passed calibration and returned the probability that a conversation matches each definition. Engineering review flagged that it was OpenAI-only and couldn't follow MAVEN to Amazon Bedrock for EU data residency, so within a day I rebuilt the scorer to be provider-neutral:

    • Which model: gpt-5.4-mini missed the target and swung between runs; gpt-5.4 passed, and the scorer can switch to a Bedrock model later.

    • How it scores: five fixed scores that map cleanly to no, unsure and yes, with a short reason before every score, which keeps answers steady.

 

Three options against three tests

 

Outputs & Impact

  • Usage insights by customer: each category's share of conversations and its trend over time, built for the dashboard our CS team sends each account so customers see where they get the most from MAVEN.

  • Expansion signal: CSMs see where usage is strong and where there's room to grow.

  • Accuracy: Intelligence 100%, Collection 100% and Creation 95% agreement with my 40 labels, against a 90% target.

  • Calibration tool: anyone can rerun it without database access, and it shows the model's reasoning for every disagreement.

  • Speed: a few hours of hands-on work, with a working engine the same day as the design spec.

What I'd validate next: which views customers actually act on, and whether accuracy holds with a second labeller and held-out conversations.

Evals are product work: writing the definitions and setting the pass bar were the real product decisions.