Project 04 · segment-brief

A proposal that
passes the linter.

The question of which segment to work on takes about a week in almost every CRM operation. The repetitive part of that work is querying and comparing, and that is the part the agent runs. The acceptance criterion stays in code: the proposed journey has to pass lifecycle-lint.

Own project
Tool using agent
Synthetic data

4 tools, released by the router according to intent

2 correction rounds, a cost decision

13 eval cases, ten on routing and three on sessions

17 automated tests in CI

How these numbers are counted: four tools is what tools.py declares, run_sql, cohort_size, rfm and propose_journey, and the router decides which of them the model sees on each run. Two rounds is the ceiling in agent.py. Thirteen cases is the count in evals/cases.yaml, split into two blocks because they measure different things. The seventeen tests run in CI on the scripted provider, with no API key and no cost.

Narrative

Problem

Someone pulls the data, someone crosses it with revenue, someone writes a document, someone disputes the cut. A week later there is a segment brief whose quality nobody can assess without redoing the whole path.

Decision

A deterministic router at the entrance, an agent in the middle, a validator at the exit. The router classifies the question as exploration, diagnosis or proposal from lexical markers, and defines which tools the model sees on that run. An exploratory question cannot emit a journey by accident of phrasing.

Execution

The agent queries DuckDB in read only mode with a 50 row cap, sizes cohorts against a sample floor, computes RFM quintiles and closes with a journey in the canonical format. The YAML leaves the model and enters lifecycle-lint. On failure the findings return to the agent as a tool result.

Learning

The first scripted proposal passed with warnings, which left the correction loop without a demonstration. The agent's validator became stricter than the linter: four rules classified as warnings there fail here. The linter audits flows written by people, where a warning starts a conversation with whoever wrote it. The agent's proposal is automatic, and redoing it costs one round.

Three intents, three scopes

The same base, the same agent, three questions. What changes between them is the set of tools the model sees, and the router decides that, in code. The output stays in Portuguese, the language of the operation the tool was written for.

$ segment-brief ask "Que segmento a gente deveria estar trabalhando para reter melhor? Proponha uma jornada."

intenção: proposta · ferramentas: run_sql, cohort_size, rfm, propose_journey

→ run_sql({"query": "WITH d7 AS (SELECT u.user_id, u.canal_aquisicao, ..."})

50 linhas · afiliados × mensal × aulas na 1a semana

→ cohort_size({"where": "canal='afiliados' AND plano='mensal'"})

n = 3.412 · acima do piso de amostra (500)

→ rfm({"segment": "afiliados_mensal"})

quintis: recência alta · frequência baixa · valor médio

→ propose_journey({"id": "ativacao_afiliados_mensal", ...}) rodada 1/2

⇒ validador: reprovado (2 erro(s), 0 aviso(s))

ERRO L007 Canal intrusivo sem janela de silêncio

ERRO L009 Métrica de sucesso ausente ou de vaidade

→ propose_journey({"id": "ativacao_afiliados_mensal", ...}) rodada 2/2

⇒ validador: aprovado (0 erro(s), 0 aviso(s))

briefing · afiliados × mensal × d7 < 2 aulas · churn 44% contra 28%

jornada: ativacao_afiliados_mensal · holdout 0.1 · success: retencao_d30

exit code 0jornada aprovada

Notice what is missing from the second and third tabs: propose_journey is never even offered to the model. The permission scope is auditable precisely because that decision sits outside it.

An agent that also routes and validates itself returns a result with no verifiable acceptance criterion.

Code, model, code

Routing and validation sit outside the model by design. Step through the six stages to watch the validator's rejection return to the agent as a tool result.

step 1/6

The natural language question enters the pipeline. Nothing has been decided yet.

The permission scope has to be auditable, and the acceptance criterion has to be the same on every run. The agent occupies the middle step, which is where the task is open ended enough to justify a model.

The two round limit is a cost decision. An agent that does not converge with the linter findings in hand rarely converges in ten attempts, and every extra round is token spend on the same doubt.

The router uses lexical markers and gets ambiguous phrasing wrong. Classifying with a model would make the permission scope depend on a non deterministic output, and the trade is not worth it: the cost of the current error is one tool less on the run, and the question can be rephrased.

The planted cohort

The eval is only verifiable because the right answer is planted at a known position. Students from affiliates on the monthly plan who watch fewer than two classes in the first week churn well above the rest, and that is the cut the agent has to find on its own.

44%

28%

31%

22%

affiliates · monthly
d7 under 2 classes other
monthly paid
overall organic
overall

90 day churn · scale 0 to 50% · synthetic base with a versioned fixed seed

A synthetic edtech base, generated by a versioned script with a fixed seed: 20 thousand students, 18 months of events, enrollment seasonality in January, March and August. The risk cohort at a known position is what makes the eval verifiable, because there is a right answer to compare against.

Synthetic data is a requirement here, stated in the README. Real employer data does not go into a public repository.

The validator is stricter than the linter

Four rules that lifecycle-lint classifies as warnings fail here, plus the sample floor, which is exclusive to the validator. The difference is not strictness for its own sake: it changes who receives the finding.

Rule	lifecycle-lint (human written flow)	validator.py (automatic proposal)
L005 · Segment with no time decay	aviso	reprova
L007 · Intrusive channel with no quiet hours	aviso	reprova
L008 · Active journey with no control group	aviso	reprova
L009 · Success metric missing or a vanity metric	aviso	reprova
Cohort sample floor	not present	reprova

rodada 1/2 · propose_journey

Journey ativacao_afiliados_mensal with an intrusive channel, no declared window, and taxa_abertura as the success metric.

✕ reprovado · 2 erros

L007 L009 The findings return to the agent as a tool result.

rodada 2/2 · propose_journey

Quiet hours declared, and the metric swaps opens for retencao_d30.

✓ aprovado · 0 erros

Brief delivered with a 0.1 holdout. If the hypothesis is right, the test will say so, and that is what the holdout is for.

The linter audits flows written by people, where a warning starts a conversation with whoever wrote it. The agent's proposal is automatic, and redoing it costs one round. At that price, the standard goes up.

The ten routing cases

A deterministic block, no model and no cost, running in CI on every push. It checks the classification of the question and the set of tools released. Case 9 is a boundary case: it carries markers of two intents.

#	Question	Intent	Tools released
1	"Que dados temos na base?"	exploracao	run_sql
2	"Quais eventos existem?"	exploracao	run_sql
3	"Quantos alunos ativos por plano?"	exploracao	run_sql
4	"Compare o churn por canal de aquisição"	diagnostico	run_sql, rfm
5	"Diagnostique por que mensais evadem"	diagnostico	run_sql, rfm
6	"Qual o tamanho da coorte de afiliados?"	diagnostico	run_sql, cohort_size
7	"Mostre o RFM dos inativos"	diagnostico	run_sql, rfm
8	"Proponha uma régua para afiliados mensais"	proposta	all 4 tools
9	"Compare churn por canal e proponha jornada para o pior" boundary	proposta	all 4 tools
10	"Que segmento trabalhar para reter melhor?"	proposta	all 4 tools

Why the boundary case matters: a question mixing diagnosis and proposal has to resolve to the wider scope, otherwise the agent would lose the tool it needs mid run. The rule is explicit and tested, rather than emerging from model behaviour.

Two evaluation blocks

Routing · 10 cases

No model

Deterministic. Checks the classification of the question and the set of tools released, including a boundary case carrying markers of two intents.

Sessions · 3 cases

Plumbing or model

On the scripted provider it measures tools, validator and correction loop, and runs in CI at no cost. Against the API it measures the model, and the number obtained holds for that run.

runs/proposta_afiliados.json

# o registro do que o agente fez, e não do que ele diz que fez
{
  "rota": "proposta",
  "ferramentas_liberadas": ["run_sql", "cohort_size", "rfm", "propose_journey"],
  "chamadas": [
    {"tool": "run_sql", "linhas": 50},
    {"tool": "cohort_size", "result": {"n": 3412, "piso": 500, "ok": true}},
    {"tool": "propose_journey", "rodada": 1, "veredito": "reprovado", "achados": ["L007", "L009"]},
    {"tool": "propose_journey", "rodada": 2, "veredito": "aprovado"}
  ],
  "tokens": 4213, "tempo_s": 9.8
}

A scoreboard that sums both blocks says nothing about either one, so the report separates them and prints the label of what is being measured. Each run's transcript is saved under runs/, with route, calls, verdicts, tokens and time.

The suite

Three repositories on the same thesis: a customer journey is a versionable artifact. If it can be declared in a file, it can be read, audited and reviewed in a pull request, with history and peer review. Today it lives inside the automation tool's interface, where there is no diff between versions and no record of why that delay is 48 hours.

The output of one is the input of the next, which makes the agent's acceptance criterion objective: the proposed journey has to pass the linter.

01 · Audit

lifecycle-lint

Reads journeys declared in YAML and flags the defect before publication, including the one that only exists between two flows.

02 · Propose

segment-brief

Reads product events and returns the segment brief with the proposed journey, already validated by the linter.

This page

03 · Read

lentes

Holds the versioned reading prompts, with verifiable assertions and a scoreboard that measures regression in CI.

Stack

Python 3.10+
DuckDB
LLM API
lifecycle-lint
pytest
GitHub Actions

View the repository

Sections on this page