Prompts with a
regression scoreboard.
A prompt has an input and output contract and usually gets treated as loose text: tested in chat, saved in a note, forgotten. When the model is updated months later, the degradation goes unnoticed for lack of a baseline. Here the prompt, the cases and the assertions live in the same file.
Sections on this page
Narrative
A prompt in production has no baseline. It works when it is written, the model changes months later, and the drop in quality shows up as an impression in whoever reads the output. With no recorded case and no verifiable assertion, there is no way to separate model degradation from a change in what is expected of it.
Each lens is a YAML file with four parts, matching the competencies of the AI Fluency framework: task for what is delegated and what stays with the human, versions for the prompt with its history preserved, checks for the assertions and provenance for model, reviewer, date and the limits on how the output may be used.
The provenance.nao_decide field is mandatory and the parser refuses a lens without it. It records the limit on how the output may be used, which is the first piece of information to disappear when a prompt circulates through a team. Fixtures recorded per version and per case let the scoreboard run in CI with no key and no cost.
The segment naming lens has three versions in the file, and the difference between them is verifiable. v1 asked for the name without specifying a format, and the model returned a prose title in two of the five cases. v3 declares the output format in two lines and passes on all of them. CI locks the regression in both directions.
Version history, executable
The three versions of the segment naming lens stay in the file, and all of them stay runnable. The difference between them stops being a memory and becomes the result of a command.
Harness runs
✓ a v3 precisa passar
The obvious direction of the lock. If the current version stops passing, something changed: the prompt, the assertion or the model.
✕ a v1 precisa continuar falhando
The direction almost nobody writes. If v1 stops failing, the formato_snake_case assertion broke, and the scoreboard would start reporting green for everything with nobody noticing.
Summing code checks and model judge checks into a single number gives the result a precision the second half does not have.
The anatomy of a lens
Four parts in the same file, matching the competencies of the AI Fluency framework. The prompt without the cases is loose text, and the cases without the usage limit are an invitation to use the output for what it does not serve.
# 1. o que se delega e o que fica com a pessoa
task:
delega: propor nome de uso interno a partir da definição do filtro
humano_decide: aprovar o nome e publicar no dicionário de segmentos
# 2. o prompt, com o histórico preservado e executável
versions:
- id: v1 # sem formato declarado
- id: v2
- id: v3 # formato de saída declarado em duas linhas
# 3. as asserções, com a família explícita
checks:
- nome: formato_snake_case familia: codigo
- nome: so_cita_entrada familia: codigo
- nome: sem_antitese familia: codigo
- nome: justifica_a_escolha familia: juiz
# 4. de onde veio e até onde vale
provenance:
modelo: <id do modelo avaliado>
revisor: Danilo Espeleta
data: 2026-01-14
nao_decide: não substitui a aprovação do dicionário de segmentos ← obrigatórioThe five lenses
One per recurring operational task. Each has an input and an output contract, and that contract is what the assertions verify.
| Lens | Takes | Returns |
|---|---|---|
| nomear-segmento | the technical definition of the filter | the internal name, with a rationale |
| diagnostico-queda | the time series of a metric | hypotheses ordered by cost of testing |
| copy-por-estagio | a lifecycle stage and a value proposition | variations, with the hypothesis each one tests |
| leitura-cancelamento | open ended answers from a churn survey | reason axes with counts and quotes, restricted to the text |
| auditoria-tom | the numbered messages of a flow | where the register breaks between them |
The scoreboard does not sum the families
The two families measure things with different reliability, so the report prints them separately and labelled. Numbers for one lens, on the current version:
Deterministic, reproducible and free of API cost. Eight checks registered in checks.py cover format, containment and text patterns.
Uses a model to judge what form cannot reach. It carries the judge's own bias, which is why it appears labelled rather than diluted into the total.
the report does not sum the two families into a single score
Fails any number in the output that is absent from the input. It covers the most expensive failure mode in CRM analysis, which is the model producing a plausible value and nobody checking it.
Fails the "X is not A, it is B" pattern. At high density it works as a signature of model generated text, and the rule landed here after showing up in a review of my own writing.
Limitations
The assertions cover form, and most of a lens's value sits in the content. contagem_itens verifies that four hypotheses came back, without assessing whether they are good hypotheses. The model judge covers part of that distance, with the bias the family label makes visible.
Fixtures age. An output recorded six months ago tests the assertions against the behavior of a model that has changed, and re-recording is a manual operation.
The suite
Three repositories on the same thesis: a customer journey is a versionable artifact. If it can be declared in a file, it can be read, audited and reviewed in a pull request, with history and peer review. Today it lives inside the automation tool's interface, where there is no diff between versions and no record of why that delay is 48 hours.
The output of one is the input of the next, which makes the agent's acceptance criterion objective: the proposed journey has to pass the linter.
Reads journeys declared in YAML and flags the defect before publication, including the one that only exists between two flows.
Reads product events and returns the segment brief with the proposed journey, already validated by the linter.
Holds the versioned reading prompts, with verifiable assertions and a scoreboard that measures regression in CI.
This page