Skip to content

12.5 The Certification Lab: Build the Evidence Pack

This is the chapter’s main build. You will take a bounded problem from brief to review and produce the kind of evidence that makes a scenario answer believable. The lab can be run with a real SDK in a disposable project or completed as a fully disclosed simulation. Tool access is not the competency. Sound decisions, failure detection, and honest evidence are.

What you will learn

  • Write a vision brief with observable done-means.
  • Build a small agentic workflow around synthetic data.
  • Evaluate both valid and invalid outputs.
  • Capture a failure, correction, reviewer response, and residual limitation.
  • Package the work so another person can inspect or teach from it.

Builder principle

A lab is complete when another person can tell what happened, what was tested, what failed, and what remains unknown.

The scenario: Community Workshop Digest

The Tech Hustle is preparing a digest for a fictional community workshop. The agent receives six synthetic event records and drafts a concise internal review document. It must not send email, publish, register attendees, or make claims not supported by the fixture.

Done-means

  • Every event with status: confirmed appears once.
  • Draft contains date, time zone, location or “online,” organizer, and source ID.
  • Draft labels missing information instead of inventing it.
  • High-risk or ambiguous records are surfaced for human review.
  • The workflow never sends an external message.
  • A reviewer can reproduce the result from the fixture and the documented steps.

Not today

  • No real calendar, inbox, CRM, social account, payment system, or attendee data.
  • No automatic publication or message sending.
  • No claim that the digest improves attendance, revenue, or career outcomes.
  • No broad retrieval across a personal drive or organization workspace.

Required evidence pack

Create a private folder such as certification-lab/ with:

certification-lab/
├── README.md                 # scope, route, claim boundary, how to review
├── VISION.md                 # user, problem, done-means, not-today
├── CONTEXT.md                # stable project facts and task context
├── TOOLS.md                  # contracts, permissions, alternatives
├── TRUST_MATRIX.md           # data flow and approval boundaries
├── EVAL.md                   # cases, expected outputs, results
├── BUILD_LOG.md              # decisions, attempts, costs, failures, revisions
├── REVIEW.md                 # independent reviewer findings and disposition
├── RESPONSIBLE_BUILD.md      # data, provenance, people, disclosure, abuse
└── fixtures/
    ├── events.json
    └── expected-digest.json

Your implementation or simulation may have additional files. Keep the evidence folder readable and do not put credentials, personal information, or provider secrets inside it.

Lab sequence

1. Write the vision before the prompt

Name the user, the current problem, the decision the artifact supports, the boundaries, and observable acceptance criteria. If “good digest” cannot be tested, the lab is still an idea.

2. Classify the data

Use synthetic records and write why. If you later use public event data, record provenance and license/permission. If you use real organizational data, stop and obtain the required authorization and controls before sending it to a provider.

3. Define the narrowest tool surface

The initial version needs one read-only operation: return the supplied fixture. The summarizer may be a model, a deterministic formatter, or a peer simulation. The evaluator is deterministic where possible. Sending and publishing capabilities are deliberately absent.

4. Design context

CONTEXT.md should include:

  • purpose and audience;
  • fixture location and field definitions;
  • output requirements;
  • prohibited actions and approval boundary;
  • source/provenance rules;
  • commands or simulation steps;
  • known limitations and escalation path.

Do not fill persistent context with secrets, stale temporary decisions, or a giant copy of the input data. Keep task-specific details close to the task.

5. Make a first attempt

Run or simulate the workflow. Save the initial output and the evidence mode:

Evidence mode: live / simulated / delegated / mixed
Agent or tool surface:
Input fixture version:
Output version:
Human decisions made:
Machine checks run:
Not observed:

6. Seed a harmless failure

Choose one controlled defect:

  • remove a required time zone;
  • duplicate one event;
  • insert an unsupported organizer claim;
  • change a status from confirmed to draft;
  • return a malformed date;
  • inject an instruction into an event title.

The point is to test the evaluator and reviewer, not to trick a person unfairly.

7. Verify and revise

Your evaluation set should contain valid, invalid, ambiguous, and adversarial-but- synthetic cases. Record expected behavior before looking at the result. For each case, classify:

  • pass: output met the acceptance criteria;
  • fail: output violated a criterion;
  • false positive: system rejected acceptable content;
  • unknown: result or test evidence is missing;
  • blocked: authorization or environment prevented the test.

Never convert “the model sounded confident” into pass.

8. Ask for independent review

Give a reviewer the brief, fixture, output, evaluation results, and review contract. Do not coach them through the answer. Ask them to return:

  1. one observed strength;
  2. one unsupported claim;
  3. one safety or provenance concern;
  4. one missing test;
  5. one recommended next change;
  6. ACCEPT, REVISE, or REJECT for the lab’s stated scope.

If the reviewer cannot reproduce the evidence, record that as a delivery failure.

No-code lane

Use a spreadsheet, document, or static table for the fixture and expected output. You may play the role of the model yourself, but label the record as simulated. Use a checklist to verify required fields, duplicates, unsupported claims, and the prohibited external-send action. Record a screen recording or annotated screenshots only if they contain no private data.

Engineer lane

Use a disposable repository. Keep the deterministic fixture parser and evaluator separate from the model/agent adapter so you can test each boundary. Add a simple schema validator and a regression fixture for the seeded failure. If you use the Agent SDK, link to the current reference and record the package/runtime version you actually used. Do not assume that a passing unit test covers model quality, data rights, accessibility, or external effects.

Evidence rubric

Use the shared chapter rubric, then add this lab-specific check:

Dimension Strong evidence
Framing The outcome is narrow, observable, authorized, and explicitly bounded.
Architecture Tool, context, model/agent surface, evaluator, and human gate are justified.
Execution Another person can run or simulate the workflow from the README.
Reliability Valid, invalid, ambiguous, and injection-like cases are tested.
Safety No unnecessary write path, secret, sensitive data, or unsupported claim is present.
Review Independent review changes the artifact or clearly records why not.
Transfer Learner explains what would need re-verification in another provider or environment.

Facilitator talking points

  • “What did the lab prove, and what did it intentionally not prove?”
  • “Why is the evaluator separate from the generator?”
  • “Which failure would be most embarrassing to miss, and why?”
  • “What evidence is live versus simulated?”
  • “What is the smallest change that would make this safer?”
  • “Would you approve a write capability now? What evidence is still missing?”

Check your understanding

  • Why must the vision brief come before the prompt?
  • What makes a failure drill fair and useful?
  • What is the difference between a false positive and an unknown result?
  • Which files let a reviewer understand the lab without sitting beside you?
  • What does an independent reviewer add that the agent’s self-report cannot?

Evidence to save

  • Completed certification-lab/ evidence folder
  • Before and after evaluation results
  • Seeded failure and correction
  • Independent review and disposition
  • Cost/time/access notes, with estimates labeled as estimates
  • Claim ledger with no inflation beyond the evidence

Reference path