Norla
⌘K
Bag Orders
← Back to the Academy

NORLA ACADEMY / RESOURCE DETAILS

Practical Agent Evaluation

Norla Editorial3 hours$79 USD
$79 USDPay on this website — the lessons open on your account as soon as payment is recorded.
Buy this course · $79

Build a small, inspectable evaluation process for one agent workflow, including inputs that should trigger clarification or refusal to proceed.

Before you start

  • Access to one prompt or agent workflow
  • Permission to use the selected test inputs
  • A spreadsheet or similar review tool

What you will learn

  • Define observable acceptance criteria
  • Create representative and edge-case tests
  • Compare two workflow versions
  • Write a release decision with explicit limitations

Final project: A 20-case evaluation workbook, scoring rubric, failure log and one-page release recommendation.

Included scope

  • Written lessons, exercise instructions and reusable worksheet structures. Model usage is not included.

These notes describe the scope. Saving a selection does not provision a system or collect payment.

Your learning path

Module 1 is free to read · buy the course to unlock all 4
01 / Define the job before scoring the answer

Start with a narrow workflow such as turning an approved campaign brief into a draft email. Identify the input fields, the intended user, the required output and the actions the workflow must not take. A description such as produces good copy is too broad to evaluate consistently.

Translate the job into observable checks: preserves the approved offer, includes the required disclosure, asks when a required field is missing, and produces a draft rather than sending a message. Separate hard failures from stylistic preferences so a pleasant tone cannot hide a factual error.

Practice: Write six acceptance criteria for one workflow and classify each as required, contextual or stylistic.

Keep: A workflow contract and scoring rubric.

02 / Build a test set with difficult edgesUnlocks after purchase
03 / Inspect failures and calibrate reviewersUnlocks after purchase
04 / Make a bounded release decisionUnlocks after purchase
CASE WORKSHOP / APPLY THE LESSON

Evaluate a source-bounded support draft

Practice brief: A support team is considering an assistant that drafts answers from approved product guidance. Build the course's 20-case evaluation workbook around applicability, missing evidence, conflicting guidance, and unauthorized requests. The final recommendation should specify a supervised scope and unresolved risks rather than promise a universal accuracy level.

Work through the case

  1. Define the intended answer format and acceptance rubric before constructing cases. Include supported questions, absent sources, outdated guidance, contradictory records, and requests that exceed the assistant's permitted role.
  2. Create 20 cases with expected behavior and a rationale for their inclusion. Have two reviewers inspect a selected subset, then resolve rubric ambiguities without quietly removing difficult cases from the evaluation.
  3. Compare two workflow versions, preserve failure severity and disagreement notes, and prepare the one-page release recommendation. Name any failure that blocks use even if the aggregate result appears favorable.

Review your result

  • Each case tests a stated behavior and has enough context for independent review. The set includes consequential edge cases rather than only convenient answerable questions.
  • The scoring record distinguishes a supported answer from an answer with a merely decorative citation, and preserves limitations that affect applicability.
  • The release decision uses case-level evidence, specifies oversight, and identifies the next verification step. A small evaluation is not represented as comprehensive assurance.
Buy the course to download the workbookReturn to catalogue

NORLA EDITORIAL / FIELD NOTES

Continue exploring.

Working methods, decisions to document, and useful questions to take into your next project.

All field notes