Add criteria and check packs

Each step lists its criteria: what the screen has to meet. Kaloko knows three kinds, and a good scenario uses the cheapest one that can answer the question.

Countable: a deterministic check #

Anything you can count or read from the page is checked in the page itself, with the evidence attached:

criteria:
  - id: AC1
    text: The cart shows two items.
    check: { type: deterministic, assert: count, selector: ".cart-item", min: 2, max: 2 }
  - id: AC2
    text: Paying ends on the thank-you page.
    check: { type: deterministic, assert: url_matches, pattern: "/thanks$" }
  - id: AC3
    text: The Pay button uses the brand colour.
    check: { type: deterministic, assert: style_equals, selector: "button.pay", property: background-color, token: color.brand.primary }

Other checks read text, attributes, the HTTP status, overflow on a phone or a value the page exposes (js_equals). The full list is in checks and packs. A step script never has to leave a marker for a criterion to find.

A question: a semantic check #

What needs judgement goes to the evaluator as a question about the page:

  - id: AC4
    text: The primary call to action is clear.
    check: { type: semantic, question: Does the page show one clear primary call to action?, scope: main }

The evaluator reads a reduced outline of the page and answers with a score. A clear yes passes, a clear no fails, and anything in between goes to a person. Add vision: true to the check when the look is the point (layout, imagery, "looks like the brand"): the screenshot then gets a second look from a vision model. Keep it to the steps that need it.

A person decides: a manual check #

  - id: AC5
    text: Legal approved the wording of the consent.
    check: { type: manual }

A manual criterion stays "not evaluated" until a reviewer decides it on the canvas.

Check packs #

A pack adds a fixed set of criteria to a step, so common checks are not written by hand:

nodes:
  - id: home
    packs: [seo, a11y, perf, console]

To change a limit, write your own criterion with the same id as the pack's (for example AC401 for LCP); yours wins. How verdicts come together is explained in how verdicts are decided.

Kaloko · latest · 2026-10-06