Scenario file
A scenario is a YAML file (usually in qa/flows/) that describes one flow: its steps, the arrows between them and what each step must meet. Its format is qawalk.scenario.v1; kaloko validate <file> checks a file against it. How to write one step by step: Write a scenario.
schema: qawalk.scenario.v1
id: WEB-12-checkout
title: Checkout
locales: [en, cs]
viewports: { desktop: [1280, 800], mobile: iphone-15 }
environments: [local, staging]
nodes:
- id: cart
kind: screen
title: { en: Cart, cs: Košík }
path: /cart
purpose: The shopper sees what they are buying and can continue.
execution: { strategy: playwright, script: steps/cart.mjs, instructions: Add one product and open the cart. }
criteria:
- { id: AC1, text: Shows the total, check: { type: deterministic, assert: exists, selector: .cart-total } }
- id: payment
kind: screen
title: Payment
path: /checkout/payment
packs: [a11y]
edges:
- { from: cart, to: payment }
Top-level keys
schema, id, title, nodes and edges are required.
| Key | What it holds |
|---|---|
schema | always qawalk.scenario.v1 |
id | the scenario's id (lowercase letters, digits, dashes); a ticket in it (WEB-12-…) links the issue |
title | a name for people |
kind | task (default) or flow |
archived | true hides a retired scenario from the list; its runs stay reachable |
source | path to the acceptance plan (ACCEPTANCE.md), used by kaloko validate --coverage |
version | a number you raise when the flow changes |
locales | languages to walk, e.g. [en, cs] |
viewports | names with a size ([1280, 800]), a device preset (iphone-15, ipad-landscape), or a list of preset names |
browsers | chromium (default), webkit, firefox |
color_schemes | light, dark |
reduced_motion | no-preference, reduce |
forced_colors | none, active (Windows high contrast) |
display_modes | browser, standalone (an installed web app) |
devices | app scenarios: the emulators, simulators or phones to walk on |
token_modes | design environments: which modes of the token set to render ([brand-a, brand-b]), or false for the environment's default_token_mode only; not set: every mode |
environments | environments of the config the scenario may run on |
readonly | true for a scenario that only reads; required on a read-only environment |
platform | web (default), android, ios, electron, desktop, windows |
health | acceptance: [seo]: packs whose checks count for acceptance in this scenario; without it pack checks are advisory health (Check the health of your app). site: true (or { paths, ai_crawlers, sitemaps, environments }) adds the site checks once per run: robots.txt, sitemaps, llms.txt, redirects (Criteria, checks and packs). design_packs: [console]: code-only packs (seo, aeo, perf, console, errors) that still run on design environments, where they are off otherwise |
traffic | Search Console and GA4 numbers for this scenario's pages; replaces the config's traffic (Configuration file) |
tags | free labels |
refs | requirement references the scenario verifies, as your tracker or QMS writes them ([URS-12, FS-3.4]); they reach every criterion in the trace matrix (quality evidence) |
nodes | the steps (below) |
edges | arrows between steps (below) |
lanes | labels of the rows on the canvas, index = lane number (a link ending in #lane=<number> opens that row) |
issue | the ticket: number, repo (owner/name), key (Jira or Linear, ENG-42), url, title |
sessions | people in the flow, each with title, optional mailbox, account, blocked_on |
project | the Kaloko project for its runs; overrides the config. When the config asks every scenario for its own project (a comment, or a product with projects:), kaloko validate and kaloko start warn when it is missing |
product, module | which product and module of products: the flow shows, for the changelog and docs |
capture | capture quality for every step (below) |
pii | redact or pseudonymize: personal data hidden in every capture, found automatically (how-to) |
Step keys (nodes)
Every step needs id, kind and title.
| Key | What it holds |
|---|---|
id | the step id, used in links, --steps and step references (kaloko:<scenario>/<step>) |
kind | screen, email, external (a third-party screen), decision, stack (many pages of one template) |
title | text, or one per language ({ en: Cart, cs: Košík }) |
path | the address relative to base_url, or one per language; a deep link in app scenarios; {random} makes an address that cannot exist |
purpose | one sentence about what the step is for; the evaluator reads it |
refs | requirement references of the step; they reach its criteria in the trace matrix |
lane | the row on the canvas |
readonly | this step only reads |
execution | strategy (playwright, script, agent, manual), script (path to the step script), instructions (required), requires (e.g. browser) |
mail | e-mail steps: subject and to |
criteria | what the step must meet (below) |
session | which person from sessions does this step |
affects | file globs that shape the step, for kaloko walk --affected, or always |
packs | ready-made sets of health checks: seo, usability, a11y, perf, console, errors, app, aeo, locale, and design, which counts for acceptance |
expect_status | the HTTP status the step should answer with, e.g. 404 (or a list). A status of 400 or more marks an error page: the errors pack checks HC601 (soft 404) and HC605 (the explanation) only on error pages, that is with expect_status or on a 4xx/5xx answer |
capture | capture quality for this step only |
record | true records this step even when the walk does not; false never |
ignore | regions left out of pixel comparisons: { selector } or { rect: [x, y, w, h], viewport }, optionally limited by viewports, locales, with a note |
items | stack steps: where the pages come from (urls, file, sitemap (gzipped too; sitemap: robots or robots: true reads the sitemaps of /robots.txt for the walked locale) with match/exclude/limit, crawl, script). match and exclude are a regex or a map by locale ({ default: '^/eshop/', fi: '^/fi/tuotteet/' }) and apply before limit for every source; rotate: week or rotate: run takes limit pages from a different place each week or run, so a big list is covered in turns; variants: hreflang (with source_locale) takes the list in one locale and walks each page's hreflang alternate in the others, so a page and its translations are one page of the stack and a missing translation is reported (HC1005), in kaloko walk and hosted runs alike; params go to script, which also gets node (id, title, lane, lane_title, session); a script or a JSON list may give each page edit_url (the page in the CMS) and content_ref (its record), and resolve: steps/resolve.mjs (default export (item, ctx) => ({ edit_url, content_ref })) adds them to the walked pages, for the stack index and kaloko export findings; on a design environment the variants are folders of the step, named by urls or found as subfolders with an index.html |
fields | stack steps: up to 12 columns of the stack index (id, label, from: title, h1, description, canonical, lang, words, http, jsonld_types, a meta tag such as og:title or meta:robots, a CSS selector, selector@attribute, count:selector) |
details | named parts of a screen step for guides: { id, name, selector }, optionally context (section, card, page or a CSS selector), padding and states; shown in docs with kaloko:<scenario>/<step>#<id> (Publish living docs) |
states | other states the step is captured in, e.g. [empty, error, loading]; any name works (below) |
Criteria
Each criterion has an id (AC1, AC12b), a text for people, an optional highlight (a CSS selector to outline on the screenshot), optional refs (requirement IDs it verifies, free text) and exactly one check: deterministic, semantic or manual. Every check type and option is listed in Criteria, checks and packs.
environments: [production] limits a criterion to some environments, states: [empty] to some step states. On the other ones it shows as not applicable and does not make the step partial.
Step states
A cart is not one screen: it can be empty, fail to load or still be loading. states: on a step captures each of them next to the default one:
- id: cart
kind: screen
title: Cart
path: /cart
states: [empty, error, loading]
execution: { strategy: playwright, script: steps/cart.mjs, instructions: Open the cart. }
criteria:
- { id: AC1, text: Shows the total, check: { type: deterministic, assert: exists, selector: .cart-total } }
- { id: AC2, text: Says the cart is empty, states: [empty], check: { type: semantic, question: Does the page say the cart is empty? } }
- A walk runs the step script once per state and passes the state as
state('default'first). The script puts the page into that state: an empty cart, a mocked API error, a slow response. - A design step renders
cart/empty.htmlfor the stateempty(per languagecart/cs/empty.html). Without that page Kaloko openscart/index.html?state=empty, so one page can switch its states itself. - Each state has its own screenshots and verdicts. The canvas gets a State switcher next to the language, review mode and the grid show the state you pick, and a step without that state shows its default screen.
Edges
| Key | What it holds |
|---|---|
from, to | step ids |
label | text on the arrow, e.g. the choice at a decision |
kind | main (default) or alt for an alternative path |
Capture
| Key | What it holds |
|---|---|
format | webp (default) or png (lossless, for print) |
scale | 1 or 2 (device pixels of the screenshot) |
quality | WebP quality from 0.3 to 1 |
mask | personal data hidden in every output of the step: CSS selectors, auto (detection), or both; the scenario's and the step's add up (how-to) |
annotate | numbered marks on the elements of criteria with highlight |
full_page | the whole page (default, stack pages too) or, with false, the first screen: several times smaller on long pages |
screenshot | thumb keeps only the small thumbnail (640 px wide, as tall as the captured page) as the step's picture: for stacks of hundreds of pages whose checks read the page, not the picture, the smallest together with full_page: false; default full |
record | true, step, flow, failed or off; see kaloko walk --record |
trace | keep a trace of actions, requests and console per step (default true) |
A step's capture wins over the scenario's, and the scenario's over the config's; masks are the exception and add up.