All guides › By use case
By use caseRegression before a release: compare the run with the accepted baseline
Run the flows against staging, compare with the baseline run the team accepted, and see what moved: step statuses, criteria, evidence, pixels.
Three agents, four branches, one staging, zero memory of what changed. Before a release we want one answer: what looks different from the version we accepted, and is it on purpose.
The problem
Regression suites tell us whether the code still passes. They do not show us the screens, and they do not know which differences a person would care about. So we open staging, scroll, and hope.
What changes with Kaloko
A run the team accepted becomes the scenario's baseline. Every later run can be compared with it on the canvas: pixel differences per step, criteria whose verdict or evidence changed, added or removed steps. The same comparison prints as text for the agent and the release notes.
How it works
- Accept a baseline
on a good run the reviewer presses *Accept run* and *Set as baseline*. One baseline per scenario.
- Run the flows against staging
kaloko start --scenario qa/flows/checkout.yml --env staging,kaloko walk,kaloko evaluate,kaloko share. Test data belongs to the staging prepare step; the scenario creates only what it needs. - Compare
on the canvas choose *Compare → baseline*: badges with the share of changed pixels per step, Δ on changed criteria, side-by-side and diff views in the lightbox. In the terminal
kaloko compareprints the same. - Decide
intended changes become the new baseline; unintended ones go back as returned steps with notes.
Skills and prompts
Run every flow in qa/flows against staging, compare each with its baseline and list what changed with the evidence. Do not accept anything.Expected outcome: shared runs per scenario, a text diff per scenario (statuses, verdicts, evidence), canvas links with #cmp=<baseline>.
Explain the differences between run A and its baseline in two sentences per changed step, for the release notes.Expected outcome: a short list the agent derives from kaloko compare --json; nothing invented, every line points at a step.
What you get
- Differences that a person would notice, per step and locale, with the previous verdict and evidence underneath.
- Pixel diffs computed in the browser; nothing is uploaded twice.
- A baseline that moves only when a reviewer says so.
FAQ
Are small rendering differences noise?
Fonts and anti-aliasing move a few tenths of a percent; the badge shows the share so you can ignore it. Masks for dynamic regions are on the roadmap.
Can I compare two arbitrary runs?
Yes: the *Compare* selector lists the baseline, the previous run and other runs of the scenario; #cmp=<run id> in the URL shares the view.
What it looks like
A real run of Kaloko on its own public pages, refreshed daily. This is the canvas your team gets.