Docs

Kaloko

Download
Markdown with pictures (.zip)Docusaurus folder (.zip)VitePress folder (.zip)MkDocs folder (.zip)Offline HTML (.zip)PDFChangelog as JSON
Pages

Run Kaloko in CI and comment on pull requests

In CI, Kaloko runs the same commands as on your laptop. The usual setup walks what a pull request changed on its preview deployment (Vercel, Netlify, Cloudflare Pages or any review app) and puts the canvas link into a comment on the pull request.

1. Write the workflow

npx kaloko ci github     # writes .github/workflows/kaloko.yml
npx kaloko ci gitlab     # writes kaloko.gitlab-ci.yml; include it from .gitlab-ci.yml
npx kaloko ci github --print   # show it, write nothing

--scenario a.yml,b.yml limits it to some scenarios. The command also adds a preview environment to kaloko.config.yml with base_url: ${KALOKO_PREVIEW_URL}. A scenario that lists environments: has to include preview, or kaloko start refuses it; the command tells you which ones.

2. Add the token

Create a token with the tester role in Settings → API tokens and add it to the repository's CI secrets as KALOKO_TOKEN. On GitLab, add KALOKO_GITLAB_TOKEN too: a project access token with the api scope, so Kaloko can write the note on the merge request. Commit the workflow.

What the workflow does

On every successful preview deployment of a pull request:

  1. It installs the project and the browser (npx kaloko browser install --with-deps), plus WebKit and Firefox when a scenario asks for them.
  2. npx kaloko ci preview-url --wait 600 --github-env finds the preview address, the pull request and its base branch and passes them on as KALOKO_PREVIEW_URL, KALOKO_PR and KALOKO_BASE.
  3. For each scenario, kaloko affected --fail-if-none skips it when the change does not reach any of its steps. Otherwise it runs:
npx kaloko start --scenario qa/flows/checkout.yml --env preview
npx kaloko walk --ephemeral --affected
npx kaloko share --pr

share --pr adds one comment per scenario to the pull request and updates it on later pushes. Uploads go by content, so screenshots that did not change are not sent again.

The Kaloko check on the pull request

With the Kaloko GitHub App (an admin installs it in Settings → Integrations → GitHub, Team and up), every pull request in a connected repository gets one check named Kaloko on its newest commit. It turns green when the runs shared for that commit pass, or when the change reaches no scenario. It waits while a scenario the change reaches has no run yet, and turns red when one fails. The runs can come from this workflow, from a coding agent in the cloud or from a laptop: what counts is the commit. A run of uncommitted changes does not count.

To block merging until it is green, require the status check Kaloko in the repository's branch protection. Kaloko never changes branch protection itself.

  • Modes (Settings, per repository): off, manual (only after the label kaloko:run or /kaloko walk), automatic (the default) and strict (every scenario, a review required, a time limit).
  • One pull request: the labels kaloko:skip, kaloko:all, kaloko:review, the comment /kaloko skip <reason> (this commit only) or the buttons on the check. Only people with the access your admin chose may skip, never apps or bots, and the check names who skipped it and why.
  • In the repository: the github: section of kaloko.config.yml on the base branch can make the check stricter, never looser.
  • npx kaloko ci plan tells the check exactly which scenarios a commit reaches. The generated workflow sends it before walking.
  • No check appears? Settings → Integrations → GitHub lists the repositories and says why, for example a repository that is not in the app's installation yet.

Good to know

  • walk --affected needs the git history: check out with fetch-depth: 0 on GitHub or GIT_DEPTH: "0" on GitLab.
  • --trigger pull_request runs on every push instead of the deployment event and waits for the preview.
  • kaloko ci preview-url prints only the address, so it works in any CI: export KALOKO_PREVIEW_URL="$(npx kaloko ci preview-url)".
  • A preview behind Vercel's deployment protection needs a VERCEL_AUTOMATION_BYPASS_SECRET secret and the access.headers block of the preview environment, which the generated config has commented out.
  • Runs shared from CI also compare their health (the advisory checks of packs) with the previous run of the scenario and alert when there is a new error. npx kaloko health --fail-on error stops the job on health errors, if the team wants that.
  • No comment on the pull request? Usually there was no preview deployment, or the token is missing. The job log says which.

When release-please, changesets or semantic-release publishes your releases, npx kaloko ci github --release adds a second workflow that records each release on Kaloko. All options are in the CLI reference.

On this pageOn this page