Run Kaloko in CI and comment on pull requests

In CI, Kaloko runs the same commands as on your laptop. The usual setup walks what a pull request changed on its preview deployment (Vercel, Netlify, Cloudflare Pages or any review app) and puts the canvas link into a comment on the pull request.

1. Write the workflow #

npx kaloko ci github     # writes .github/workflows/kaloko.yml
npx kaloko ci gitlab     # writes kaloko.gitlab-ci.yml; include it from .gitlab-ci.yml
npx kaloko ci github --print   # show it, write nothing

--scenario a.yml,b.yml limits it to some scenarios. The command also adds a preview environment to kaloko.config.yml with base_url: ${KALOKO_PREVIEW_URL}. A scenario that lists environments: has to include preview, or kaloko start refuses it; the command tells you which ones.

2. Add the token #

Create a token with the tester role in Settings → API tokens and add it to the repository's CI secrets as KALOKO_TOKEN. On GitLab, add KALOKO_GITLAB_TOKEN too: a project access token with the api scope, so Kaloko can write the note on the merge request. Commit the workflow.

What the workflow does #

On every successful preview deployment of a pull request:

1. It installs the project and the browser (npx kaloko browser install --with-deps), plus WebKit and Firefox when a scenario asks for them.
2. npx kaloko ci preview-url --wait 600 --github-env finds the preview address, the pull request and its base branch and passes them on as KALOKO_PREVIEW_URL, KALOKO_PR and KALOKO_BASE.
3. For each scenario, kaloko affected --check skips it when the change does not reach any of its steps. Otherwise it runs:

npx kaloko start --scenario qa/flows/checkout.yml --env preview
npx kaloko walk --ephemeral --affected
npx kaloko share --pr

share --pr adds one comment per scenario to the pull request and updates it on later pushes. Uploads go by content, so screenshots that did not change are not sent again.

Good to know #

• walk --affected needs the git history: check out with fetch-depth: 0 on GitHub or GIT_DEPTH: "0" on GitLab.
• --trigger pull_request runs on every push instead of the deployment event and waits for the preview.
• kaloko ci preview-url prints only the address, so it works in any CI: export KALOKO_PREVIEW_URL="$(npx kaloko ci preview-url)".
• A preview behind Vercel's deployment protection needs a VERCEL_AUTOMATION_BYPASS_SECRET secret and the access.headers block of the preview environment, which the generated config has commented out.
• No comment on the pull request? Usually there was no preview deployment, or the token is missing. The job log says which.

When release-please, changesets or semantic-release publishes your releases, npx kaloko ci github --release adds a second workflow that records each release on Kaloko. All options are in the CLI reference.