All guides › By agent
By agentKaloko in CI: every pull request preview walked and commented
One command writes the workflow: it waits for the Vercel, Netlify or Cloudflare preview, walks the steps the change affects and comments on the PR.
Every pull request already gets a preview deployment, and the review comment says "looks good" before anyone opened it on a phone. Kaloko walks the preview for you, only the screens the change reaches, and puts the canvas link into the pull request.
The problem
Screenshots produced in CI end up as artifacts nobody downloads. Walking every flow on every push takes long enough that teams stop doing it, and the preview link in the PR is clicked once, on a laptop, in one language.
What changes with Kaloko
npx kaloko ci github writes the workflow. When the preview deployment succeeds, CI walks the steps the pull request affects on it and comments once per scenario. People give verdicts on the canvas; the agent or the next job reads them back with kaloko feedback.
How it works
- Generate the workflow
npx kaloko ci github(ornpx kaloko ci gitlab) writes.github/workflows/kaloko.ymland adds apreviewenvironment tokaloko.config.ymlwhosebase_urlis${KALOKO_PREVIEW_URL}. - Add one secret
KALOKO_TOKEN, a token with the tester role and the upload scope (Settings → API tokens; on Enterprise a service account's token, which no person owns).TYPESAFE_API_KEYonly if you want your own evaluator key. - Find the preview
Vercel, Netlify and Cloudflare Pages report previews as GitHub deployments.
kaloko ci preview-urlreads the URL, the pull request and its base branch; any other review app setsKALOKO_PREVIEW_URL. - Walk what changed
kaloko affected --fail-if-noneskips scenarios the change does not reach;kaloko walk --affectedwalks the steps whose routes, imported components oraffects:files changed, plus the steps before them. - Comment
kaloko share --pruploads the run (unchanged screens are not sent again) and keeps one comment per scenario on the pull request up to date. - Block the merge if you want
with the Kaloko GitHub App (Team and up), the pull request also gets one
Kalokocheck on its newest commit. It turns green when the runs shared for that commit pass or when the change reaches no scenario, so branch protection can require it. Before walking, the workflow sendskaloko ci plan, which tells the check which scenarios to wait for.
Skills and prompts
Run it in the repository, review the file, add the secret and commit:
npx kaloko ci github --scenario qa/flows/checkout.ymlThis is the workflow it writes:
# .github/workflows/kaloko.yml
name: Kaloko
on:
# Vercel, Netlify and Cloudflare Pages report each preview as a GitHub deployment
deployment_status:
workflow_dispatch:
concurrency:
group: kaloko-${{ github.event.deployment.sha || github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: true
permissions:
contents: read
deployments: read
pull-requests: write
jobs:
walk:
if: github.event_name != 'deployment_status' || (github.event.deployment_status.state == 'success' && !startsWith(github.event.deployment_status.environment, 'Production') && !startsWith(github.event.deployment_status.environment, 'production'))
runs-on: ubuntu-latest
timeout-minutes: 30
env:
KALOKO_TOKEN: ${{ secrets.KALOKO_TOKEN }}
TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
GH_TOKEN: ${{ github.token }}
KALOKO_AGENT: github-actions
KALOKO_BROWSER_CHANNEL: chromium
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.deployment.sha || github.event.pull_request.head.sha || github.sha }}
fetch-depth: 0 # walk --affected compares with the pull request's base
- uses: actions/setup-node@v4
with:
node-version: 22
- run: npm ci
- run: npx kaloko browser install --with-deps # the Chromium of kaloko's own Playwright
- name: Preview deployment
# the deployment's URL, the pull request and its base; waits for the preview when it is still building
run: npx kaloko ci preview-url --wait 600 --github-env
- name: Walk, share, comment
run: |
status=0
npx kaloko ci plan --scenario qa/flows/checkout.yml || echo "kaloko ci plan did not reach Kaloko; the GitHub check guesses from the changed files."
for s in qa/flows/checkout.yml; do
if ! npx kaloko affected --scenario "$s" --check; then echo "Nothing in $s is affected; skipped."; continue; fi
npx kaloko start --scenario "$s" --env preview
npx kaloko walk --ephemeral --affected || status=1
npx kaloko share --pr || status=1
done
exit $statusOn GitLab, npx kaloko ci gitlab writes kaloko.gitlab-ci.yml for merge request pipelines; include it from .gitlab-ci.yml and add KALOKO_TOKEN and KALOKO_GITLAB_TOKEN (a project access token with the api scope, for the merge request note).
Or ask your agent:
Set up Kaloko in CI for this repository: walk the checkout and sign-up flows on every pull request preview and comment on the PR. Show me the workflow before you commit it.Expected outcome: on the next pull request, a comment per scenario with the summary line and the canvas link, updated on every push.
What you get
- Every preview walked in every locale and viewport of the scenario, not just clicked once.
- Only the steps a pull request reaches, with the reason for each in the job log.
- One comment per scenario, updated in place; the canvas for decisions.
- A
Kalokocheck with a stable name that branch protection can require.
FAQ
What if my previews are not on Vercel, Netlify or Cloudflare?
Set KALOKO_PREVIEW_URL in the job (for example https://pr-${{ github.event.number }}.review.example.com) and generate the workflow with --trigger pull_request. Any URL CI can reach works.
How long does a run take in CI?
Roughly a minute per ten steps in two locales and two viewports, plus evaluation. With --affected a typical pull request walks a few steps. Locales walk side by side in CI.
Our coding agents run in the cloud. Do we need this workflow?
Not for the check. An agent in Claude Code on the web, Codex or Cursor runs the same kaloko walk --affected and kaloko share in its sandbox, and the Kaloko check on the pull request picks up the run by its commit. Commit before walking: a run of uncommitted changes does not count. The workflow is for when the sandbox cannot run a browser or reach the preview.
Can we skip the check on one pull request?
Yes, if you have the access your admin chose (write by default): the label kaloko:skip, or a comment /kaloko skip <reason> for the current commit. Apps and bots cannot skip, and the check shows who skipped it and why.
Can CI accept the run automatically?
No. A person accepts the run; CI can only check that someone did (kaloko feedback --json).
What it looks like
A real run of Kaloko on its own public pages, refreshed daily. This is the canvas your team gets.