Kalokoby

All guides › By agent

By agent

Kaloko in CI: every pull request preview walked and commented

One command writes the workflow: it waits for the Vercel, Netlify or Cloudflare preview, walks the steps the change affects and comments on the PR.

Every pull request already gets a preview deployment, and the review comment says "looks good" before anyone opened it on a phone. Kaloko walks the preview for you, only the screens the change reaches, and puts the canvas link into the pull request.

The problem

Screenshots produced in CI end up as artifacts nobody downloads. Walking every flow on every push takes long enough that teams stop doing it, and the preview link in the PR is clicked once, on a laptop, in one language.

What changes with Kaloko

npx kaloko ci github writes the workflow. When the preview deployment succeeds, CI walks the steps the pull request affects on it and comments once per scenario. People give verdicts on the canvas; the agent or the next job reads them back with kaloko feedback.

How it works

  1. Generate the workflow

    npx kaloko ci github (or npx kaloko ci gitlab) writes .github/workflows/kaloko.yml and adds a preview environment to kaloko.config.yml whose base_url is ${KALOKO_PREVIEW_URL}.

  2. Add one secret

    KALOKO_TOKEN, a token with the tester role and the upload scope (Settings → API tokens; on Enterprise a service account's token, which no person owns). TYPESAFE_API_KEY only if you want your own evaluator key.

  3. Find the preview

    Vercel, Netlify and Cloudflare Pages report previews as GitHub deployments. kaloko ci preview-url reads the URL, the pull request and its base branch; any other review app sets KALOKO_PREVIEW_URL.

  4. Walk what changed

    kaloko affected --fail-if-none skips scenarios the change does not reach; kaloko walk --affected walks the steps whose routes, imported components or affects: files changed, plus the steps before them.

  5. Comment

    kaloko share --pr uploads the run (unchanged screens are not sent again) and keeps one comment per scenario on the pull request up to date.

  6. Block the merge if you want

    with the Kaloko GitHub App (Team and up), the pull request also gets one Kaloko check on its newest commit. It turns green when the runs shared for that commit pass or when the change reaches no scenario, so branch protection can require it. Before walking, the workflow sends kaloko ci plan, which tells the check which scenarios to wait for.

Skills and prompts

Run it in the repository, review the file, add the secret and commit:

npx kaloko ci github --scenario qa/flows/checkout.yml

This is the workflow it writes:

# .github/workflows/kaloko.yml
name: Kaloko
on:
  # Vercel, Netlify and Cloudflare Pages report each preview as a GitHub deployment
  deployment_status:
  workflow_dispatch:
concurrency:
  group: kaloko-${{ github.event.deployment.sha || github.event.pull_request.head.sha || github.sha }}
  cancel-in-progress: true
permissions:
  contents: read
  deployments: read
  pull-requests: write
jobs:
  walk:
    if: github.event_name != 'deployment_status' || (github.event.deployment_status.state == 'success' && !startsWith(github.event.deployment_status.environment, 'Production') && !startsWith(github.event.deployment_status.environment, 'production'))
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      KALOKO_TOKEN: ${{ secrets.KALOKO_TOKEN }}
      TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
      GH_TOKEN: ${{ github.token }}
      KALOKO_AGENT: github-actions
      KALOKO_BROWSER_CHANNEL: chromium
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.deployment.sha || github.event.pull_request.head.sha || github.sha }}
          fetch-depth: 0   # walk --affected compares with the pull request's base
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx kaloko browser install --with-deps   # the Chromium of kaloko's own Playwright
      - name: Preview deployment
        # the deployment's URL, the pull request and its base; waits for the preview when it is still building
        run: npx kaloko ci preview-url --wait 600 --github-env
      - name: Walk, share, comment
        run: |
          status=0
          npx kaloko ci plan --scenario qa/flows/checkout.yml || echo "kaloko ci plan did not reach Kaloko; the GitHub check guesses from the changed files."
          for s in qa/flows/checkout.yml; do
            if ! npx kaloko affected --scenario "$s" --check; then echo "Nothing in $s is affected; skipped."; continue; fi
            npx kaloko start --scenario "$s" --env preview
            npx kaloko walk --ephemeral --affected || status=1
            npx kaloko share --pr || status=1
          done
          exit $status

On GitLab, npx kaloko ci gitlab writes kaloko.gitlab-ci.yml for merge request pipelines; include it from .gitlab-ci.yml and add KALOKO_TOKEN and KALOKO_GITLAB_TOKEN (a project access token with the api scope, for the merge request note).

Or ask your agent:

Set up Kaloko in CI for this repository: walk the checkout and sign-up flows on every pull request preview and comment on the PR. Show me the workflow before you commit it.

Expected outcome: on the next pull request, a comment per scenario with the summary line and the canvas link, updated on every push.

What you get

FAQ

What if my previews are not on Vercel, Netlify or Cloudflare?

Set KALOKO_PREVIEW_URL in the job (for example https://pr-${{ github.event.number }}.review.example.com) and generate the workflow with --trigger pull_request. Any URL CI can reach works.

How long does a run take in CI?

Roughly a minute per ten steps in two locales and two viewports, plus evaluation. With --affected a typical pull request walks a few steps. Locales walk side by side in CI.

Our coding agents run in the cloud. Do we need this workflow?

Not for the check. An agent in Claude Code on the web, Codex or Cursor runs the same kaloko walk --affected and kaloko share in its sandbox, and the Kaloko check on the pull request picks up the run by its commit. Commit before walking: a run of uncommitted changes does not count. The workflow is for when the sandbox cannot run a browser or reach the preview.

Can we skip the check on one pull request?

Yes, if you have the access your admin chose (write by default): the label kaloko:skip, or a comment /kaloko skip <reason> for the current commit. Apps and bots cannot skip, and the check shows who skipped it and why.

Can CI accept the run automatically?

No. A person accepts the run; CI can only check that someone did (kaloko feedback --json).

What it looks like

A real run of Kaloko on its own public pages, refreshed daily. This is the canvas your team gets.

11 of 12 steps passed · 2026-11-16Open the run in Kaloko ↗

More guides

QA with Claude Code: install the skill, then ask in plain wordsQA with Codex: the skill in skills/kaloko and a pointer in AGENTS.md

Install once, then work through your agent

Kaloko runs where your code and your agent are. The service stores and versions the results, shows the canvas and collects approvals.

  1. Add the CLI to the project
    npm install --save-dev kaloko

    Needs Node 20 or newer. Update later with npm update kaloko.

  2. Create the config and install the skill
    npx kaloko init --agent claude --org <your-org>

    The skill is copied to .claude/skills/kaloko. npx kaloko doctor checks Chrome, the config and the token.

  3. Create your organization and a token

    Create an organization; you become its admin. The start page offers a tester token in one click, later under Settings → API tokens. Put it into the project .env:

    KALOKO_TOKEN=kal_…
    TYPESAFE_API_KEY=…   # optional: semantic evaluator

Then just ask your agent

The skill teaches your agent the whole loop: it writes the acceptance plan and the scenario from the task, walks the screens, evaluates, shares the canvas, reads what reviewers said and fixes it. You don’t type the commands; you look at the canvas.

What the agent runs (or run it yourself, e.g. in CI)

The same loop by hand:

npx kaloko start --scenario docs/tasks/TASK-123/qa/scenario.yml --env local
npx kaloko walk        # playwright steps; agent/manual steps: kaloko capture
npx kaloko evaluate
npx kaloko share --pr
npx kaloko feedback    # what reviewers said, with ids to answer