Kaloko

All guides › By agent

By agent

Kaloko in CI: headless runs, read-only tokens, one comment per scenario

Run the same walkthrough from GitHub Actions or any CI: bundled Chromium, a tester token to share, a read-only token to read, results in the PR and on the canvas.

The agent's walkthrough is also a job. In CI the same four commands run against review or staging on every push, and the PR gets one comment per scenario that updates itself.

The problem

Screenshots produced in CI end up as artifacts nobody downloads. Visual checks live in a separate tool with its own login and its own idea of what a step is.

What changes with Kaloko

CI runs kaloko start, walk, evaluate, share --pr. The canvas is the artifact, the PR comment is the notification, verdicts are given by people on the canvas and read back by the agent or the next job.

How it works

  1. Config for CI

    a copy of kaloko.config.yml without browser.channel so bundled Chromium is used (npx playwright install --with-deps chromium), environments limited to what CI can reach.

  2. Secrets

    KALOKO_TOKEN with the tester role to share runs; TYPESAFE_API_KEY if you want semantic criteria evaluated in CI. A read-only token is enough for jobs that only read results or export runs.

  3. Job

    checkout, npm ci, install Chromium, then the four commands with --config kaloko.ci.config.yml; kaloko export if you want the zip as an artifact too.

  4. Gate

    kaloko feedback --json returns the run verdict; a job can wait for accepted before deploying. Kaloko's own daily demo refresh is exactly such a job.

Skills and prompts

For CI there is no prompt, only the workflow. Kaloko's own workflow is a template: demo-run.yml and scripts/demo-run.mjs.

KALOKO_TOKEN=… node scripts/demo-run.mjs

Expected outcome: a shared run, a PR comment (with --pr on a branch with a pull request), the summary line in the job log.

What you get

FAQ

How long does a run take in CI?

Roughly a minute per ten steps in two locales and two viewports, plus evaluation. Runs are read-only against production and never create data.

Can CI accept the run automatically?

No, and that is the point: acceptance is a person's stamp. CI can check that a person gave it.

What it looks like

A real run of Kaloko on its own public pages, refreshed daily. This is the canvas your team gets.

10 of 10 steps passed · 2026-09-29Open the run in Kaloko ↗

More guides

QA with Claude Code: install the skill, ask in plain words, read the canvasQA with Codex: the skill in skills/qawalk and a pointer in AGENTS.md

Install once, then work through your agent

Kaloko runs where your code and your agent are. The service stores and versions the results, shows the canvas and collects approvals.

  1. Add the CLI to the project
    npm install --save-dev kaloko

    Needs Node 20 or newer. Update later with npm update kaloko.

  2. Create the config and install the skill
    npx kaloko init --agent claude --org <your-org>

    The skill is copied to .claude/skills/kaloko. npx kaloko doctor checks Chrome, the config and the token.

  3. Create your organization and a token

    Create an organization; you become its admin. The start page offers a tester token in one click, later under Settings → API tokens. Put it into the project .env:

    KALOKO_TOKEN=qwk_…
TYPESAFE_API_KEY=…   # optional: semantic evaluator

Then just ask your agent

The skill teaches your agent the whole loop: it writes the acceptance plan and the scenario from the task, walks the screens, evaluates, shares the canvas, reads what reviewers said and fixes it. You do not type the commands; you review the canvas.

What the agent runs (or run it yourself, e.g. in CI)

The same loop by hand:

npx kaloko start --scenario docs/tasks/TASK-123/qa/scenario.yml --env local
npx kaloko walk        # playwright steps; agent/manual steps: kaloko capture
npx kaloko evaluate
npx kaloko share --pr
npx kaloko feedback    # what reviewers said, with ids to answer