Docs

Kaloko

Download
Markdown with pictures (.zip)Docusaurus folder (.zip)VitePress folder (.zip)MkDocs folder (.zip)Offline HTML (.zip)PDFChangelog as JSON
Pages

Quality evidence for your QMS

This page is for quality, validation and release people whose team works under a quality management system: ISO 9001, ISO 13485, IEC 62304, ISO 27001, or GxP in pharma and medical devices, where the rules are strictest. Kaloko does not make you compliant or certified, and nobody has certified Kaloko against any of these. It gives you evidence: a record you can check, the data your own documents need, and an honest list of what it does and does not do, so your QMS can use it and you can validate that use your way.

What Kaloko is for here

Kaloko walks the screens of a web or mobile application against criteria you wrote, records what it saw (screenshots, page state, results), and collects what people decided about it: approvals, returns, comments and the acceptance of a run. From that it makes a record signed by the service that anyone can check later.

It is not your eQMS. It does not manage controlled documents, deviations, CAPA, training or change requests, and it does not release anything by itself. AI evaluators may suggest results, but their verdicts are advisory: the decisions that count are people's.

What Kaloko records, and what you can rely on

A record holds:

  • the run, scenario and environment, with the scenario's version, the SHA-256 of the scenario file as walked and the base URL
  • every criterion as it was walked (its text, not today's), its result in each language and screen size, and who assessed it: a person, a deterministic check, or an AI model (named, and marked advisory unless a person decided)
  • requirement references (refs:) on the scenario, its steps and criteria
  • the records the run serves and why it was walked, when the run names them (see below)
  • the SHA-256 of every screenshot and of the run manifest
  • decisions on steps and on the run, with the files attached to a step decision (name, size, SHA-256, who and when) and earlier decisions on the step; approvals carried over from earlier runs with where they came from, and approvals by tokens and agents listed apart from people's
  • electronic signatures and the reasons for changes, when you sign in Kaloko
  • the versions of the CLI, Node.js, the operating system, the browsers and the service
  • results that are not acceptance criteria (for example health checks), marked as advisory

The evidence package adds the trace matrix, the run's audit trail, the screenshots and the files attached to step decisions (decisions/<step>/), with a manifest of every file's SHA-256 signed by the same key.

What you can rely on:

  • It can be checked without us. kaloko signoff --verify checks a record or a package against the public key at /.well-known/kaloko-signoff.json, offline and without an account. A changed byte fails the check.
  • Old records keep verifying. Retired keys stay published.
  • The formats hold still. qawalk.evidence.v1 and kaloko.signoff.v1 only gain fields within a version; a breaking change makes a new version and the old one stays readable.
  • People and machines are kept apart. An AI result nobody decided is marked advisory, and tokens and agents never count as people or sign.

Two ways to use it

Kaloko as the execution record, signed in your eQMS. This is the default and needs no setup. Kaloko records what was walked and who decided what; your own tooling pulls the evidence (kaloko evidence, the evidence package or the evidence.ready webhook), turns it into your documents and your people sign those in your eQMS. Every evidence document says so plainly: acceptance is none, accepted or returned, and signing is none, so a generator can write "executed in Kaloko, approved in our eQMS" without guessing. Your system then covers the electronic signature, its meaning, the re-authentication, the link between the signed document and the evidence (keep the record's SHA-256 in it) and how long you keep the package.

Signing in Kaloko. On Business and Enterprise an admin can turn on electronic signatures in Settings → Security, for every project or only some. A person who accepts a run (and, if you choose, approves a step) then picks what the signature means from your organization's own list, confirms their second step again, and the record shows their printed name, e-mail, the time in UTC and in their time zone, and the meaning. The signer sees the meanings in their language (Reviewed, Approved and Verified come translated; your own get a Czech label after a bar, QA release | Uvolnění QA), and the record keeps the label they saw, its id and the language. The signature also covers the decision itself: its note and the SHA-256 of the files attached to it. Changing a signed decision asks for a reason; the signature stays in the record, marked as withdrawn, with who, when and why. Tokens and agents never sign. With signatures off, nothing changes: people accept and approve as before.

Traceability

Write your requirement IDs into the scenario, as your QMS names them:

refs: [URS-12]
nodes:
  - id: release-form
    refs: [FS-3.4]
    criteria:
      - id: AC1
        text: The batch number is shown before release
        refs: [URS-12.1, RA-7]

Kaloko does not interpret them. The trace matrix has one row per requirement, criterion, step, language and screen size, with the screenshot and its SHA-256, the result, who assessed it and the decisions on the step and the run. It comes per run and per release, as JSON (qawalk.evidence.v1, the stable contract for generators) and CSV.

Each row also says who performs the step and in which part of the process: the step's session: (the role, such as the QC expert) and its lane (the sub-process, such as data collection). Without lanes: in the scenario, each session is a lane. So a validator can filter the matrix to one role in one sub-process, and the canvas opens on a lane with #lane=<number> (lanes count from 0) or on one role's steps with #session=<key>.

To show a step's screenshot next to the test step in your own documentation portal, the portal fetches it with a read token limited to the project: GET /api/v1/orgs/<org>/runs/<run>/steps/<step>/screen?locale=en&viewport=desktop&size=thumb&sha256=<hash>. Keep the run, the step and the screenshot's SHA-256 from the trace matrix. If the run holds a different file, the answer is 409, never a newer picture. Kaloko pages cannot be embedded in other sites, so the portal shows the image it fetched and links to the step on the canvas, where readers sign in with their own account.

One scenario, many records

A scenario is written once and walked again for each change: a dry run on local for change request CC-2026-001, a formal run on staging for the same request, later a smoke run on production for CC-2026-002. Name the record and the purpose when the run starts:

kaloko start --scenario qa/batch.yml --env staging --ref CC-2026-001 --purpose formal

In CI, KALOKO_REFS=CC-2026-001 (several separated by commas or spaces) and KALOKO_PURPOSE=formal do the same. A run takes up to 20 references of up to 80 characters: letters, digits and . _ - / : # +. The purpose is dry_run, formal or smoke; it is a label for people and portals and changes nothing about how the run is decided.

The canvas shows both in the run's header. Your portal lists the runs of a record with a read token, without parsing scenario files or run names: GET /api/v1/orgs/<org>/runs?ref=CC-2026-001 returns each run with its verdict, environment, refs and purpose; &purpose=formal keeps the formal ones. The evidence carries them as run.refs and run.purpose, and the evidence of a release takes the same ref and purpose to keep only those runs.

How it fits your framework

The clauses are where auditors usually look for this kind of record. Your QA decides what counts as evidence.

FrameworkHow Kaloko helpsWhat stays with you
ISO 9001Signed records that cannot change unnoticed, with an audit trail (7.5); verification records of design and development: runs against criteria, people's decisions, requirement refs (8.3.4); the acceptance record says which criteria, which version and who accepted (8.6)Document control, keeping records in your own archive, the release decision itself
ISO 13485Material for validating software used in your QMS: the supplier summary, a pinned CLI, our change control, tool versions in every record (4.1.6); legible, identifiable records protected by signatures and hashes (4.2.5); verification and validation results traced to requirements (7.3.6, 7.3.7)Validating your use of Kaloko by risk, retention and archive, the verification and validation plan
IEC 62304Traceability from requirement to test to result in the trace matrix; software system test records: what was tested, on which version, with which tools, when and who decided; input for the release recordThe development plan, safety classification, risk management, residual anomalies and the release itself
ISO 27001Acceptance criteria and their results as a record of security testing in development and acceptance (A.8.29); acceptance tied to a version, a PR and people for change management (A.8.32); a package an auditor can verify offlineThe ISMS, risk assessment, Statement of Applicability and your change process
GxP (21 CFR Part 11, EU GMP Annex 11)See the section belowValidation (CSV or CSA), SOPs, training, archiving, and your reading of each requirement
ISO 26262, IEC 61508A supporting tool: records of UI checks as additional evidenceTool qualification and the safety case; Kaloko is not a safety verification tool on its own

Controls in place today

  • Access: roles per organization and project, restricted projects and guests; two-step verification can be required for everyone (Business); session lifetime and idle timeout (Business); company sign-in, SCIM and an IP allowlist (Enterprise). API tokens are scoped, expire and can be revoked.
  • Audit trail: security-relevant actions, run verdicts, signatures, deleted step decisions and record downloads go to an append-only audit log with who, when, from where and through what (browser, token or agent). Step decisions and criterion verdicts are kept with their author and time. The run's audit trail in the evidence package combines both. Entries about a run's decisions are kept as long as the run; other entries for two years.
  • Record integrity: Ed25519 signatures over canonical JSON, SHA-256 of every file, public keys published, retired keys stay published so older records keep verifying.
  • Retention: on Business and Enterprise, accepted runs (and the runs you keep) stay while the plan lasts, and so do their records. Other runs expire after the plan's retention. A person with the right role can still delete a run (it stays in the trash for 7 days, and the deletion is in the audit log). Keep the evidence packages you need in your own system.
  • Backups: the database can be restored to any minute of the last 30 days; files changed in the last 30 days are also copied nightly to a separate store.
  • Time: service times are UTC from the service's clock, kept in sync by the hosting provider. Walk times come from the clock of the machine that walked.

Our change control

The CLI is released to npm with semantic versions and a changelog; you choose when to update and can pin a version. Data formats are versioned (qawalk.run.v1, qawalk.evidence.v1, kaloko.signoff.v1): a version only gains fields, and a breaking change makes a new version while the old one stays readable. The service is updated continuously; each record names the service version that signed it.

GxP: 21 CFR Part 11 and EU GMP Annex 11

GxP is the strictest profile Kaloko is built for. Kaloko is not a validated system and makes no claim of Part 11 or Annex 11 compliance; here is what it gives you for the requirements people ask about first.

RequirementWhat Kaloko gives youWhat your validation has to settle
Part 11 §11.10 (controls for closed systems)Access by role, an audit trail with who, when and through what, authority checks on accepting and signing, accurate copies of records (JSON, PDF, the package)Whether these controls are enough for your intended use; SOPs and training
Part 11 §11.50 (signature manifestations)Printed name, time with time zone and the meaning on every signature in the recordYour list of meanings
Part 11 §11.70 (signature and record linking)The signature is bound to the run and to the decision (its note and attached files by SHA-256); the signed record cannot change unnoticedKeeping the record's SHA-256 in documents you sign elsewhere
Part 11 §11.200 (signature components)The person's session plus a fresh second step of their own (a passkey or an authenticator app) at signingWhether that pair meets your reading of §11.200; there are no passwords
Annex 11 §4 (validation)The supplier summary, a pinned CLI, versioned formats and our change controlValidating your use
Annex 11 §9 (audit trails)The audit log and the run's audit trail in the packageReviewing the audit trail as your SOPs require
Annex 11 §12 (security)Roles, required two-step verification, session limits, company sign-in, IP allowlistUser management and access reviews
Annex 11 §14 (electronic signature)Optional signatures with meaning and re-authentication, withdrawn only with a reasonDeciding where you sign: in Kaloko or in your eQMS

How it maps to a validation approach:

Your activityWhat Kaloko gives you
Supplier assessmentThe supplier summary, the changelog and the trust page
Installation (IQ)A pinned CLI version from npm (kaloko@x.y.z); every record names the CLI, Node.js, browser and service versions it ran with
Operation (OQ)Your own scenario against a page you control, with known passes and known failures: deterministic checks must catch the failures, records and packages must verify, and a changed file must fail the check
Performance or UAT (PQ)Your real scenarios with refs:, walked on the validation environment; the record or the package is the executed evidence, people's decisions are the review
Risk-based assurance (CSA)Deterministic checks for high-risk criteria, AI criteria only as advisory support with a person deciding, the trace matrix for coverage

Known gaps

  • No certification or validation of Kaloko by anyone, and no claim of compliance with any of the frameworks above.
  • No SOC 2 report yet; the preparation is described on the trust page.
  • Signing in Kaloko uses the person's session (e-mail link or company sign-in) plus a fresh second step of their own (a passkey or an authenticator app). There are no passwords. The signature is made by the service on the person's behalf, not with the person's own certificate, and it is not a qualified electronic signature under eIDAS.
  • You cannot hold the service at a version; only the CLI can be pinned.
  • The time source is not validated, and records carry no timestamp from an outside authority yet.
  • Retention follows the plan: there is no legal hold, and the PDF is not PDF/A. Archive the package yourself.
  • The audit log keeps entries other than decisions for two years. It only gains entries through the service, but nothing outside the service seals it; the signed package is what you can check independently.
  • Editing a comment's text keeps no earlier version, and changing a decision that is not signed asks for no reason.
  • Masking personal data changes what the screenshot shows. Decide whether runs you keep as evidence mask or not.
  • There is no ready-made supplier audit package yet; we answer questionnaires and describe our change control, backups and restore on request.
  • AI evaluators run at outside providers and their models change; their verdicts stay advisory.

Questions to settle with your QA

  • Which framework your evidence has to serve, and which clauses it supports.
  • Which path you take: sign in Kaloko, or sign the generated documents in your eQMS.
  • Which meanings your signatures use, and whether step approvals are signed too.
  • Where the evidence packages are archived and for how long.
  • Which criteria may be assessed by AI at all, and who decides them.
  • How you treat service updates you cannot postpone.
On this pageOn this page