Quality evidence for your QMS
This page is for quality, validation and release people whose team works under a quality management system: ISO 9001, ISO 13485, IEC 62304, ISO 27001, or GxP in pharma and medical devices, where the rules are strictest. Kaloko does not make you compliant or certified, and nobody has certified Kaloko against any of these. It gives you evidence: a record you can check, the data your own documents need, and an honest list of what it does and does not do, so your QMS can use it and you can validate that use your way.
What Kaloko is for here
Kaloko walks the screens of a web or mobile application against criteria you wrote, records what it saw (screenshots, page state, results), and collects what people decided about it: approvals, returns, comments and the acceptance of a run. From that it makes a record signed by the service that anyone can check later.
It is not your eQMS. It does not manage controlled documents, deviations, CAPA, training or change requests, and it does not release anything by itself. AI evaluators may suggest results, but their verdicts are advisory: the decisions that count are people's.
What Kaloko records, and what you can rely on
A record holds:
- the run, scenario and environment, with the scenario's version, the SHA-256 of the scenario file as walked and the base URL
- every criterion as it was walked (its text, not today's), its result in each language and screen size, and who assessed it: a person, a deterministic check, or an AI model (named, and marked advisory unless a person decided)
- requirement references (
refs:) on the scenario, its steps and criteria - the records the run serves and why it was walked, when the run names them (see below)
- the SHA-256 of every screenshot and of the run manifest
- decisions on steps and on the run, with the files attached to a step decision (name, size, SHA-256, who and when) and earlier decisions on the step; approvals carried over from earlier runs with where they came from, and approvals by tokens and agents listed apart from people's
- electronic signatures and the reasons for changes, when you sign in Kaloko
- the versions of the CLI, Node.js, the operating system, the browsers and the service
- results that are not acceptance criteria (for example health checks), marked as advisory
The evidence package adds the trace matrix, the run's audit trail, the screenshots and the files attached to step decisions (decisions/<step>/), with a manifest of every file's SHA-256 signed by the same key.
What you can rely on:
- It can be checked without us.
kaloko signoff --verifychecks a record or a package against the public key at/.well-known/kaloko-signoff.json, offline and without an account. A changed byte fails the check. - Old records keep verifying. Retired keys stay published.
- The formats hold still.
qawalk.evidence.v1andkaloko.signoff.v1only gain fields within a version; a breaking change makes a new version and the old one stays readable. - People and machines are kept apart. An AI result nobody decided is marked advisory, and tokens and agents never count as people or sign.
Two ways to use it
Kaloko as the execution record, signed in your eQMS. This is the default and needs no setup. Kaloko records what was walked and who decided what; your own tooling pulls the evidence (kaloko evidence, the evidence package or the evidence.ready webhook), turns it into your documents and your people sign those in your eQMS. Every evidence document says so plainly: acceptance is none, accepted or returned, and signing is none, so a generator can write "executed in Kaloko, approved in our eQMS" without guessing. Your system then covers the electronic signature, its meaning, the re-authentication, the link between the signed document and the evidence (keep the record's SHA-256 in it) and how long you keep the package.
Signing in Kaloko. On Business and Enterprise an admin can turn on electronic signatures in Settings → Security, for every project or only some. A person who accepts a run (and, if you choose, approves a step) then picks what the signature means from your organization's own list, confirms their second step again, and the record shows their printed name, e-mail, the time in UTC and in their time zone, and the meaning. The signer sees the meanings in their language (Reviewed, Approved and Verified come translated; your own get a Czech label after a bar, QA release | Uvolnění QA), and the record keeps the label they saw, its id and the language. The signature also covers the decision itself: its note and the SHA-256 of the files attached to it. Changing a signed decision asks for a reason; the signature stays in the record, marked as withdrawn, with who, when and why. Tokens and agents never sign. With signatures off, nothing changes: people accept and approve as before.
Traceability
Write your requirement IDs into the scenario, as your QMS names them:
refs: [URS-12]
nodes:
- id: release-form
refs: [FS-3.4]
criteria:
- id: AC1
text: The batch number is shown before release
refs: [URS-12.1, RA-7]
Kaloko does not interpret them. The trace matrix has one row per requirement, criterion, step, language and screen size, with the screenshot and its SHA-256, the result, who assessed it and the decisions on the step and the run. It comes per run and per release, as JSON (qawalk.evidence.v1, the stable contract for generators) and CSV.
Each row also says who performs the step and in which part of the process: the step's session: (the role, such as the QC expert) and its lane (the sub-process, such as data collection). Without lanes: in the scenario, each session is a lane. So a validator can filter the matrix to one role in one sub-process, and the canvas opens on a lane with #lane=<number> (lanes count from 0) or on one role's steps with #session=<key>.
To show a step's screenshot next to the test step in your own documentation portal, the portal fetches it with a read token limited to the project: GET /api/v1/orgs/<org>/runs/<run>/steps/<step>/screen?locale=en&viewport=desktop&size=thumb&sha256=<hash>. Keep the run, the step and the screenshot's SHA-256 from the trace matrix. If the run holds a different file, the answer is 409, never a newer picture. Kaloko pages cannot be embedded in other sites, so the portal shows the image it fetched and links to the step on the canvas, where readers sign in with their own account.
One scenario, many records
A scenario is written once and walked again for each change: a dry run on local for change request CC-2026-001, a formal run on staging for the same request, later a smoke run on production for CC-2026-002. Name the record and the purpose when the run starts:
kaloko start --scenario qa/batch.yml --env staging --ref CC-2026-001 --purpose formal
In CI, KALOKO_REFS=CC-2026-001 (several separated by commas or spaces) and KALOKO_PURPOSE=formal do the same. A run takes up to 20 references of up to 80 characters: letters, digits and . _ - / : # +. The purpose is dry_run, formal or smoke; it is a label for people and portals and changes nothing about how the run is decided.
The canvas shows both in the run's header. Your portal lists the runs of a record with a read token, without parsing scenario files or run names: GET /api/v1/orgs/<org>/runs?ref=CC-2026-001 returns each run with its verdict, environment, refs and purpose; &purpose=formal keeps the formal ones. The evidence carries them as run.refs and run.purpose, and the evidence of a release takes the same ref and purpose to keep only those runs.
How it fits your framework
The clauses are where auditors usually look for this kind of record. Your QA decides what counts as evidence.
| Framework | How Kaloko helps | What stays with you |
|---|---|---|
| ISO 9001 | Signed records that cannot change unnoticed, with an audit trail (7.5); verification records of design and development: runs against criteria, people's decisions, requirement refs (8.3.4); the acceptance record says which criteria, which version and who accepted (8.6) | Document control, keeping records in your own archive, the release decision itself |
| ISO 13485 | Material for validating software used in your QMS: the supplier summary, a pinned CLI, our change control, tool versions in every record (4.1.6); legible, identifiable records protected by signatures and hashes (4.2.5); verification and validation results traced to requirements (7.3.6, 7.3.7) | Validating your use of Kaloko by risk, retention and archive, the verification and validation plan |
| IEC 62304 | Traceability from requirement to test to result in the trace matrix; software system test records: what was tested, on which version, with which tools, when and who decided; input for the release record | The development plan, safety classification, risk management, residual anomalies and the release itself |
| ISO 27001 | Acceptance criteria and their results as a record of security testing in development and acceptance (A.8.29); acceptance tied to a version, a PR and people for change management (A.8.32); a package an auditor can verify offline | The ISMS, risk assessment, Statement of Applicability and your change process |
| GxP (21 CFR Part 11, EU GMP Annex 11) | See the section below | Validation (CSV or CSA), SOPs, training, archiving, and your reading of each requirement |
| ISO 26262, IEC 61508 | A supporting tool: records of UI checks as additional evidence | Tool qualification and the safety case; Kaloko is not a safety verification tool on its own |
Controls in place today
- Access: roles per organization and project, restricted projects and guests; two-step verification can be required for everyone (Business); session lifetime and idle timeout (Business); company sign-in, SCIM and an IP allowlist (Enterprise). API tokens are scoped, expire and can be revoked.
- Audit trail: security-relevant actions, run verdicts, signatures, deleted step decisions and record downloads go to an append-only audit log with who, when, from where and through what (browser, token or agent). Step decisions and criterion verdicts are kept with their author and time. The run's audit trail in the evidence package combines both. Entries about a run's decisions are kept as long as the run; other entries for two years.
- Record integrity: Ed25519 signatures over canonical JSON, SHA-256 of every file, public keys published, retired keys stay published so older records keep verifying.
- Retention: on Business and Enterprise, accepted runs (and the runs you keep) stay while the plan lasts, and so do their records. Other runs expire after the plan's retention. A person with the right role can still delete a run (it stays in the trash for 7 days, and the deletion is in the audit log). Keep the evidence packages you need in your own system.
- Backups: the database can be restored to any minute of the last 30 days; files changed in the last 30 days are also copied nightly to a separate store.
- Time: service times are UTC from the service's clock, kept in sync by the hosting provider. Walk times come from the clock of the machine that walked.
Our change control
The CLI is released to npm with semantic versions and a changelog; you choose when to update and can pin a version. Data formats are versioned (qawalk.run.v1, qawalk.evidence.v1, kaloko.signoff.v1): a version only gains fields, and a breaking change makes a new version while the old one stays readable. The service is updated continuously; each record names the service version that signed it.
GxP: 21 CFR Part 11 and EU GMP Annex 11
GxP is the strictest profile Kaloko is built for. Kaloko is not a validated system and makes no claim of Part 11 or Annex 11 compliance; here is what it gives you for the requirements people ask about first.
| Requirement | What Kaloko gives you | What your validation has to settle |
|---|---|---|
| Part 11 §11.10 (controls for closed systems) | Access by role, an audit trail with who, when and through what, authority checks on accepting and signing, accurate copies of records (JSON, PDF, the package) | Whether these controls are enough for your intended use; SOPs and training |
| Part 11 §11.50 (signature manifestations) | Printed name, time with time zone and the meaning on every signature in the record | Your list of meanings |
| Part 11 §11.70 (signature and record linking) | The signature is bound to the run and to the decision (its note and attached files by SHA-256); the signed record cannot change unnoticed | Keeping the record's SHA-256 in documents you sign elsewhere |
| Part 11 §11.200 (signature components) | The person's session plus a fresh second step of their own (a passkey or an authenticator app) at signing | Whether that pair meets your reading of §11.200; there are no passwords |
| Annex 11 §4 (validation) | The supplier summary, a pinned CLI, versioned formats and our change control | Validating your use |
| Annex 11 §9 (audit trails) | The audit log and the run's audit trail in the package | Reviewing the audit trail as your SOPs require |
| Annex 11 §12 (security) | Roles, required two-step verification, session limits, company sign-in, IP allowlist | User management and access reviews |
| Annex 11 §14 (electronic signature) | Optional signatures with meaning and re-authentication, withdrawn only with a reason | Deciding where you sign: in Kaloko or in your eQMS |
How it maps to a validation approach:
| Your activity | What Kaloko gives you |
|---|---|
| Supplier assessment | The supplier summary, the changelog and the trust page |
| Installation (IQ) | A pinned CLI version from npm (kaloko@x.y.z); every record names the CLI, Node.js, browser and service versions it ran with |
| Operation (OQ) | Your own scenario against a page you control, with known passes and known failures: deterministic checks must catch the failures, records and packages must verify, and a changed file must fail the check |
| Performance or UAT (PQ) | Your real scenarios with refs:, walked on the validation environment; the record or the package is the executed evidence, people's decisions are the review |
| Risk-based assurance (CSA) | Deterministic checks for high-risk criteria, AI criteria only as advisory support with a person deciding, the trace matrix for coverage |
Known gaps
- No certification or validation of Kaloko by anyone, and no claim of compliance with any of the frameworks above.
- No SOC 2 report yet; the preparation is described on the trust page.
- Signing in Kaloko uses the person's session (e-mail link or company sign-in) plus a fresh second step of their own (a passkey or an authenticator app). There are no passwords. The signature is made by the service on the person's behalf, not with the person's own certificate, and it is not a qualified electronic signature under eIDAS.
- You cannot hold the service at a version; only the CLI can be pinned.
- The time source is not validated, and records carry no timestamp from an outside authority yet.
- Retention follows the plan: there is no legal hold, and the PDF is not PDF/A. Archive the package yourself.
- The audit log keeps entries other than decisions for two years. It only gains entries through the service, but nothing outside the service seals it; the signed package is what you can check independently.
- Editing a comment's text keeps no earlier version, and changing a decision that is not signed asks for no reason.
- Masking personal data changes what the screenshot shows. Decide whether runs you keep as evidence mask or not.
- There is no ready-made supplier audit package yet; we answer questionnaires and describe our change control, backups and restore on request.
- AI evaluators run at outside providers and their models change; their verdicts stay advisory.
Questions to settle with your QA
- Which framework your evidence has to serve, and which clauses it supports.
- Which path you take: sign in Kaloko, or sign the generated documents in your eQMS.
- Which meanings your signatures use, and whether step approvals are signed too.
- Where the evidence packages are archived and for how long.
- Which criteria may be assessed by AI at all, and who decides them.
- How you treat service updates you cannot postpone.