Verification performance, measured honestly
Wall-clock comparisons of OSDF verification against PGP/GnuPG and OpenTDF, produced by a reproducible Hyperfine harness. We publish exactly what each bar measures, what it does not, and the commands to reproduce every number yourself.
Illustrative figures - not yet from a verified benchmark run
The numbers below are placeholders that show the expected shape of results. They will be replaced automatically once a real Hyperfine run is imported. Do not cite them as measured data.
Wall-clock verify latency
Mean time per invocation across warmed-up Hyperfine runs. Each bar is a different tool doing a different job - read the methodology before comparing them.
| Tool | Command | Mean (ms) | Std dev (ms) | Min (ms) | Max (ms) | Runs |
|---|---|---|---|---|---|---|
| OSDF | OSDF full verify | 8.1 | 0.6 | 7.2 | 9.4 | 10 |
| PGP / GnuPG | GPG detached verify | 3.2 | 0.3 | 2.8 | 3.9 | 10 |
| OpenTDF | OpenTDF structural read | 1.6 | 0.2 | 1.3 | 2.1 | 10 |
| OpenTDF | OpenTDF decrypt (online) | 124.0 | 11.0 | 108.0 | 146.0 | 10 |
Lower is better. Bars share a single linear scale; sub-millisecond bars are clamped to a minimum width so they remain visible.
How the numbers are produced
The harness uses Hyperfine for warmed-up, statistically-summarized wall-clock timing. Every comparator bar is added only when its tooling and fixtures are present, each with a clear label, so a single export describes exactly which tools ran.
Warmups
5 warmup runs discarded before timing begins, to stabilize OS caches.
Timed runs
At least 10 timed runs per command; mean, standard deviation, and min/max are reported.
Fixtures
A committed valid OSDF package, a locally-signed GPG payload, and an official OpenTDF golden .tdf vector.
| Bar | Typical operation | What is measured | What is not |
|---|---|---|---|
| OSDF full verify | osdf verify package.osdf | Container open + manifest parse + hash of every declared object + revision chain + optional transparency-log inclusion. | PDF rendering, policy decrypt. |
| GPG detached verify | gpg --batch --verify file.sig file | Ed25519 signature check over a single payload file. | Manifest object model, transparency log, revision chain. |
| OpenTDF structural read (offline) | open the golden .tdf ZIP + JSON-parse 0.manifest.json | Container open + manifest parse - the offline analogue of OSDF's structural read. | KAS round-trip, key unwrap, payload decrypt, policy gate. |
| OpenTDF decrypt (optional, online) | otdfctl decrypt --host $OTDF_PLATFORM … | Attribute + policy gate + key unwrap + payload decrypt against a live platform. | Declarative audit of every declared object (different design center). |
The crucial caveat: OpenTDF and OSDF measure different operations. The default OpenTDF bar is a structural manifest read (open a ZIP, parse one JSON file) - it is not a decrypt. A true decrypt requires a running OpenTDF platform and otdfctl, and is opt-in. OSDF, by contrast, is an integrity/audit format and does not encrypt payloads at all. Treat any single cross-tool number as a shape, not a verdict.
Read this before you quote a number
Radical transparency is the point. These caveats are not fine print - they define what the numbers mean.
These are trust-model comparisons, not byte-for-byte equivalence. Each tool is doing a different amount of work.
OSDF full verify does strictly more than a GPG detached verify or an OpenTDF structural read - it hashes every declared object and checks the revision chain.
The OpenTDF decrypt bar measures a live KAS round-trip and payload decrypt. OSDF has no equivalent because it is an integrity/audit format, not an encryption gate. Never compare that bar to an OSDF bar as if equal.
Numbers are machine-specific and dominated by process startup at these sizes. Regenerate on your own hardware before drawing conclusions.
A lower number is not automatically 'better' when the operations differ - read what each bar measures first.
What this does NOT prove
- That OSDF is 'faster than PGP' or 'faster than OpenTDF' in general - only that these specific operations, on this fixture, took this long.
- Throughput at scale, under concurrency, or on large manifests - see the Criterion and scale_bench harnesses in the repo for that.
- Anything about security strength. Speed is not a security property; see the Security pages for guarantees.
- That the structural-read and decrypt operations are interchangeable. They are not.
Run it yourself
The harness auto-detects comparators: a machine with only Hyperfine and the OSDF CLI still works - every other bar is skipped with a printed message, never a hard failure.
macOS / Linux
# from the osdf repo root
./scripts/run-benchmarks.shWindows (PowerShell)
# from the osdf repo root
.\scripts\run-benchmarks.ps1Optional: add the true OpenTDF decrypt bar
export OTDF_PLATFORM="https://localhost:8080"
# optional auth flags forwarded verbatim to otdfctl decrypt:
export OTDF_DECRYPT_ARGS="--tls-no-verify --with-client-creds-file creds.json"
./scripts/run-benchmarks.shThen refresh this page's data
The run writes docs/assets/benchmarks/hyperfine-results.json in the osdf repo. Import it into this site:
# from the osdf-web repo root
npm run benchmarks:importThat normalizes the Hyperfine export into src/lib/content/benchmarks/results.json and flips this page from illustrative to verified on the next build.
Measurement context
Placeholder context - real values are captured automatically on import.
Representative x86-64 workstation (placeholder)
Placeholder OS - replace with real run
Dual-channel DDR5 (placeholder)
Source: illustrative.ts - placeholder dataset, not a measured run
Run the checks yourself
The OSDF Verification CLI produces deterministic results for CI and audit. Pair it with the open benchmark harness to measure trust on your own hardware.