Verification

Verification is selective. Only installed and enabled managed artifacts from the installer state are checked. If no managed artifacts match the requested scope, verify returns no-managed-artifacts instead of ok.

verify checks installer ownership and file integrity. It does not prove that an agent runtime has loaded a skill. Use smoke for the separate agent-discovery compatibility check; smoke results can be ok, degraded, unsupported, or skipped with reasons.

Use verification after any applied install, uninstall, migration, adoption, or rollback. It is intentionally narrower than precheck: precheck checks software availability, while verify checks whether this installer still owns the files and managed instruction blocks it recorded. For adopted user-owned files, verification checks that the file still matches the hash recorded at adoption time.

Use lifecycle-test as the default installer acceptance gate. It creates fake roots, runs dry-run install, confirms the dry-run did not write files, applies the install, compares normalized dry-run and applied actions, runs verify and smoke, dry-runs uninstall, applies uninstall, and confirms the fake root returns to its baseline outside installer state, including directories. Fake roots are deleted after successful cases unless --keep-fake-roots is passed. fake-root-lifecycle runs the same checks for a caller-selected install scope. The matrix treats forced symlink mode as an expected-degraded smoke scenario when Codex or DeepSeek is included, because those adapters may not load file-symlinked SKILL.md files without native evidence. Use --matrix stress for broader local coverage: all skills, all portable workflow artifacts with backing skills, individual-agent installs, paths with spaces, changed managed files, missing managed files, outside-root state tampering, and corrupt state reporting.

Common commands:

make lifecycle-test ARGS="--matrix default --platform-shape all"
make lifecycle-test ARGS="--matrix full --platform-shape linux"
make lifecycle-test ARGS="--matrix stress --platform-shape linux"
make fake-root-lifecycle ARGS="--skill zotero --platform-shape linux"
make fake-root-lifecycle ARGS="--skill self-improving-agent --platform-shape all"
make runtime-smoke
make runtime-smoke ARGS="--skills self-improving-agent"
make installed-runtime-smoke
make install ARGS="--profile research-core --apply --root <fake-root> --post-install-smoke strict"
make verify ARGS="--root <fake-or-real-root>"
make verify ARGS="--skill zotero --root <fake-or-real-root>"
make verify ARGS="--skills zotero,docling --root <fake-or-real-root>"
make smoke ARGS="--skill zotero --root <fake-or-real-root>"
python3 -m installer.ai_agents_skills --json runtime-inventory --source-root <runtime-root>

Fast local maintainer checks:

make static-check
make sanitize-check
make test
make docs-check
make runtime-smoke
make lifecycle-test ARGS="--matrix default --platform-shape all"

Closest single-host CI parity pass:

make static-check
make sanitize-check
make test
make docs-check
python3 -m pip install networkx psutil
make runtime-smoke
python3 -m pip install -r docs/requirements.txt
make docs-site
make lifecycle-test ARGS="--matrix stress --platform-shape all"

Linux CI runs the stress lifecycle matrix across all platform shapes. Python 3.10 compatibility, macOS, and Windows jobs run narrower subsets that still include static checks, sanitizer checks, tests, and generated-doc checks.

CI checks generated docs with make docs-check, which renders expected docs without mutating README.md or docs/. Run make docs only when you intend to refresh generated files. Run make docs-site after installing docs/requirements.txt when Sphinx rendering matters.

Post-install smoke:

install --apply runs post-install smoke by default in auto mode.
auto runs installer verification, agent-visible skill smoke, and offline runtime smoke for installed runtime-backed skills with safe manifest contracts. Smoke failures are reported, but a successful apply still exits 0.
--post-install-smoke verify runs only installer integrity verification.
--post-install-smoke strict returns nonzero if any post-install check fails, degrades, or is unsupported; the install is still recorded as applied.
--post-install-smoke off skips these checks.

The post-install runtime layer is offline-only. It uses the installed runtime runner, copies managed runtime files into a temporary scratch workspace, strips secret-like environment variables, and forbids live APIs, package installation, MCP/client config writes, and background server starts.

Result meanings:

ok: all selected managed artifacts passed their checks.
no-managed-artifacts: the selected scope has no installer-managed files to check.
missing or failed checks: a managed file, marker, block, or format-specific condition no longer matches recorded state.

CLI exit codes: verify, smoke, runtime-smoke, installed-runtime-smoke, lifecycle-test, and docs-check exit 0 only for ok. Status values such as no-managed-artifacts, degraded, stale, or failed are nonzero unless a higher-level lifecycle scenario intentionally records them as expected.

Current skill checks:

L1 file-exists
L2 installed-signature-match
L3 metadata-valid
L4 managed-marker for copy and reference installs
L5 symlink, source-exists, and source-match for symlink installs
L6 no-secret-leak
L7 agent-visible
L8 adopted-hash-match for adopted user-owned files

Current instruction-block checks:

S1 file-exists
S2 managed-block-present
S3 no-secret-leak for the managed block text only; surrounding user instructions are outside installer ownership

Current support-file checks:

A1 file-exists
A2 installed-signature-match
A3 managed-marker for copied support files
A4 symlink, source-exists, and source-match for symlinked support files
A5 no-secret-leak

Current runtime-file checks:

R1 file-exists
R2 installed-signature-match
R3 source-hash-match after declared newline normalization
R4 runtime-mode
R5 runtime-newline-policy
R6 no-secret-leak

Current optional artifact checks:

O1 file-exists
O2 installed-signature-match
O3 managed-marker
O4 no-secret-leak
O5 format-specific checks for Codex TOML personas and Claude frontmatter

The verifier intentionally skips skills and artifacts that were not installed. Lifecycle tests include smoke checks for installed runtime-backed skills when a smoke command is declared. Runner-specific doctor commands and direct agent-loads-config checks are not automatic yet; use precheck, skill doctors, and the agent’s own diagnostics for those layers.

Use runtime-smoke to install the portable runtime files into a temporary Codex root and execute the installed native runtime runner for the current host. On Windows it exercises run_skill.ps1. CMD runtime entrypoints are not published because CMD cannot preserve arbitrary argument vectors safely. On Linux and macOS it exercises run_skill.sh. The default runtime smoke currently covers autonomous-research-loop-runtime, axiom-axle-mcp, deep-research-workflow, formal-skeleton-helper, get-available-resources, graph-verifier, lean-explore-mcp, lean-formalization-intake, lean-research-library, lean-strict-verification-gate, manim-math-animation, opengauss, remote-bridge, self-improving-agent, send-email, slides-to-video, submission-venue-selector, url-to-screenshot-runtime, venue-ranking-evidence, forcing copy-mode runtime installation in a temporary root. It requires Python plus any dependencies needed by the selected smoke contracts, including psutil and networkx for the default CI path. Passing --skills may only select skills that are supported by this runtime-smoke harness.

Runtime smoke coverage classes are explicit for every runtime-backed skill:

Skill	Coverage	Smoke Contract	Reason
`annotated-review`	`manual-native`	no	Annotation workflows require user-provided documents and optional local tooling; no safe generic offline smoke is declared.
`autonomous-research-loop-runtime`	`offline-smoke`	yes	Smoke validates local loop ledger initialization, append, validation, and status without network, package installs, provider CLIs, config writes, or subagent spawning. The headless driver checks additionally require a host that can contain a primary process, and report themselves as skipped where it cannot.
`axiom-axle-mcp`	`offline-smoke`	yes	Smoke validates inert AXLE setup guidance without installing packages or starting services.
`calibre`	`manual-native`	no	Calibre workflows depend on the user’s local ebook library and profile selection.
`deep-research-workflow`	`offline-smoke`	yes	Selftest smoke is offline and validates the workflow guard contracts.
`digest-bridge`	`static-only`	no	Digest bridge helpers are covered by static/runtime inventory checks; no generic input digest is shipped for smoke.
`docling`	`doctor-only`	no	Docling conversion and OCR need local parser dependencies and documents; use the doctor path for environment checks.
`formal-skeleton-helper`	`offline-smoke`	yes	Smoke writes a minimal local skeleton and validates JSON output without network or secrets.
`get-available-resources`	`offline-smoke`	yes	Smoke records local resource metadata to a temporary file without network or secrets.
`getscipapers-requester`	`manual-native`	no	External paper retrieval is intentionally manual/network-gated and has no generic offline smoke.
`graph-verifier`	`offline-smoke`	yes	Smoke validates a small local graph fixture and JSON result without network.
`hetzner-research-compute`	`manual-native`	no	Hetzner lifecycle verbs require an HCLOUD_TOKEN and provision paid servers; they are not safe for generic offline smoke. The offline dry-run and guard paths are covered by tests/test_hetzner_research_compute.py.
`kaggle-research-compute`	`manual-native`	no	Kaggle lifecycle verbs require the new Kaggle API token (KAGGLE_API_TOKEN or ~/.kaggle/access_token) and push real kernels; per Kaggle ToS no live call is made in the build. The offline dry-run, resume-loop, fan-out, and guard paths are covered by tests/test_kaggle_research_compute.py (all kaggle CLI calls and the kagglehub-validate hook mocked).
`lean-explore-mcp`	`offline-smoke`	yes	Smoke validates inert LeanExplore MCP setup guidance without installing packages, starting services, or calling live APIs.
`lean-formalization-intake`	`offline-smoke`	yes	Doctor smoke records local Lean availability without installing dependencies.
`lean-research-library`	`offline-smoke`	yes	Doctor smoke reports tool/config state offline; every network verb is marked and excluded from smoke.
`lean-strict-verification-gate`	`offline-smoke`	yes	Doctor smoke records local Lean availability and scanner status without installing dependencies.
`manim-math-animation`	`offline-smoke`	yes	Selftest validates scene-spec round-trips, the generated Manim source (Write/MathTex/TransformMatchingTex/emphasis), and the manim/ffmpeg argv builders with no network, package install, Manim, LaTeX, or ffmpeg.
`modal-research-compute`	`manual-native`	no	Modal workflows require explicit external compute credentials and are not safe for generic offline smoke.
`opengauss`	`offline-smoke`	yes	Smoke validates inert OpenGauss readiness guidance without installing OpenGauss, starting gauss, or calling backends. Full install/session is manual-native.
`remote-bridge`	`offline-smoke`	yes	Selftest exercises mailbox arm/CAS/approval single-use, inbox claim/consume, /aas parser boundaries, and in-memory config redaction without network, path synchronization, or real-secret discovery; installed legacy sync/publisher names are inert revocation stubs.
`research-digest-wrapper`	`manual-native`	no	Digest runs depend on configured topics and external feeds; no generic offline smoke is declared.
`rss-news-digest`	`manual-native`	no	RSS digesting depends on configured feeds and network access.
`sagemath`	`manual-native`	no	SageMath availability is host-dependent and too heavy for default offline CI smoke.
`self-improving-agent`	`offline-smoke`	yes	Smoke validates local learning-plan generation without network, package installs, or config writes.
`send-email`	`offline-smoke`	yes	Selftest builds, serializes, and re-parses plain-text, HTML, and attachment messages in memory and checks cc/bcc envelope expansion, port/security inference, header-injection rejection, and password redaction with no network, SMTP connection, package install, or real secrets.
`slides-to-video`	`offline-smoke`	yes	Selftest validates the deterministic core (1:1 pairing, duration re-basing, language-aware engine ladder, math verbalization, effect filtergraph building, caption formatting, clip args, and the SHA-pinned approval gate) with no network, package install, ffmpeg, or TTS.
`submission-venue-selector`	`offline-smoke`	yes	Smoke validates schemas, privacy gates, and offline not-ready behavior without retrieval or secrets.
`tikz-draw`	`manual-native`	no	TikZ workflows depend on TeX toolchains and user-provided figure specs.
`url-to-screenshot-runtime`	`offline-smoke`	yes	Selftest validates the deterministic core (browser-detection candidate order for linux/macos/windows synthetic layouts, SSRF URL-admission gate, CDP command JSON with no –remote-allow-origins flag and a –host-resolver-rules MAP pin, consent-selector list, viewport/full-page arg builders, in-memory blank-output detector, the verify gate on synth golden+blank, and per-OS process-kill strategy selection) with no network, browser launch, or package install.
`venue-ranking-evidence`	`offline-smoke`	yes	Smoke validates local source descriptors and deterministic evidence contracts without network access, browser launch, credentials, or package installation.
`vnthuquan`	`manual-native`	no	Vietnam Thu Quan discovery/download flows are network and library-profile gated.
`zotero`	`manual-native`	no	Zotero workflows depend on the user’s local library, profile, and optional cloud credentials.

make runtime-smoke
make runtime-smoke ARGS="--skills graph-verifier,formal-skeleton-helper"
make runtime-smoke ARGS="--skills self-improving-agent"

Use installed-runtime-smoke after dependencies have been restored to test the managed runtime already installed under the selected root. With no skill filter it executes every installed offline-smoke contract. Declared manual-native, doctor-only, and static-only coverage entries are reported as neutral exclusions; they do not hide an offline-contract failure. A missing or unknown coverage class is a hard failure, so newly added runtime skills cannot silently escape the restore verification gate. A missing native runtime runner also fails when an installed offline contract needs to execute. For a closure restore, --require-complete-coverage additionally requires managed runtime files for every runtime-backed skill declared by the pinned revision. Before execution, managed-state integrity is verified; runtime files are then descriptor-read, SHA-256 checked, copied into an isolated scratch runtime, and only the verified scratch runner is executed.

That scratch runtime is a temporary per-user tree, so it can never be the root-owned component generation run_skill.sh requires of a credential-bearing launch. The harness therefore relaxes credential_runtime_enforcement in the scratch copy, exactly as runtime-smoke relaxes its own ephemeral install, and creates every scratch directory owner-write-only so the runner’s command-chain check still applies. Without both, the gate refuses each credential-bearing skill with exit 127 before its offline contract runs, so those contracts go unexercised while reporting as skill failures. The installed runtime itself is never patched, and the command-chain, workspace, and system-Python checks are never relaxed.

Every installed-runtime report, including an early skipped result, uses the stable top-level schema ai-agents-skills.installed-runtime-smoke.v1 with schema_version: 1. Restore orchestrators must require that exact schema and version, status: ok, unknown_coverage_count: 0, and missing_managed_runtime_count: 0; a skipped report is not readiness evidence. Declared exclusions remain visible through declared_exclusion_count and declared_exclusions and are the only neutral non-executed coverage class.

make installed-runtime-smoke
make installed-runtime-smoke ARGS="--skills graph-verifier,formal-skeleton-helper"
make installed-runtime-smoke ARGS="--require-complete-coverage"

self-improving-agent has a portable offline smoke contract for its cross-target learning review, command-safety, error-detection, and canonical integration-plan helper surface. Native Windows PowerShell/CMD behavior still requires running the Windows ./make.ps1 and runtime runner checks on Windows; Linux-hosted Windows platform-shape tests verify install layout, not native Windows execution.

Docling has a skill-specific runtime doctor because it may rely on a dedicated Docling environment and heavier OCR/model packages that are not part of the default runtime-smoke harness:

bash "${AAS_RUNTIME_ROOT:-$HOME/.local/share/ai-agents-skills/runtime}/run_skill.sh" skills/docling/run_docling.sh doctor

smoke can also return no-managed-artifacts when no managed skill-file artifacts match the selected scope.