awesome-copilot/skills/webmcpify/references/verify.md

Verify — proving every tool works in a real browser

Environment

• Current Chrome (the API moved during the trial — the old navigator.modelContextTesting surface was removed 2026-07 in favor of production document.modelContext.getTools()/executeTool()).
• Enable: chrome://flags/#enable-webmcp-testing, or launch with --enable-features=WebMCP,WebMCPTesting (covers both current and older builds).
• Headed only — WebMCP requires a visible tab by design. In CI, run under xvfb-run. Headless will never work; don't heal toward it.
• App running locally via app.startCommand, against dev/test data only.
• Each tool's manifest entry tells you where and how: route (navigate there), auth (sign in with the recorded test fixture; verify under EACH role for role-scoped tools), examples (what to execute), expect (what to assert), cleanup (how to undo a mutating tool's effect after the test).

The enumeration/execution surface (probe, don't assume)

In the page context, prefer the production surface and fall back for older builds:

const mc = document.modelContext ?? navigator.modelContext;
const tools = mc?.getTools
  ? await mc.getTools()
  : await navigator.modelContextTesting?.listTools();   // removed 2026-07; legacy only

Contract facts that generated assertions MUST respect:

• Enumerated inputSchema is a stringified JSON Schema — JSON.parse before comparing against the manifest entry.
• executeTool(...) resolves to a string result, or null when the execution navigated (normal for declarative forms that submit-navigate).
• Execution and declarative-validation failures reject the promise — they do not resolve to "ERROR: ...". Only imperative tools following the runtime's convention resolve with "ERROR: ..." strings. Assert accordingly per tool kind.
• Registration is asynchronous — registerTool() returns a promise, so a tool is not enumerable the instant the page loads. Poll for it (waitForTool in the template) or await a toolchange event; never assert presence immediately after goto.
• Mutating declarative forms pause mid-execution: Chrome fills the form, then waits for a real submit interaction before letting executeTool settle — awaiting it alone deadlocks into a timeout. Use the concurrent pattern: start executeTool unawaited → wait for an agent-filled value to appear → click submit → await the result (full example in the template).
• These surfaces are for agents/harnesses only — they must never appear in shipped application code.

For declarative tools also verify the synthesized schema: the form-control → schema mapping is only partially specified, so check each annotated control appears as the expected property in the actual target Chrome build.

Per-tool checks

1. Registered (poll — registration is async) with the expected name, the (parsed) schema, and the manifest annotations on the enumerated tool. The legacy modelContextTesting fallback cannot enumerate annotations — skip that assertion there and note the gap in the report.
2. Valid example executes: assert the result per expect — expect.result as a substring of the resolved string, or expect.navigation as the destination when executeTool resolves null (it navigated) — and the expect.ui state as a delta (capture the relevant state before executing; mere visibility of something already on screen proves nothing). A tool that reports success without the UI changing is a fail (UI-settled rule). Because executions can navigate, restore the manifest route in beforeEach, not beforeAll.
3. Invalid example: prove the tool is present first (a rejection from a never-registered tool is not a validation rejection). Then: imperative → resolves "ERROR: ..."; declarative/schema violation → rejects. Zero-param read tools with examples.invalid: null get the dual-outcome assertion instead: {"unexpected": true} may be rejected with a validation reason OR resolve benignly — both pass; a missing tool/surface fails.
4. Mutating tools: run against disposable data, verify the mutation through the same read path the UI uses, then execute the manifest cleanup — a mutating: "server" tool without working cleanup blocks at the gate, and heal-loop retries of mutating tools must clean up between attempts.

Harness

Instantiate templates/webmcp.spec.ts (bundled with this skill) — Playwright, headed persistent Chrome, one describe-block per tool generated from the manifest, with real assertions (never commented-out placeholders). Put the generated spec next to the repo's existing e2e tests.

Repos without a test setup — the standalone-harness recipe. The spec stays in .webmcpify/webmcp.spec.ts (single source of truth, committed per the gate's commitWebmcpifyDir choice); the Playwright installation lives in a scratch harness OUTSIDE the repo so the target gains no dependencies:

mkdir -p /tmp/webmcpify-harness && cd /tmp/webmcpify-harness
npm init -y && npm i -D @playwright/test typescript @types/node
cat > playwright.config.ts <<'EOF'
import { defineConfig } from '@playwright/test';
export default defineConfig({
  testDir: process.env.WEBMCP_SPEC_DIR,   // → <target-repo>/.webmcpify
  workers: 1,                              // one shared headed Chrome — never parallelize
});
EOF
WEBMCP_SPEC_DIR=<target-repo>/.webmcpify WEBMCP_BASE_URL=http://localhost:5173 \
NODE_PATH=/tmp/webmcpify-harness/node_modules npx playwright test

NODE_PATH lets the out-of-repo spec resolve @playwright/test; if the target's tooling ignores NODE_PATH, symlink instead: ln -s /tmp/webmcpify-harness/node_modules <target-repo>/.webmcpify/node_modules (and make sure it isn't committed). Note in the report that verification ran from a standalone harness.

Alternative: Puppeteer ships a first-class experimental WebMCP API (https://pptr.dev/guides/webmcp) — prefer it when the target repo already uses Puppeteer.

Tool-selection evals (recommended; mandatory for SaaS-scale toolsets)

Schema-level verification proves tools work, not that an LLM picks them. For apps exposing more than a handful of tools, run Google's WebMCP Evals CLI (GoogleChromeLabs/webmcp-tools, evals-cli): write one eval case per tool from the manifest examples ("user says X → expect tool Y with args Z") and run them — this catches ambiguous names/descriptions and overlapping tools that Playwright cannot.

Manual QA (tell the human in the report)

• DevTools → Application → WebMCP pane: live tool list, invocation log, "Run tool" with editable params.
• Model Context Tool Inspector Chrome extension (by Google's François Beaufort): natural-language smoke tests of tool selection.
• Chrome's WebMCP audits flag missing toolname/toolparamdescription/ label[for]/name on declarative forms.