awesome-copilot/extensions/connector-namespaces/test/README.md

# MCP smoke test

A standalone harness that proves the Microsoft first-party MCP servers behind a
connector gateway actually work end-to-end:

```
connect → initialize → tools/list → a safe tools/call
```

It imports the `connector-namespaces` extension's real pipeline (`install.mjs`,
`catalog.mjs`, `armClient.mjs`) and connects through the same native Streamable
HTTP endpoint that the extension writes to the GitHub Copilot MCP config. The probe
uses the configured `X-API-Key`, follows `Mcp-Session-Id`, and accepts standard
JSON or SSE JSON-RPC responses.

The whole point: it runs with **Node and a browser sign-in**. No Copilot app or
canvas is required. Hand it to anyone (e.g. Arjun) and they can reproduce an
MCP server issue locally.

## Prerequisites

1. **A browser for Microsoft Entra sign-in.** The harness opens the same
   interactive Azure sign-in as the extension when its encrypted Azure Identity
   session is not already available.
2. **A gateway already picked once.** The harness reads gateway coordinates from
   `~/.copilot/extensions/connector-namespaces/artifacts/gateway-config.json`
   (`{ subscriptionId, resourceGroup, gatewayName }`). Pick a gateway once in
   the connector-namespaces canvas, or write that file by hand.
3. **An operating-system secure credential store.** Windows and macOS provide one
   by default. Linux and WSL require a Secret Service-compatible keyring, such as
   GNOME Keyring, with `libsecret` available. Unencrypted token storage is
   intentionally disabled.
4. **Node 20+** (developed on Node 24).
5. **Extension dependencies installed.** From the repository root, run:

   ```bash
   npm install --prefix extensions/connector-namespaces
   ```

## Run it

```bash
node extensions/connector-namespaces/test/smoke.mjs
```

Options:

| flag | effect |
|---|---|
| `--only=a,b` | only test these `apiName`s (comma-separated) |
| `--limit=N` | stop after N connectable servers |
| `--open-consent` | open consent URLs in the browser for OAuth servers that need it |
| `--no-cleanup` | leave fresh keyless installs in place (default: uninstall them) |

Examples:

```bash
# just the three already-connected WorkIQ servers
node extensions/connector-namespaces/test/smoke.mjs --only=WorkIQMail,WorkIQSharePoint,WorkIQTeams

# first 5 connectable servers, open any consent prompts
node extensions/connector-namespaces/test/smoke.mjs --limit=5 --open-consent
```

## One-time connector consent

OAuth-backed servers (most of them) need a human to consent **once** in a
browser. Azure ARM sign-in is restored from the operating system's encrypted
credential store described in the prerequisites; the one-time behavior below
applies to the connector's own consent. The model:

1. **First run** hits a server that needs consent → the harness prints a consent
   URL and marks it `NEEDS_CONSENT`. It saves a pending record to
   `~/.copilot/extensions/connector-namespaces/artifacts/smoke-pending-consent.json` (not in
   the repo). No tool call is attempted.
2. **You open that URL once** and sign in / consent. After sign-in the browser
   may show "this site can't be reached" on a `127.0.0.1:7333/auth/callback/`
   page — **that is expected and harmless.** Consent completes gateway-side; the
   loopback page is just a redirect target and nothing is listening on it.
3. **Re-run the harness.** It sees the pending record, confirms the gateway
   connection is now `Connected`, finishes the install (mints the API key,
   writes the Copilot MCP entry), and probes it headless. From then on the connector is
   reused without repeating its consent.

So the server taxonomy is:

- **Already connected** (e.g. the three WorkIQ servers) → probed immediately.
- **Keyless / SP / AAD** (e.g. Microsoft Learn Docs) → installed + probed +
  cleaned up immediately, no consent.
- **Consent-once OAuth** → surfaced on run 1, converts to headless on run 2.

That's why the **first** run may probe fewer than 10 servers — the rest are
waiting on their one-time consent. Consent the URLs it prints, re-run, and the
count climbs. This is inherent to the consent model, not a harness bug.

## Tool-call safety

The harness never blindly calls the first tool a server advertises (mutation
risk). `safe-tools.mjs` picks a tool to call by:

1. a **curated map** of known-safe read tools per server (e.g. Microsoft Learn
   Docs → `microsoft_docs_search`, WorkIQ Teams → `ListTeams`), then
2. a **read-only-name heuristic** fallback — the first tool whose name starts
   with `list`/`get`/`search`/`read`/`find`/… **and** whose required arguments
   are empty or trivially fillable with benign values.

If nothing looks safe, it does `tools/list` only and records the call as
`SKIPPED` (tools proven to load, no call made). Expand the curated map in
`safe-tools.mjs` as you learn each server.

## Reading the report

Each run prints a summary and writes two files to `test/reports/` (gitignored —
they contain live endpoint URLs):

- `mcp-smoke-<timestamp>.log` — human-readable table. **This is the handoff
  artifact** — attach it to a bug or send it to whoever needs to repro.
- `mcp-smoke-<timestamp>.json` — machine-readable, same data.

Per server you get: classification, `initialize` pass/fail + latency, tool
count, which tool was called and why, the call result preview or error, and a
direct transport error on failure. API keys are redacted; endpoint URLs are not,
which is why the reports stay out of git.

Exit code is **non-zero if any probed server failed a step**, so it's CI-usable.

## Files

| file | role |
|---|---|
| `smoke.mjs` | orchestrator — bootstrap, classify each server, probe, report |
| `mcp-probe.mjs` | drives the native Streamable HTTP JSON-RPC handshake |
| `safe-tools.mjs` | curated safe-read-tool map + read-only heuristic + arg filler |
| `reports/` | generated `.log` + `.json` artifacts (gitignored) |

## Scope

Microsoft first-party servers only (`category === "Microsoft"` in the catalog).
Partner servers (Box, Celonis, …) are filtered out — they need partner accounts
and OAuth we can't automate.