mirror of
https://github.com/github/awesome-copilot.git
synced 2026-09-07 08:15:58 +00:00
chore: publish from main
This commit is contained in:
github-actions[bot]
@@ -0,0 +1,160 @@
|
||||
---
|
||||
name: docs-sync-audit
|
||||
description: Run a read-only documentation drift audit for a feature, PR, branch, release, API, configuration change, workflow, CLI, package, or repository area. Use when the user asks whether docs are stale, missing, inconsistent with code, or need updates after code changes. Checks README files, setup guides, API docs, env docs, changelogs, examples, comments, generated docs, and user-facing instructions. This is not a general code review; it compares what the docs claim against what the code does.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Docs Sync Audit
|
||||
|
||||
Check whether documentation still matches the code, configuration, API behavior, commands, examples, and user workflows. Report stale or missing docs with concrete evidence and update direction.
|
||||
|
||||
## Core Rules
|
||||
|
||||
- Stay read-only unless the user explicitly asks to update docs.
|
||||
- Default to a full-repository docs audit when the user does not provide a specific scope. Inventory the repo's docs surfaces (README, docs directories, examples, CLI help, API contracts, config samples) and compare them against the code they describe.
|
||||
- Full-repo audits are breadth-first, then depth-limited. Inventory the repo, rank surfaces by risk, deep-inspect as many high-risk surfaces as the turn allows, and list the rest under **Surveyed But Not Deeply Inspected** with a pointer to run another pass on them. State the surface counts in the report header. Never present a shallow sweep as complete coverage.
|
||||
- Ground every finding in both sides of the mismatch: the code/config/source of truth and the stale or missing documentation.
|
||||
- Separate confirmed drift from inferred doc gaps.
|
||||
- Prefer user-impacting docs drift over cosmetic wording issues.
|
||||
- Do not report style preferences unless they make instructions misleading, incomplete, or hard to follow.
|
||||
- Treat generated docs carefully: identify the generator, source file, and expected generation command before recommending direct edits.
|
||||
- If generated docs appear stale but were not regenerated, say so explicitly and report the residual risk instead of implying the generated output was verified.
|
||||
- Avoid creating docs during the audit phase.
|
||||
|
||||
## Inputs
|
||||
|
||||
Accept any docs-sync target, including:
|
||||
|
||||
- PRs or branches: `audit docs for this PR`, `what docs need updating before release`.
|
||||
- Features: `docs sync for uploads`, `check billing docs after this change`.
|
||||
- APIs: `audit OpenAPI docs against handlers`, `check SDK examples for the new endpoint`.
|
||||
- Config/setup: `env docs drift`, `README setup audit`, `Docker docs sync`.
|
||||
- CLI/workflows: `check command docs`, `does onboarding match the current flow`.
|
||||
- Whole repo docs hygiene when explicitly requested.
|
||||
|
||||
If scope is unclear, infer the smallest useful boundary and state it. If no scope is stated, do not ask for one; proceed with a full-repo docs audit. Ask only when different scopes would produce materially different doc checks.
|
||||
|
||||
## Discovery Workflow
|
||||
|
||||
1. Establish source of truth.
|
||||
- Check `git status --short`.
|
||||
- For PR/branch audits, identify the base and changed files when possible.
|
||||
- Locate manifests, scripts, routes, configs, schema files, migrations, API handlers, CLI entrypoints, env validation, generated-doc sources, and tests that reveal expected behavior.
|
||||
|
||||
2. Locate related documentation.
|
||||
- Search README files, docs folders, API docs, OpenAPI/Swagger specs, changelogs, setup guides, deployment docs, env examples, examples, fixtures, comments, storybook/docs pages, package docs, and runbooks.
|
||||
- Include docs near the feature and docs users would reasonably consult first.
|
||||
- For generated docs, locate the source file, generator command, committed output, and any docs build or codegen step before deciding where updates belong.
|
||||
|
||||
3. Compare code and docs.
|
||||
- Run the bundled `scripts/docs_drift.py` first when it is available. It checks only claims with a definite answer: documented `npm run` scripts and `make` targets against the ones that exist, relative Markdown links against the filesystem, and environment variable names in both directions between docs and code. The path is relative to this skill's own directory, which varies by host. Use `python` if `python3` is not on PATH.
|
||||
- `python <skill-dir>/scripts/docs_drift.py --top 30`, or `--format json` to filter results yourself.
|
||||
- It flags a documented setting that is read only inside a module nothing imports, which is config that reads as working but cannot take effect. Confirm the module really is unreachable before reporting it: the check uses name matching and cannot see dynamic imports.
|
||||
- Add `--check-paths` only when you want backticked paths checked too. It is off by default because most such references are ambiguous, and on a large repo the noise buries the real findings. Read its output as leads, not findings.
|
||||
- The script never judges prose. Wording, completeness, and whether an explanation is actually correct are your job, and are usually where the important drift is.
|
||||
- Commands/scripts: names, arguments, package manager, working directory, prerequisites, outputs.
|
||||
- APIs: routes, methods, auth requirements, request/response shape, status codes, errors, pagination, webhooks, versioning.
|
||||
- Config/env: required vars, defaults, examples, secrets, feature flags, deployment settings.
|
||||
- UI/workflows: screens, labels, steps, permissions, roles, states, screenshots, examples.
|
||||
- Data/schema: fields, migrations, enums, limits, constraints, seed data, import/export formats.
|
||||
- Tests/examples: sample code, fixtures, SDK usage, curl examples, screenshots, expected outputs.
|
||||
|
||||
4. Verify safely.
|
||||
- Run low-risk commands that reveal docs/source mismatch when available: docs build, link check, typecheck examples, OpenAPI generation, CLI help, package scripts, or focused tests.
|
||||
- Do not install dependencies or regenerate large docs unless the user asks or the repo clearly expects it.
|
||||
- Never run a command that writes into the repository as a side effect. `python -m compileall` and `py_compile` emit `.pyc` files, formatters rewrite sources, and installers touch lockfiles. `.pyc` output is usually gitignored, so `git status` will look clean while the tree has in fact been modified. Prefer checks that write nothing, and if a language offers no read-only check, say so under checks skipped.
|
||||
- Record checks run and checks skipped.
|
||||
|
||||
## What To Look For
|
||||
|
||||
- README setup instructions that no longer work.
|
||||
- Missing docs for new routes, commands, env vars, permissions, flags, migrations, webhooks, or user workflows.
|
||||
- Old names, paths, screenshots, labels, examples, or config keys after a rename.
|
||||
- API docs that disagree with handlers, schemas, validation, auth, errors, or status codes.
|
||||
- Changelog/release notes missing user-visible or operational changes.
|
||||
- `.env.example`, deployment docs, or runbooks missing required configuration.
|
||||
- Example code that imports old paths, calls old APIs, uses stale package names, or omits required setup.
|
||||
- Generated docs committed but stale relative to source.
|
||||
- Comments or architecture docs that describe an older module boundary or behavior.
|
||||
|
||||
## Severity Rubric
|
||||
|
||||
- `P0`: Docs drift could cause production outage, data loss, security exposure, broken deploy, credential mishandling, or critical operational failure.
|
||||
- `P1`: High-impact docs drift that blocks setup, release, API integration, migration, support, or a common user/admin workflow.
|
||||
- `P2`: Meaningful stale or missing docs likely to confuse users, reviewers, operators, SDK consumers, or contributors.
|
||||
- `P3`: Lower-risk docs cleanup, naming drift, examples, comments, or polish that should be queued.
|
||||
|
||||
## Evidence Standards
|
||||
|
||||
- Verify every citation before you write it. Re-read the exact range and confirm it contains what you are describing. When citing a named symbol, function, CTE, or block, cite the line where the name is defined, not a line inside a neighbouring block. When quoting text, cite the file the quote is actually in. Prefer a single anchor line containing a distinctive token over a hand-counted range.
|
||||
- When you attribute a finding to a tool's output, quote the path and line the tool itself reported. Never infer which lines a linter or type checker fired on by reading the code. If the tool's output does not name the line, report the pattern without claiming the tool flagged it.
|
||||
- Never restate a count from a grep, a script, or a tool without the raw output in front of you. If you cannot re-derive the number, describe the pattern instead of counting it.
|
||||
- Before reporting that something is absent -- undocumented config, an unused dependency, a missing control, a variable nothing reads -- check every plausible location, not the first one. For a config variable that means the README, env sample files, deploy manifests, comments, and the transitive callers of whatever helper reads it. For a dependency it means whether it is a documented transitive requirement of something you do use. A negative claim from a single grep is not evidence.
|
||||
- Cite the source of truth and the stale/missing documentation.
|
||||
- For missing docs, cite the code/config/change that should be documented and the doc area where users would expect it.
|
||||
- Include exact paths and line references whenever possible.
|
||||
- State whether the docs are confirmed stale, likely stale, or missing based on inference.
|
||||
- Do not claim docs are safe to delete unless references, links, generated sources, and navigation were checked.
|
||||
|
||||
## Report Format
|
||||
|
||||
Use this structure unless the user asks otherwise:
|
||||
|
||||
```markdown
|
||||
**Docs Sync Audit: <scope>**
|
||||
|
||||
No code changed. I compared <source/code/change scope> against <docs checked>. <verification summary>. No P0s found / P0s found: <count>.
|
||||
|
||||
1. **P1: <finding title>.**
|
||||
Drift: <what docs say or omit vs what code/config does>.
|
||||
Impact: <who is misled or blocked>.
|
||||
Evidence: source `<path>:<line>`; docs `<path>:<line>`.
|
||||
Suggested update: <specific docs change direction>.
|
||||
|
||||
2. **P2: <finding title>.**
|
||||
Drift: <what is stale/missing>.
|
||||
Impact: <why it matters>.
|
||||
Evidence: source `<path>:<line>`; docs `<path>:<line>` or expected docs area.
|
||||
Suggested update: <specific direction>.
|
||||
|
||||
**Likely Docs To Update**
|
||||
- `<path>`: <why>
|
||||
|
||||
**Surveyed But Not Deeply Inspected**
|
||||
- <For full-repo audits only: surfaces that were inventoried but not inspected deeply this pass, and which to run next. Omit this section entirely for scoped audits.>
|
||||
|
||||
**Checks Run**
|
||||
- `<command>`: <result>
|
||||
|
||||
**Not Tested**
|
||||
- <docs build, link check, generated-doc rebuild, or external-doc gaps and why; state residual risk when generated output was not rebuilt>
|
||||
|
||||
**Assumptions**
|
||||
- <only include if useful>
|
||||
```
|
||||
|
||||
If no drift is found, say that clearly and list residual risks such as generated docs not rebuilt, docs build/link checks not run, or external docs not accessible.
|
||||
|
||||
## Post-Audit Update Workflow
|
||||
|
||||
When the user asks to update docs:
|
||||
|
||||
- Update only docs related to confirmed drift or explicitly selected inferred gaps.
|
||||
- Preserve the repo's documentation style, structure, and terminology.
|
||||
- Update generated docs from the source/generator when practical instead of editing generated output directly.
|
||||
- Update examples, screenshots, changelogs, env examples, API specs, and runbooks together when they describe the same behavior.
|
||||
- Run docs build, link check, example typecheck, or focused verification when available.
|
||||
- Final response should map findings to updated files and list checks run.
|
||||
|
||||
## Related Skills
|
||||
|
||||
This skill is one of seven review skills that share a single report contract:
|
||||
every finding carries a `P0`-`P3` severity and a `path:line` you can open. The
|
||||
other five cover launch readiness, security, repo structure, improvement ideas,
|
||||
and pull request communication. They are at https://github.com/specialone0007/review-skills.
|
||||
|
||||
## Agent Portability Notes
|
||||
|
||||
- Use available shell, search, git, browser, GitHub, docs, or MCP tools as appropriate.
|
||||
- If web docs, private docs, rendered docs, or external API docs are unavailable, continue with local source inspection and state the limitation.
|
||||
- If the host supports inline review comments, emit them only for confirmed actionable docs drift and keep ranges tight.
|
||||
@@ -0,0 +1,474 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check machine-verifiable documentation claims against the repository.
|
||||
|
||||
Read-only. Standard library only. Writes nothing.
|
||||
|
||||
python docs_drift.py # text report
|
||||
python docs_drift.py --repo ../other # a different repo
|
||||
python docs_drift.py --format json
|
||||
python docs_drift.py --no-git-root # scope to one package of a monorepo
|
||||
|
||||
Only checks claims that have a definite answer:
|
||||
|
||||
commands `npm run x`, `make x`, `./scripts/x` in a fenced block, against the
|
||||
scripts, targets and files that actually exist
|
||||
links relative Markdown links and images, against the filesystem
|
||||
paths backticked paths, against the filesystem (opt-in, --check-paths)
|
||||
env vars names documented in docs or .env.example, against names actually
|
||||
read by the code, in both directions
|
||||
staleness a doc untouched for far longer than the code it describes
|
||||
|
||||
It does not judge prose. Wording, tone, completeness and accuracy of explanation
|
||||
are the reviewing agent's job; this exists so the agent does not spend forty tool
|
||||
calls confirming whether a path exists.
|
||||
|
||||
Backticked-path checking is opt-in. On real repositories most such references are
|
||||
ambiguous -- a path the doc is telling you to create, or one an archived report
|
||||
described accurately at the time -- and reporting them buries the findings that
|
||||
are unambiguous. Markdown links are always checked, because a link is a promise
|
||||
to resolve.
|
||||
|
||||
Values are never read out of environment files. Only the names to the left of `=`
|
||||
are used, because the right-hand side is a credential by design.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
if hasattr(sys.stdout, "reconfigure"):
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
GIT_TIMEOUT = 30
|
||||
MAX_READ = 2_000_000
|
||||
STALE_DAYS = 120
|
||||
|
||||
SKIP_DIRS = {
|
||||
".git", "node_modules", "vendor", "venv", ".venv", "dist", "build", "target",
|
||||
"__pycache__", ".next", "coverage", ".terraform", "site-packages",
|
||||
}
|
||||
DOC_EXTS = {".md", ".mdx", ".rst", ".txt"}
|
||||
CODE_EXTS = {
|
||||
".py", ".js", ".jsx", ".ts", ".tsx", ".mjs", ".cjs", ".vue", ".svelte",
|
||||
".go", ".rs", ".rb", ".php", ".java", ".kt", ".swift", ".cs", ".ex", ".exs", ".sh",
|
||||
}
|
||||
|
||||
FENCE = re.compile(r"^```")
|
||||
# Commands worth checking. Anything else in a fenced block is left alone.
|
||||
CMD_NPM = re.compile(r"\b(?:npm|pnpm|yarn|bun)\s+run\s+([A-Za-z0-9:_.-]+)")
|
||||
CMD_MAKE = re.compile(r"\bmake\s+([A-Za-z0-9_.-]+)")
|
||||
CMD_SCRIPT = re.compile(r"(?:^|\s)(\./[A-Za-z0-9_./-]+|(?:python3?|node|bash|sh|ruby)\s+([A-Za-z0-9_./-]+\.[A-Za-z0-9]+))")
|
||||
|
||||
MD_LINK = re.compile(r"!?\[[^\]]*\]\(([^)\s]+)")
|
||||
BACKTICK = re.compile(r"`([^`\n]+)`")
|
||||
|
||||
# Placeholder shapes that are not meant to resolve.
|
||||
PLACEHOLDER = re.compile(
|
||||
r"[<>{}$*]|^\.{3}|\.{3}$|(^|/)(path/to|your[-_]|my[-_]|example|foo|bar|baz|placeholder)",
|
||||
re.I)
|
||||
|
||||
ENV_IN_CODE = [
|
||||
re.compile(r"process\.env\.([A-Z][A-Z0-9_]*)"),
|
||||
re.compile(r"""process\.env\[\s*['"]([A-Z][A-Z0-9_]*)['"]"""),
|
||||
re.compile(r"""os\.environ(?:\.get)?\[?\(?\s*['"]([A-Z][A-Z0-9_]*)['"]"""),
|
||||
re.compile(r"""os\.getenv\(\s*['"]([A-Z][A-Z0-9_]*)['"]"""),
|
||||
re.compile(r"""getenv\(\s*['"]([A-Z][A-Z0-9_]*)['"]"""),
|
||||
re.compile(r"""ENV\[\s*['"]([A-Z][A-Z0-9_]*)['"]"""),
|
||||
re.compile(r"""Deno\.env\.get\(\s*['"]([A-Z][A-Z0-9_]*)['"]"""),
|
||||
]
|
||||
ENV_NAME = re.compile(r"\b([A-Z][A-Z0-9_]{2,})\b")
|
||||
|
||||
# Import forms across the languages handled above. Four alternatives, so findall
|
||||
# returns tuples and the caller takes the first non-empty group.
|
||||
IMPORT_SPEC = re.compile(
|
||||
r"""(?:from|import)\s+['"]([^'"]+)['"]"""
|
||||
r"""|require\(\s*['"]([^'"]+)['"]\s*\)"""
|
||||
r"""|^\s*from\s+([A-Za-z0-9_.]+)\s+import"""
|
||||
r"""|^\s*import\s+([A-Za-z0-9_.]+)""",
|
||||
re.M)
|
||||
|
||||
warnings: list[str] = []
|
||||
|
||||
|
||||
def run_git(args: list[str], cwd: Path) -> str | None:
|
||||
git = shutil.which("git")
|
||||
if git is None:
|
||||
return None
|
||||
try:
|
||||
p = subprocess.run([git, *args], cwd=str(cwd), text=True, timeout=GIT_TIMEOUT,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
encoding="utf-8", errors="replace")
|
||||
except (OSError, subprocess.SubprocessError) as exc:
|
||||
warnings.append(f"git {' '.join(args[:2])} failed: {exc}")
|
||||
return None
|
||||
return p.stdout if p.returncode == 0 else None
|
||||
|
||||
|
||||
def list_files(repo: Path) -> list[str]:
|
||||
out = run_git(["ls-files", "--cached", "--other", "--exclude-standard"], repo)
|
||||
if out is not None and out.strip():
|
||||
return sorted(x.strip() for x in out.splitlines() if x.strip())
|
||||
warnings.append("git unavailable or empty index; walking the filesystem instead")
|
||||
files = []
|
||||
for p in repo.rglob("*"):
|
||||
if p.is_file() and not any(part in SKIP_DIRS for part in p.parts):
|
||||
files.append(p.relative_to(repo).as_posix())
|
||||
return sorted(files)
|
||||
|
||||
|
||||
def read(path: Path) -> str:
|
||||
try:
|
||||
if path.stat().st_size > MAX_READ:
|
||||
return ""
|
||||
return path.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def available_commands(repo: Path, files: list[str]) -> tuple[dict[str, set[str]], set[str]]:
|
||||
"""Real npm scripts and make targets, keyed by the directory that declares them."""
|
||||
npm: dict[str, set[str]] = {}
|
||||
make: set[str] = set()
|
||||
for rel in files:
|
||||
base = Path(rel).name
|
||||
prefix = str(Path(rel).parent).replace("\\", "/")
|
||||
if base == "package.json":
|
||||
try:
|
||||
data = json.loads(read(repo / rel))
|
||||
except ValueError:
|
||||
continue
|
||||
if isinstance(data.get("scripts"), dict):
|
||||
npm.setdefault(prefix, set()).update(data["scripts"].keys())
|
||||
elif base == "Makefile":
|
||||
targets = re.findall(r"^([A-Za-z0-9][A-Za-z0-9_.-]*):(?!=)", read(repo / rel), re.M)
|
||||
make.update(targets)
|
||||
return npm, make
|
||||
|
||||
|
||||
def fenced_blocks(text: str) -> list[tuple[int, str]]:
|
||||
"""Yield (line_number, line) for lines inside fenced code blocks."""
|
||||
out: list[tuple[int, str]] = []
|
||||
inside = False
|
||||
for i, line in enumerate(text.splitlines(), start=1):
|
||||
if FENCE.match(line.strip()):
|
||||
inside = not inside
|
||||
continue
|
||||
if inside:
|
||||
out.append((i, line))
|
||||
return out
|
||||
|
||||
|
||||
def env_names_from_code(repo: Path, files: list[str]) -> dict[str, list[str]]:
|
||||
"""Env var names the code reads, mapped to every location that reads them."""
|
||||
found: dict[str, list[str]] = {}
|
||||
for rel in files:
|
||||
if Path(rel).suffix not in CODE_EXTS or any(p in SKIP_DIRS for p in Path(rel).parts):
|
||||
continue
|
||||
text = read(repo / rel)
|
||||
if not text or "env" not in text.lower():
|
||||
continue
|
||||
for i, line in enumerate(text.splitlines(), start=1):
|
||||
for pattern in ENV_IN_CODE:
|
||||
for name in pattern.findall(line):
|
||||
found.setdefault(name, []).append(f"{rel}:{i}")
|
||||
return found
|
||||
|
||||
|
||||
def unreferenced_modules(repo: Path, files: list[str]) -> set[str]:
|
||||
"""Code files that nothing imports, and that are not plausible entrypoints.
|
||||
|
||||
A documented setting read only inside such a file is configuration that cannot
|
||||
take effect, which reads as working config in the docs. Deliberately
|
||||
conservative: basename matching, and anything entrypoint-shaped is excluded, so
|
||||
it under-reports rather than accusing live code of being dead.
|
||||
"""
|
||||
code = [f for f in files
|
||||
if Path(f).suffix in CODE_EXTS and not any(p in SKIP_DIRS for p in Path(f).parts)]
|
||||
entrypoint = re.compile(
|
||||
r"(^|/)(server|main|index|app|cli|__init__|__main__|conftest|setup|wsgi|asgi)\.[A-Za-z]+$",
|
||||
re.I)
|
||||
imported: set[str] = set()
|
||||
for rel in code:
|
||||
text = read(repo / rel)
|
||||
if not text:
|
||||
continue
|
||||
for groups in IMPORT_SPEC.findall(text):
|
||||
ref = next((g for g in groups if g), "").strip()
|
||||
if not ref:
|
||||
continue
|
||||
imported.add(Path(ref).name.lower())
|
||||
imported.add(Path(ref).stem.lower())
|
||||
for part in re.split(r"[./\:]", ref):
|
||||
if part:
|
||||
imported.add(part.lower())
|
||||
out = set()
|
||||
for rel in code:
|
||||
if entrypoint.search(rel):
|
||||
continue
|
||||
stem = Path(rel).stem.lower()
|
||||
if stem not in imported and Path(rel).name.lower() not in imported:
|
||||
out.add(rel)
|
||||
return out
|
||||
|
||||
|
||||
def env_names_documented(repo: Path, files: list[str]) -> dict[str, str]:
|
||||
"""Env var names named in docs or declared in an env sample file.
|
||||
|
||||
Only the key to the left of `=` is ever read from an env file. The value is a
|
||||
credential by design and is never touched.
|
||||
"""
|
||||
documented: dict[str, str] = {}
|
||||
for rel in files:
|
||||
base = Path(rel).name
|
||||
is_env_sample = base.startswith(".env")
|
||||
if not is_env_sample and Path(rel).suffix not in DOC_EXTS:
|
||||
continue
|
||||
text = read(repo / rel)
|
||||
if not text:
|
||||
continue
|
||||
for i, line in enumerate(text.splitlines(), start=1):
|
||||
if is_env_sample:
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith("#") or "=" not in stripped:
|
||||
continue
|
||||
key = stripped.split("=", 1)[0].strip().lstrip("export ").strip()
|
||||
if ENV_NAME.fullmatch(key or ""):
|
||||
documented.setdefault(key, f"{rel}:{i}")
|
||||
else:
|
||||
# In prose, a backticked all-caps token is weak evidence: `SKILL.md`
|
||||
# and `README` are not configuration. Require an underscore, which is
|
||||
# what actually distinguishes API_TOKEN from a shouted word, and skip
|
||||
# anything that looks like a filename.
|
||||
for chunk in BACKTICK.findall(line):
|
||||
if "." in chunk or "/" in chunk:
|
||||
continue
|
||||
for name in ENV_NAME.findall(chunk):
|
||||
if "_" not in name:
|
||||
continue
|
||||
documented.setdefault(name, f"{rel}:{i}")
|
||||
return documented
|
||||
|
||||
|
||||
def newest_commit_epoch(repo: Path, pathspec: str) -> int | None:
|
||||
out = run_git(["log", "-1", "--format=%at", "--", pathspec], repo)
|
||||
if not out or not out.strip().isdigit():
|
||||
return None
|
||||
return int(out.strip())
|
||||
|
||||
|
||||
def build(repo: Path, files: list[str], check_paths: bool = False) -> dict:
|
||||
findings: list[dict] = []
|
||||
|
||||
def add(kind: str, severity: str, doc: str, line: int | None, detail: str,
|
||||
source: str | None = None) -> None:
|
||||
findings.append({"kind": kind, "severity": severity, "doc": doc, "line": line,
|
||||
"detail": detail, "source": source})
|
||||
|
||||
file_set = set(files)
|
||||
npm_scripts, make_targets = available_commands(repo, files)
|
||||
all_npm = set().union(*npm_scripts.values()) if npm_scripts else set()
|
||||
docs = [f for f in files
|
||||
if Path(f).suffix in DOC_EXTS and not any(p in SKIP_DIRS for p in Path(f).parts)]
|
||||
|
||||
for doc in docs:
|
||||
text = read(repo / doc)
|
||||
if not text:
|
||||
continue
|
||||
|
||||
# 1. commands
|
||||
for lineno, line in fenced_blocks(text):
|
||||
for script in set(CMD_NPM.findall(line)):
|
||||
if not all_npm:
|
||||
continue
|
||||
if script not in all_npm:
|
||||
near = ", ".join(sorted(s for s in all_npm if s.startswith(script.split(":")[0]))[:4])
|
||||
hint = f" Closest existing: {near}." if near else ""
|
||||
add("missing-script", "high", doc, lineno,
|
||||
f"documents `{script}`, which is not a script in any package.json.{hint}",
|
||||
source="package.json")
|
||||
for target in set(CMD_MAKE.findall(line)):
|
||||
if make_targets and target not in make_targets and target not in ("-j", "all"):
|
||||
add("missing-make-target", "high", doc, lineno,
|
||||
f"documents `make {target}`, which is not a target in the Makefile.",
|
||||
source="Makefile")
|
||||
for whole, inner in CMD_SCRIPT.findall(line):
|
||||
candidate = (inner or whole).lstrip("./")
|
||||
if not candidate or PLACEHOLDER.search(candidate):
|
||||
continue
|
||||
if candidate.endswith("/") or any(part in SKIP_DIRS
|
||||
for part in Path(candidate).parts):
|
||||
continue
|
||||
if candidate not in file_set and not (repo / candidate).exists():
|
||||
add("missing-script-file", "high", doc, lineno,
|
||||
f"documents running `{candidate}`, which does not exist.")
|
||||
|
||||
# 2 and 3. links and backticked paths
|
||||
for i, line in enumerate(text.splitlines(), start=1):
|
||||
for target in MD_LINK.findall(line):
|
||||
t = target.split("#")[0].strip()
|
||||
if not t or t.startswith(("http://", "https://", "mailto:", "#", "tel:", "data:")):
|
||||
continue
|
||||
if PLACEHOLDER.search(t):
|
||||
continue
|
||||
if not (repo / Path(doc).parent / t).exists():
|
||||
add("broken-link", "high", doc, i,
|
||||
f"relative link `{t}` does not resolve.")
|
||||
for chunk in (BACKTICK.findall(line) if check_paths else []):
|
||||
c = chunk.strip()
|
||||
# Only treat it as a path claim when it looks like one.
|
||||
if "/" not in c or " " in c or PLACEHOLDER.search(c):
|
||||
continue
|
||||
# A leading slash means a slash-command or an absolute path, e.g.
|
||||
# `/security-review`. Neither is a claim about this repository.
|
||||
if c.startswith("/"):
|
||||
continue
|
||||
if not re.match(r"^[A-Za-z0-9._/-]+$", c) or c.endswith("/"):
|
||||
continue
|
||||
# Require a real file extension. Extensionless slashed tokens are
|
||||
# ambiguous by nature -- `origin/staging` is a git ref, `src/utils/billing`
|
||||
# is an illustrative example, `@scope/pkg` is a package. Accusing those
|
||||
# of being broken paths produced far more noise than signal.
|
||||
if not re.match(r"^\.[A-Za-z0-9]{1,5}$", Path(c).suffix):
|
||||
continue
|
||||
if c in file_set or (repo / c).exists():
|
||||
continue
|
||||
# Docs routinely write paths relative to their own directory.
|
||||
if (repo / Path(doc).parent / c).exists():
|
||||
continue
|
||||
# A directory prefix that exists is close enough not to report.
|
||||
if any(f.startswith(c.rstrip("/") + "/") for f in file_set):
|
||||
continue
|
||||
# Docs also reference a shape that repeats, e.g. `agents/openai.yaml`
|
||||
# when the real files are skills/<name>/agents/openai.yaml. If it is
|
||||
# the tail of a real path, the claim is true enough.
|
||||
tail = "/" + c
|
||||
if any(f.endswith(tail) for f in file_set):
|
||||
continue
|
||||
add("missing-path", "medium", doc, i,
|
||||
f"references `{c}`, which does not exist in the repository.")
|
||||
|
||||
# 4. env vars, both directions
|
||||
in_code = env_names_from_code(repo, files)
|
||||
in_docs = env_names_documented(repo, files)
|
||||
dead = unreferenced_modules(repo, files)
|
||||
for name, where in sorted(in_docs.items()):
|
||||
readers = in_code.get(name, [])
|
||||
doc_path, _, doc_line = where.rpartition(":")
|
||||
if not readers:
|
||||
add("documented-unused-env", "medium", doc_path, int(doc_line),
|
||||
f"`{name}` is documented but nothing in the code reads it. "
|
||||
"Either it is dead configuration or the docs promise a knob that does not exist.",
|
||||
source="no reader found")
|
||||
elif all(r.rsplit(":", 1)[0] in dead for r in readers):
|
||||
where_read = ", ".join(readers[:3])
|
||||
add("documented-env-in-unreferenced-module", "medium", doc_path, int(doc_line),
|
||||
f"`{name}` is read only in a module nothing imports, so the documented setting "
|
||||
"cannot take effect. The docs describe a working knob that does nothing.",
|
||||
source=where_read)
|
||||
for name, readers in sorted(in_code.items()):
|
||||
if name not in in_docs:
|
||||
add("undocumented-env", "medium", "(docs)", None,
|
||||
f"`{name}` is read by the code but is not documented anywhere, "
|
||||
"and is not in an env sample file.",
|
||||
source=readers[0])
|
||||
|
||||
# 5. staleness
|
||||
code_dirs = {str(Path(f).parent).replace("\\", "/") for f in files
|
||||
if Path(f).suffix in CODE_EXTS}
|
||||
newest_code = max((e for e in (newest_commit_epoch(repo, d) for d in list(code_dirs)[:40])
|
||||
if e), default=None)
|
||||
if newest_code:
|
||||
for doc in docs:
|
||||
doc_epoch = newest_commit_epoch(repo, doc)
|
||||
if not doc_epoch:
|
||||
continue
|
||||
days = (newest_code - doc_epoch) / 86400
|
||||
if days > STALE_DAYS:
|
||||
add("stale-doc", "low", doc, None,
|
||||
f"last changed {int(days)} days before the most recent code change. "
|
||||
"Not wrong by itself, but worth reading against current behavior.")
|
||||
|
||||
order = {"high": 0, "medium": 1, "low": 2}
|
||||
findings.sort(key=lambda f: (order.get(f["severity"], 3), f["doc"], f["line"] or 0))
|
||||
return {
|
||||
"repo": str(repo),
|
||||
"totals": {"docs_checked": len(docs), "findings": len(findings),
|
||||
"npm_scripts_found": len(all_npm), "make_targets_found": len(make_targets),
|
||||
"env_names_in_code": len(in_code), "env_names_documented": len(in_docs)},
|
||||
"findings": findings,
|
||||
"warnings": warnings,
|
||||
}
|
||||
|
||||
|
||||
def render(d: dict, top: int) -> str:
|
||||
t = d["totals"]
|
||||
L = ["# Documentation Drift Check", "", f"Repo: {d['repo']}",
|
||||
f"Docs checked: {t['docs_checked']} Findings: {t['findings']}",
|
||||
f"Known npm scripts: {t['npm_scripts_found']} make targets: {t['make_targets_found']}",
|
||||
f"Env names in code: {t['env_names_in_code']} documented: {t['env_names_documented']}", ""]
|
||||
|
||||
if not d["findings"]:
|
||||
L.append("No machine-verifiable drift found. Prose accuracy is still unchecked.")
|
||||
else:
|
||||
shown = d["findings"][:top]
|
||||
if len(d["findings"]) > top:
|
||||
L.append(f"TRUNCATED: showing {top} of {len(d['findings'])} findings")
|
||||
L.append("")
|
||||
for f in shown:
|
||||
loc = f"{f['doc']}:{f['line']}" if f.get("line") else f["doc"]
|
||||
L.append(f"- [{f['severity']}] {f['kind']} -- {loc}")
|
||||
L.append(f" {f['detail']}")
|
||||
if f.get("source"):
|
||||
L.append(f" source of truth: {f['source']}")
|
||||
L.append("")
|
||||
if d["warnings"]:
|
||||
L.append("## Warnings")
|
||||
L.extend(f"- {w}" for w in d["warnings"])
|
||||
L.append("")
|
||||
L.append("Checks only claims with a definite answer. Wording, completeness and whether an")
|
||||
L.append("explanation is actually correct are not checked here. Confirm each finding by")
|
||||
L.append("opening both the doc and the source before reporting it.")
|
||||
return "\n".join(L)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="Check documentation claims against the repo. Read-only.")
|
||||
ap.add_argument("--repo", default=".", help="Path inside the repository.")
|
||||
ap.add_argument("--format", choices=["text", "json"], default="text", help="Output format.")
|
||||
ap.add_argument("--top", type=int, default=30, help="Findings to show. Default 30.")
|
||||
ap.add_argument("--check-paths", action="store_true",
|
||||
help=("Also check backticked paths against the filesystem. Off by default: on "
|
||||
"real repos most such references are ambiguous -- a path a doc tells you "
|
||||
"to create, or one an archived report described at the time -- and the "
|
||||
"noise buries the unambiguous findings. Markdown links are always checked."))
|
||||
ap.add_argument("--no-git-root", action="store_true",
|
||||
help="Treat --repo literally instead of expanding to the enclosing git repository root.")
|
||||
args = ap.parse_args()
|
||||
|
||||
repo = Path(args.repo).resolve()
|
||||
if not repo.is_dir():
|
||||
print(f"error: not a directory: {repo}", file=sys.stderr)
|
||||
return 2
|
||||
if not args.no_git_root:
|
||||
root = run_git(["rev-parse", "--show-toplevel"], repo)
|
||||
if root and root.strip():
|
||||
repo = Path(root.strip()).resolve()
|
||||
|
||||
files = list_files(repo)
|
||||
if not files:
|
||||
print(f"error: no files found under {repo}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
data = build(repo, files, check_paths=args.check_paths)
|
||||
print(json.dumps(data, indent=2) if args.format == "json" else render(data, args.top))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,172 @@
|
||||
---
|
||||
name: test-gap-audit
|
||||
description: Run a read-only audit for missing, weak, stale, or mis-scoped test coverage. If the user does not name a scope, audit the full repository and identify important code paths, routes, features, services, workflows, and contracts that lack proper tests. If the user names a feature, PR, branch, route, workflow, service, bug fix, API, security-sensitive path, or risky code change, focus only on that specific scope. Use when the user asks what tests are missing, whether coverage is enough, what regression tests to add, or how to prove a change is safe. This is not a general bug audit and not a security review; it evaluates whether behavior is covered by tests.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Test Gap Audit
|
||||
|
||||
Find the tests that should exist but do not, or tests that exist but do not prove the important behavior. Produce concrete, prioritized test recommendations grounded in code paths, risk, and existing test conventions.
|
||||
|
||||
## Core Rules
|
||||
|
||||
- Stay read-only unless the user explicitly asks to add tests.
|
||||
- Default to a full-repository audit when the user does not provide a specific scope.
|
||||
- Full-repo audits are breadth-first, then depth-limited. Inventory the repo, rank surfaces by risk, deep-inspect as many high-risk surfaces as the turn allows, and list the rest under **Surveyed But Not Deeply Inspected** with a pointer to run another pass on them. State the surface counts in the report header. Never present a shallow sweep as complete coverage.
|
||||
- When the user names a route, feature, workflow, PR, branch, service, package, directory, or other portion of the repo, limit the audit to that scope and its directly connected code paths.
|
||||
- Focus on coverage quality and regression protection, not general bug hunting.
|
||||
- Ground every gap in a behavior, changed code path, risk, or existing weak test.
|
||||
- Prefer exact test cases over generic coverage advice.
|
||||
- Infer test style from the repository before recommending unit, integration, component, browser, contract, or end-to-end tests.
|
||||
- Separate confirmed missing coverage from inferred gaps.
|
||||
- Do not treat line/branch coverage percentage as sufficient proof. Behavior coverage matters more.
|
||||
- Avoid recommending slow end-to-end tests when a lower-level test would prove the behavior reliably.
|
||||
|
||||
## Inputs
|
||||
|
||||
When no scope is given, audit the whole repository. Inventory the repo's major testable surfaces and report which important areas do not have tests, do not have enough assertions, or are only indirectly covered.
|
||||
|
||||
Accept any specific testing scope, including:
|
||||
|
||||
- Pull requests or branches: `audit test gaps in this PR`, `what tests should this branch add`.
|
||||
- Features: `test gap audit uploads`, `what coverage is missing for billing`.
|
||||
- Routes/APIs: `review tests for POST /orders`, `check auth tests around exports`.
|
||||
- Workflows: `invite teammate -> accept invite -> set role -> revoke access`.
|
||||
- Bug fixes: `what regression test should cover this fix`.
|
||||
- Security or docs follow-up: `what tests prove the security audit fixes`, `do examples have tests`.
|
||||
|
||||
If scope is blurry, infer the smallest useful boundary and state it. If no scope is stated, do not ask for one; proceed with a full-repo audit. Ask only when different scopes would require materially different test plans.
|
||||
|
||||
## Discovery Workflow
|
||||
|
||||
1. Establish repo context.
|
||||
- Check `git status --short`.
|
||||
- Identify stack, test runners, package scripts, CI checks, test file naming, fixture style, mocks, factories, browser tools, API test conventions, and monorepo boundaries.
|
||||
- Read relevant manifests, CI workflows, test configs, and nearby tests.
|
||||
|
||||
2. Map the behavior under review.
|
||||
- For full-repo audits, inventory major app surfaces, packages, routes, APIs, services, jobs, CLIs, schemas, integrations, and shared libraries before choosing the highest-risk gaps to inspect deeply.
|
||||
- For PRs, inspect changed files, changed tests, and adjacent unchanged code.
|
||||
- For features, locate routes, components, services, models, schemas, jobs, permissions, integrations, and user-facing states.
|
||||
- Identify happy paths, failure paths, edge cases, data boundaries, auth/authorization boundaries, migration/config behavior, and external integration behavior.
|
||||
|
||||
3. Map existing coverage.
|
||||
- Run the bundled `scripts/coverage_map.py` first when it is available. It detects the test framework and naming convention, then matches every source file against the tests by name, mirrored path, and what the test files actually import, and returns the unmatched files ranked with risk keywords plus test files that have cases but almost no assertions. The path is relative to this skill's own directory, which varies by host. Use `python` if `python3` is not on PATH.
|
||||
- `python <skill-dir>/scripts/coverage_map.py --top 25`, or `--format json` to filter the results yourself.
|
||||
- The matcher is heuristic and cannot see coverage that arrives through fixtures, end-to-end tests, or indirection. Treat an unmatched file as a lead, and grep for the module name to confirm before reporting it as `P0` or `P1`. Report a gap as confirmed only after you have looked.
|
||||
- If the script is unavailable, compare production/source areas against test directories and test naming conventions manually to find untested or weakly tested portions of the repo.
|
||||
- Find direct tests for the changed or requested code.
|
||||
- Find indirect tests that cover the same behavior through a higher-level workflow.
|
||||
- Inspect assertions, fixtures, mocks, setup, and test names to see what is actually proven.
|
||||
- Note stale tests whose names or fixtures no longer match current behavior.
|
||||
|
||||
4. Identify gaps.
|
||||
- Entire routes, features, services, packages, commands, jobs, or integration boundaries with no tests.
|
||||
- Missing critical path tests.
|
||||
- Tests that only render or call code without meaningful assertions.
|
||||
- Tests that mock away the behavior they claim to cover.
|
||||
- Missing negative/error/permission tests.
|
||||
- Missing tenant/ownership/role boundary tests.
|
||||
- Missing validation, pagination, sorting, filtering, time zone, race/idempotency, retry, or empty-state tests.
|
||||
- Missing regression test for a fixed bug.
|
||||
- Missing contract tests for API/schema/client changes.
|
||||
- Missing docs/example tests when examples are part of the user contract.
|
||||
- Missing migration/backward-compatibility tests when data shape changes.
|
||||
|
||||
5. Verify safely.
|
||||
- Run focused test discovery or relevant existing tests when quick and repo-conventional.
|
||||
- Use test list commands, grep/search, typecheck, lint, or focused test files as appropriate.
|
||||
- Do not install dependencies, start long-running services, or run expensive full suites unless the user asks or the repo clearly expects it.
|
||||
- Never run a command that writes into the repository as a side effect. `python -m compileall` and `py_compile` emit `.pyc` files, formatters rewrite sources, and installers touch lockfiles. `.pyc` output is usually gitignored, so `git status` will look clean while the tree has in fact been modified. Prefer checks that write nothing, and if a language offers no read-only check, say so under checks skipped.
|
||||
- Record checks run and skipped.
|
||||
|
||||
## Severity Rubric
|
||||
|
||||
- `P0`: Missing tests for code that can cause data loss, security/privacy exposure, payment/billing errors, destructive actions, or production outage with no practical safety net.
|
||||
- `P1`: High-impact missing coverage for common user paths, auth/authorization, critical API contracts, migrations, background jobs, or release-blocking behavior.
|
||||
- `P2`: Meaningful regression risk around important edge cases, validation, error handling, state transitions, integrations, or stale/weak tests.
|
||||
- `P3`: Lower-risk test cleanup, naming drift, fixture improvement, redundant tests, or useful coverage polish.
|
||||
|
||||
## Evidence Standards
|
||||
|
||||
- Verify every citation before you write it. Re-read the exact range and confirm it contains what you are describing. When citing a named symbol, function, CTE, or block, cite the line where the name is defined, not a line inside a neighbouring block. When quoting text, cite the file the quote is actually in. Prefer a single anchor line containing a distinctive token over a hand-counted range.
|
||||
- When you attribute a finding to a tool's output, quote the path and line the tool itself reported. Never infer which lines a linter or type checker fired on by reading the code. If the tool's output does not name the line, report the pattern without claiming the tool flagged it.
|
||||
- Never restate a count from a grep, a script, or a tool without the raw output in front of you. If you cannot re-derive the number, describe the pattern instead of counting it.
|
||||
- Before reporting that something is absent -- undocumented config, an unused dependency, a missing control, a variable nothing reads -- check every plausible location, not the first one. For a config variable that means the README, env sample files, deploy manifests, comments, and the transitive callers of whatever helper reads it. For a dependency it means whether it is a documented transitive requirement of something you do use. A negative claim from a single grep is not evidence.
|
||||
- Cite the behavior or changed code and the existing/missing test area.
|
||||
- Include file and line references whenever possible.
|
||||
- Explain what current tests prove and what they do not prove.
|
||||
- For inferred gaps, include `Confidence: high/medium/low`.
|
||||
- Recommend the smallest reliable test level that proves the behavior.
|
||||
- Include suggested test names or scenarios precise enough for implementation.
|
||||
|
||||
## Report Format
|
||||
|
||||
Use this structure unless the user asks otherwise:
|
||||
|
||||
```markdown
|
||||
**Test Gap Audit: <scope>**
|
||||
|
||||
No code changed. I reviewed <brief scope>, existing tests, and repo test conventions. <verification summary>. No P0s found / P0s found: <count>.
|
||||
|
||||
1. **P1: <gap title>.**
|
||||
Gap: <behavior or risk not covered>.
|
||||
Current coverage: <what existing tests cover or why none were found>.
|
||||
Evidence: code `<path>:<line>`; tests `<path>:<line>` or "no direct tests found in <area>".
|
||||
Suggested test: <specific test level, file/location, scenario, and key assertions>.
|
||||
|
||||
2. **P2: <gap title>.**
|
||||
Gap: <missing or weak coverage>.
|
||||
Current coverage: <what is currently proven>.
|
||||
Evidence: code `<path>:<line>`; tests `<path>:<line>`.
|
||||
Confidence: <high/medium/low if inferred>.
|
||||
Suggested test: <specific recommendation>.
|
||||
|
||||
**Suggested Test Plan**
|
||||
- <ordered list of concrete tests to add first>
|
||||
|
||||
**Untested Or Weakly Tested Areas**
|
||||
- <for full-repo audits, list important routes/features/services/packages/workflows that lack proper tests, with brief evidence>
|
||||
|
||||
**Existing Coverage Worth Keeping**
|
||||
- <only include useful tests that already protect important behavior>
|
||||
|
||||
**Surveyed But Not Deeply Inspected**
|
||||
- <For full-repo audits only: surfaces that were inventoried but not inspected deeply this pass, and which to run next. Omit this section entirely for scoped audits.>
|
||||
|
||||
**Checks Run**
|
||||
- `<command>`: <result>
|
||||
|
||||
**Not Tested**
|
||||
- <test suites, services, browsers, credentials, or dependency gaps and why>
|
||||
|
||||
**Assumptions**
|
||||
- <only include if useful>
|
||||
```
|
||||
|
||||
If no meaningful gaps are found, say that clearly, name the strongest coverage observed, and list any residual risk.
|
||||
|
||||
## Post-Audit Test Implementation
|
||||
|
||||
When the user asks to add tests:
|
||||
|
||||
- Implement the highest-priority gaps first.
|
||||
- Follow existing test style, factories, mocks, helpers, naming, and file placement.
|
||||
- Prefer focused tests that prove behavior with clear assertions.
|
||||
- Avoid broad snapshot tests unless snapshots are already the right local convention.
|
||||
- Update fixtures, test data, or contract examples only when needed for the selected tests.
|
||||
- Run the new tests and the closest existing related tests.
|
||||
- Final response should map gaps to added tests and list checks run.
|
||||
|
||||
## Related Skills
|
||||
|
||||
This skill is one of seven review skills that share a single report contract:
|
||||
every finding carries a `P0`-`P3` severity and a `path:line` you can open. The
|
||||
other five cover launch readiness, security, repo structure, improvement ideas,
|
||||
and pull request communication. They are at https://github.com/specialone0007/review-skills.
|
||||
|
||||
## Agent Portability Notes
|
||||
|
||||
- Use available shell, search, git, browser, CI, coverage, or MCP tools as appropriate.
|
||||
- If test execution is unavailable, continue with source and test inspection and state the limitation.
|
||||
- If the host supports inline review comments, emit them only for confirmed actionable test gaps and keep ranges tight.
|
||||
@@ -0,0 +1,416 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Map a repository's source files against its tests, so coverage gaps are evidence, not guesses.
|
||||
|
||||
Read-only. Standard library only. Writes nothing. Runs no tests.
|
||||
|
||||
python coverage_map.py # text summary of the current repo
|
||||
python coverage_map.py --repo ../other # a different repo
|
||||
python coverage_map.py --format json # machine-readable
|
||||
python coverage_map.py --top 40 # more rows per section
|
||||
|
||||
Detects the test framework and naming convention, then matches each source file to
|
||||
tests by basename, mirrored path, and — most importantly — by scanning what the test
|
||||
files actually import. The import scan is what makes "no tests found for X" worth
|
||||
reporting instead of merely plausible.
|
||||
|
||||
HEURISTIC. It cannot see coverage through indirection, fixtures, or end-to-end tests
|
||||
that exercise a file without naming it. Treat an `untested` entry as a lead to confirm
|
||||
by hand, never as a finding on its own.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
if hasattr(sys.stdout, "reconfigure"):
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
GIT_TIMEOUT = 30
|
||||
MAX_READ_BYTES = 400_000 # skip pathological files rather than stalling
|
||||
|
||||
IGNORED_DIRS = {
|
||||
".git", ".hg", ".svn", "node_modules", "vendor", "venv", ".venv", "env",
|
||||
"__pycache__", ".pytest_cache", ".mypy_cache", ".ruff_cache", ".tox",
|
||||
"dist", "build", "target", "out", ".next", ".nuxt", ".svelte-kit",
|
||||
".idea", ".vscode", ".gradle", "Pods", ".terraform", "coverage",
|
||||
}
|
||||
|
||||
SOURCE_EXTS = {
|
||||
".py", ".js", ".jsx", ".ts", ".tsx", ".mjs", ".cjs", ".vue", ".svelte",
|
||||
".go", ".rs", ".rb", ".php", ".java", ".kt", ".swift", ".scala", ".cs", ".ex", ".exs", ".dart",
|
||||
}
|
||||
|
||||
# Files that are configuration, generated, or entry-point glue rather than logic worth testing.
|
||||
NON_LOGIC = re.compile(
|
||||
r"(^|/)(setup|conftest|__init__|index|main|migrations?|__generated__)\.[A-Za-z]+$"
|
||||
r"|\.(config|d)\.[A-Za-z]+$"
|
||||
r"|(^|/)(migrations|__generated__|generated|\.storybook)/",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
TEST_PATH = re.compile(
|
||||
r"(^|/)(tests?|spec|specs|__tests__|e2e|integration|cypress|playwright)(/|$)"
|
||||
r"|(^|/)[^/]*[._-](test|spec)s?\.[A-Za-z0-9]+$"
|
||||
r"|(^|/)test_[^/]*\.[A-Za-z0-9]+$"
|
||||
r"|_test\.[A-Za-z0-9]+$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Framework marker -> label. Searched in manifests and config filenames.
|
||||
FRAMEWORKS = {
|
||||
"vitest": "Vitest", "jest": "Jest", "mocha": "Mocha", "jasmine": "Jasmine",
|
||||
"@playwright/test": "Playwright", "cypress": "Cypress", "ava": "AVA",
|
||||
"@testing-library": "Testing Library", "karma": "Karma", "node:test": "node:test",
|
||||
"pytest": "pytest", "unittest": "unittest", "nose": "nose", "tox": "tox",
|
||||
"rspec": "RSpec", "minitest": "Minitest", "phpunit": "PHPUnit", "pest": "Pest",
|
||||
"junit": "JUnit", "testng": "TestNG", "go test": "go test", "cargo test": "cargo test",
|
||||
"xunit": "xUnit", "nunit": "NUnit", "exunit": "ExUnit",
|
||||
}
|
||||
|
||||
ASSERTION_TOKENS = re.compile(
|
||||
r"\b(expect|assert|assert_|assertEqual|assertTrue|assertRaises|should|"
|
||||
r"toBe|toEqual|toThrow|toHaveBeenCalled|require\.Equal|assert\.|"
|
||||
r"refute|is_a|must_equal|shouldBe|verify)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
# Actual test cases only. `describe`, `context` and `class Test` are grouping
|
||||
# constructs, and counting them inflates the case count -- a file with one it()
|
||||
# inside one describe() would report two cases, which then skews the
|
||||
# assertions-per-case ratio below.
|
||||
TEST_CASE_TOKENS = re.compile(
|
||||
r"^\s*(it|test|def test_|func Test|scenario|@Test|it\.each|test\.each)\b",
|
||||
re.MULTILINE,
|
||||
)
|
||||
|
||||
RISK_KEYWORDS = (
|
||||
"auth", "login", "session", "token", "password", "permission", "role", "admin",
|
||||
"billing", "payment", "invoice", "subscription", "checkout", "charge", "refund",
|
||||
"migration", "delete", "destroy", "export", "import", "webhook", "crypto", "wallet",
|
||||
"security", "secret", "upload",
|
||||
)
|
||||
|
||||
# Import forms across the languages we handle. Group 1 is always the module reference.
|
||||
IMPORT_RES = [
|
||||
re.compile(r"""(?:from|import)\s+['"]([^'"]+)['"]"""), # JS/TS
|
||||
re.compile(r"""require\(\s*['"]([^'"]+)['"]\s*\)"""), # CJS
|
||||
re.compile(r"""^\s*from\s+([A-Za-z0-9_.]+)\s+import""", re.M), # Python from-import
|
||||
re.compile(r"""^\s*import\s+([A-Za-z0-9_.]+)""", re.M), # Python/Java/Go import
|
||||
re.compile(r"""use\s+([A-Za-z0-9_:]+)"""), # Rust
|
||||
re.compile(r"""require(?:_relative)?\s+['"]([^'"]+)['"]"""), # Ruby
|
||||
]
|
||||
|
||||
warnings: list[str] = []
|
||||
|
||||
|
||||
def run_git(args: list[str], repo: Path) -> str | None:
|
||||
git = shutil.which("git")
|
||||
if git is None:
|
||||
return None
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[git, *args], cwd=str(repo), text=True, timeout=GIT_TIMEOUT,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
encoding="utf-8", errors="replace",
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError) as exc:
|
||||
warnings.append(f"git {' '.join(args)} failed: {exc}")
|
||||
return None
|
||||
return r.stdout if r.returncode == 0 else None
|
||||
|
||||
|
||||
def list_files(repo: Path) -> list[str]:
|
||||
out = run_git(["ls-files", "--cached", "--other", "--exclude-standard"], repo)
|
||||
if out is not None:
|
||||
files = [ln.strip() for ln in out.splitlines() if ln.strip()]
|
||||
if files:
|
||||
return sorted(files)
|
||||
warnings.append("git unavailable or empty index; using a filesystem walk (ignore rules approximated)")
|
||||
files = []
|
||||
for root, dirnames, filenames in os.walk(repo):
|
||||
dirnames[:] = [d for d in dirnames if d not in IGNORED_DIRS and not d.startswith(".")]
|
||||
for fn in filenames:
|
||||
files.append(Path(root, fn).relative_to(repo).as_posix())
|
||||
return sorted(files)
|
||||
|
||||
|
||||
def read(path: Path) -> str:
|
||||
try:
|
||||
if path.stat().st_size > MAX_READ_BYTES:
|
||||
return ""
|
||||
return path.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def detect_frameworks(repo: Path, files: list[str]) -> list[str]:
|
||||
found: set[str] = set()
|
||||
haystacks: list[str] = []
|
||||
for name in ("package.json", "pyproject.toml", "requirements.txt", "Gemfile",
|
||||
"composer.json", "pom.xml", "build.gradle", "mix.exs", "Cargo.toml"):
|
||||
for rel in [f for f in files if Path(f).name == name][:5]:
|
||||
haystacks.append(read(repo / rel).lower())
|
||||
config_names = " ".join(Path(f).name.lower() for f in files)
|
||||
haystacks.append(config_names)
|
||||
blob = "\n".join(haystacks)
|
||||
for marker, label in FRAMEWORKS.items():
|
||||
m = marker.lower()
|
||||
# Short bare words such as "ava", "nose" and "pest" substring-match inside
|
||||
# "available", "javascript" and so on, so require word boundaries for them.
|
||||
# Markers carrying punctuation ("@playwright/test", "node:test") are already
|
||||
# specific enough, and \b would not behave around those characters anyway.
|
||||
if m.isalnum():
|
||||
if re.search(rf"\b{re.escape(m)}\b", blob):
|
||||
found.add(label)
|
||||
elif m in blob:
|
||||
found.add(label)
|
||||
# Language-implied runners that need no manifest entry.
|
||||
if any(f.endswith("_test.go") for f in files):
|
||||
found.add("go test")
|
||||
if any(f.endswith(".rs") for f in files) and any("#[test]" in read(repo / f) for f in
|
||||
[x for x in files if x.endswith(".rs")][:20]):
|
||||
found.add("cargo test")
|
||||
return sorted(found)
|
||||
|
||||
|
||||
def infer_convention(test_files: list[str]) -> list[str]:
|
||||
patterns: Counter[str] = Counter()
|
||||
for rel in test_files:
|
||||
name = Path(rel).name
|
||||
if re.match(r"^test_.*\.py$", name):
|
||||
patterns["test_*.py"] += 1
|
||||
elif re.search(r"_test\.go$", name):
|
||||
patterns["*_test.go"] += 1
|
||||
elif re.search(r"\.(test|spec)\.[jt]sx?$", name):
|
||||
patterns[f"*.{'test' if '.test.' in name else 'spec'}.[jt]s(x)"] += 1
|
||||
elif re.search(r"_spec\.rb$", name):
|
||||
patterns["*_spec.rb"] += 1
|
||||
elif re.search(r"Test\.(java|kt|cs)$", name):
|
||||
patterns["*Test.{java,kt,cs}"] += 1
|
||||
if "__tests__/" in rel:
|
||||
patterns["__tests__/ directory"] += 1
|
||||
elif re.match(r"^tests?/", rel):
|
||||
patterns["tests/ directory"] += 1
|
||||
elif re.match(r"^spec/", rel):
|
||||
patterns["spec/ directory"] += 1
|
||||
return [f"{p} ({n})" for p, n in patterns.most_common()]
|
||||
|
||||
|
||||
def module_tokens(rel: str) -> set[str]:
|
||||
"""Identifiers by which a test might refer to this source file."""
|
||||
p = Path(rel)
|
||||
stem = p.stem
|
||||
tokens = {stem.lower()}
|
||||
# A component at foo/Button/index.tsx is referred to as "Button".
|
||||
if stem.lower() in ("index", "__init__", "mod"):
|
||||
tokens.add(p.parent.name.lower())
|
||||
tokens.add(rel.lower())
|
||||
tokens.add(p.with_suffix("").as_posix().lower())
|
||||
return {t for t in tokens if t and t not in ("", ".")}
|
||||
|
||||
|
||||
def build(repo: Path, files: list[str]) -> dict:
|
||||
test_files = [f for f in files if TEST_PATH.search(f) and Path(f).suffix in SOURCE_EXTS]
|
||||
test_set = set(test_files)
|
||||
source_files = [
|
||||
f for f in files
|
||||
if Path(f).suffix in SOURCE_EXTS and f not in test_set and not NON_LOGIC.search(f)
|
||||
]
|
||||
|
||||
# Index what the tests import, plus every bare identifier they mention.
|
||||
imported: set[str] = set()
|
||||
mentioned: set[str] = set()
|
||||
weak: list[dict] = []
|
||||
for rel in test_files:
|
||||
text = read(repo / rel)
|
||||
if not text:
|
||||
continue
|
||||
for regex in IMPORT_RES:
|
||||
for m in regex.findall(text):
|
||||
ref = m.strip()
|
||||
imported.add(ref.lower())
|
||||
imported.add(Path(ref).name.lower())
|
||||
imported.add(Path(ref).stem.lower())
|
||||
for part in re.split(r"[./:\\]", ref):
|
||||
if part and part not in (".", "..", "src", "lib", "app"):
|
||||
mentioned.add(part.lower())
|
||||
assertions = len(ASSERTION_TOKENS.findall(text))
|
||||
cases = len(TEST_CASE_TOKENS.findall(text))
|
||||
lines = text.count("\n") + 1
|
||||
# A test file with cases but almost no assertions is usually asserting nothing useful.
|
||||
if cases and assertions <= max(1, cases // 4):
|
||||
weak.append({"path": rel, "test_cases": cases, "assertions": assertions, "lines": lines})
|
||||
|
||||
# Test files whose filename itself marks them as a test, as opposed to files that
|
||||
# merely live under tests/ or e2e/ (fixtures, helpers, page objects, factories).
|
||||
named_test_re = re.compile(r"([._-](test|spec)s?$)|(^test_)|(^test$)", re.IGNORECASE)
|
||||
named_tests = [t for t in test_files if named_test_re.search(Path(t).stem)]
|
||||
|
||||
matched: dict[str, list[str]] = {}
|
||||
untested: list[dict] = []
|
||||
for rel in source_files:
|
||||
tokens = module_tokens(rel)
|
||||
how: list[str] = []
|
||||
|
||||
stem = Path(rel).stem.lower()
|
||||
mirror = Path(rel).with_suffix("").as_posix().lower()
|
||||
|
||||
# 1. a test file whose *own name* carries a test marker and embeds this file's name.
|
||||
# Requiring the marker matters: a helper like e2e/lib/api.mjs is classified as a
|
||||
# test file because of its directory, and would otherwise "cover" every api.js
|
||||
# in the repo. A false match here hides a real gap, so keep this rule strict.
|
||||
for t in named_tests:
|
||||
tl = Path(t).stem.lower()
|
||||
if tl in {f"{stem}test", f"test{stem}", f"{stem}spec", f"{stem}_test",
|
||||
f"test_{stem}", f"{stem}.test", f"{stem}.spec"} or \
|
||||
re.sub(r"[._-]?(test|spec)s?$", "", tl) == stem:
|
||||
how.append(f"name match: {t}")
|
||||
break
|
||||
# 2. mirrored directory layout, e.g. src/a/b.ts -> tests/a/b.test.ts
|
||||
if not how:
|
||||
tail = "/".join(mirror.split("/")[1:]) if "/" in mirror else mirror
|
||||
if tail and any(tail in t.lower() for t in test_files):
|
||||
how.append("mirrored path match")
|
||||
# 3. a test actually imports it
|
||||
if not how and (tokens & imported):
|
||||
how.append("imported by a test")
|
||||
# 4. weakest signal: a test mentions the identifier
|
||||
if not how and stem in mentioned and len(stem) > 3:
|
||||
how.append("mentioned in a test (weak signal)")
|
||||
|
||||
if how:
|
||||
matched[rel] = how
|
||||
else:
|
||||
lines = read(repo / rel).count("\n") + 1
|
||||
hits = [k for k in RISK_KEYWORDS if k in rel.lower()]
|
||||
untested.append({
|
||||
"path": rel,
|
||||
"lines": lines,
|
||||
"dir": str(Path(rel).parent).replace("\\", "/"),
|
||||
"risk_keywords": hits,
|
||||
})
|
||||
|
||||
untested.sort(key=lambda x: (not x["risk_keywords"], -x["lines"]))
|
||||
weak.sort(key=lambda x: -x["test_cases"])
|
||||
|
||||
by_dir: dict[str, dict] = defaultdict(lambda: {"untested": 0, "lines": 0})
|
||||
for u in untested:
|
||||
by_dir[u["dir"]]["untested"] += 1
|
||||
by_dir[u["dir"]]["lines"] += u["lines"]
|
||||
|
||||
return {
|
||||
"repo": str(repo),
|
||||
"frameworks": detect_frameworks(repo, files),
|
||||
"conventions": infer_convention(test_files),
|
||||
"totals": {
|
||||
"source_files": len(source_files),
|
||||
"test_files": len(test_files),
|
||||
"matched": len(matched),
|
||||
"untested": len(untested),
|
||||
"coverage_ratio": round(len(matched) / len(source_files), 3) if source_files else None,
|
||||
},
|
||||
"untested": untested,
|
||||
"untested_by_directory": sorted(
|
||||
({"dir": d, **v} for d, v in by_dir.items()),
|
||||
key=lambda x: -x["untested"],
|
||||
),
|
||||
"weak_tests": weak,
|
||||
"matched_sample": [{"path": p, "why": w} for p, w in list(matched.items())[:15]],
|
||||
"warnings": warnings,
|
||||
}
|
||||
|
||||
|
||||
def truncate(items: list, limit: int, label: str, out: list[str]) -> list:
|
||||
if len(items) > limit:
|
||||
out.append(f" TRUNCATED: showing top {limit} of {len(items)} {label}")
|
||||
return items[:limit]
|
||||
return items
|
||||
|
||||
|
||||
def render(d: dict, top: int) -> str:
|
||||
t = d["totals"]
|
||||
L = ["# Test Coverage Map", "", f"Repo: {d['repo']}"]
|
||||
ratio = f"{t['coverage_ratio']:.0%}" if t["coverage_ratio"] is not None else "n/a"
|
||||
L.append(f"Source files: {t['source_files']} Test files: {t['test_files']} "
|
||||
f"Matched: {t['matched']} ({ratio}) Unmatched: {t['untested']}")
|
||||
L.append("")
|
||||
L.append(f"Frameworks detected: {', '.join(d['frameworks']) or 'none detected'}")
|
||||
L.append(f"Naming conventions: {', '.join(d['conventions']) or 'none inferred'}")
|
||||
L.append("")
|
||||
|
||||
L.append("## Unmatched source files (risk-flagged first, then largest)")
|
||||
if d["untested"]:
|
||||
for u in truncate(d["untested"], top, "unmatched files", L):
|
||||
flag = f" [risk: {', '.join(u['risk_keywords'])}]" if u["risk_keywords"] else ""
|
||||
L.append(f"- {u['path']}: {u['lines']} lines{flag}")
|
||||
else:
|
||||
L.append("- none; every source file matched at least one test signal")
|
||||
L.append("")
|
||||
|
||||
L.append("## Unmatched by directory")
|
||||
for row in truncate(d["untested_by_directory"], top, "directories", L):
|
||||
L.append(f"- {row['dir']}: {row['untested']} files, {row['lines']} lines")
|
||||
L.append("")
|
||||
|
||||
L.append("## Test files with test cases but few assertions")
|
||||
if d["weak_tests"]:
|
||||
for w in truncate(d["weak_tests"], top, "test files", L):
|
||||
L.append(f"- {w['path']}: {w['test_cases']} cases, {w['assertions']} assertions")
|
||||
else:
|
||||
L.append("- none flagged")
|
||||
L.append("")
|
||||
|
||||
if d["warnings"]:
|
||||
L.append("## Warnings")
|
||||
L.extend(f"- {w}" for w in d["warnings"])
|
||||
L.append("")
|
||||
|
||||
L.append("HEURISTIC MATCHER. An unmatched file is a lead, not a finding: coverage through")
|
||||
L.append("fixtures, end-to-end tests, or indirection is invisible here. Confirm by grepping for")
|
||||
L.append("the module name before reporting anything as P0 or P1.")
|
||||
return "\n".join(L)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="Map source files against tests. Read-only.")
|
||||
ap.add_argument("--repo", default=".", help="Path inside the repository.")
|
||||
ap.add_argument("--format", choices=["text", "json"], default="text", help="Output format.")
|
||||
ap.add_argument("--top", type=int, default=25, help="Rows per section. Default 25.")
|
||||
ap.add_argument(
|
||||
"--no-git-root",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Treat --repo literally instead of expanding to the enclosing git repository root. "
|
||||
"Use this to scope the survey to one package or subdirectory of a monorepo."
|
||||
),
|
||||
)
|
||||
args = ap.parse_args()
|
||||
|
||||
repo = Path(args.repo).resolve()
|
||||
if not repo.is_dir():
|
||||
print(f"error: not a directory: {repo}", file=sys.stderr)
|
||||
return 2
|
||||
if not args.no_git_root:
|
||||
root = run_git(["rev-parse", "--show-toplevel"], repo)
|
||||
if root and root.strip():
|
||||
repo = Path(root.strip()).resolve()
|
||||
|
||||
files = list_files(repo)
|
||||
if not files:
|
||||
print(f"error: no files found under {repo}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
data = build(repo, files)
|
||||
print(json.dumps(data, indent=2) if args.format == "json" else render(data, args.top))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user