mirror of
https://github.com/github/awesome-copilot.git
synced 2026-09-07 08:15:58 +00:00
chore: publish from main
This commit is contained in:
github-actions[bot]
@@ -0,0 +1,172 @@
|
||||
---
|
||||
name: test-gap-audit
|
||||
description: Run a read-only audit for missing, weak, stale, or mis-scoped test coverage. If the user does not name a scope, audit the full repository and identify important code paths, routes, features, services, workflows, and contracts that lack proper tests. If the user names a feature, PR, branch, route, workflow, service, bug fix, API, security-sensitive path, or risky code change, focus only on that specific scope. Use when the user asks what tests are missing, whether coverage is enough, what regression tests to add, or how to prove a change is safe. This is not a general bug audit and not a security review; it evaluates whether behavior is covered by tests.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Test Gap Audit
|
||||
|
||||
Find the tests that should exist but do not, or tests that exist but do not prove the important behavior. Produce concrete, prioritized test recommendations grounded in code paths, risk, and existing test conventions.
|
||||
|
||||
## Core Rules
|
||||
|
||||
- Stay read-only unless the user explicitly asks to add tests.
|
||||
- Default to a full-repository audit when the user does not provide a specific scope.
|
||||
- Full-repo audits are breadth-first, then depth-limited. Inventory the repo, rank surfaces by risk, deep-inspect as many high-risk surfaces as the turn allows, and list the rest under **Surveyed But Not Deeply Inspected** with a pointer to run another pass on them. State the surface counts in the report header. Never present a shallow sweep as complete coverage.
|
||||
- When the user names a route, feature, workflow, PR, branch, service, package, directory, or other portion of the repo, limit the audit to that scope and its directly connected code paths.
|
||||
- Focus on coverage quality and regression protection, not general bug hunting.
|
||||
- Ground every gap in a behavior, changed code path, risk, or existing weak test.
|
||||
- Prefer exact test cases over generic coverage advice.
|
||||
- Infer test style from the repository before recommending unit, integration, component, browser, contract, or end-to-end tests.
|
||||
- Separate confirmed missing coverage from inferred gaps.
|
||||
- Do not treat line/branch coverage percentage as sufficient proof. Behavior coverage matters more.
|
||||
- Avoid recommending slow end-to-end tests when a lower-level test would prove the behavior reliably.
|
||||
|
||||
## Inputs
|
||||
|
||||
When no scope is given, audit the whole repository. Inventory the repo's major testable surfaces and report which important areas do not have tests, do not have enough assertions, or are only indirectly covered.
|
||||
|
||||
Accept any specific testing scope, including:
|
||||
|
||||
- Pull requests or branches: `audit test gaps in this PR`, `what tests should this branch add`.
|
||||
- Features: `test gap audit uploads`, `what coverage is missing for billing`.
|
||||
- Routes/APIs: `review tests for POST /orders`, `check auth tests around exports`.
|
||||
- Workflows: `invite teammate -> accept invite -> set role -> revoke access`.
|
||||
- Bug fixes: `what regression test should cover this fix`.
|
||||
- Security or docs follow-up: `what tests prove the security audit fixes`, `do examples have tests`.
|
||||
|
||||
If scope is blurry, infer the smallest useful boundary and state it. If no scope is stated, do not ask for one; proceed with a full-repo audit. Ask only when different scopes would require materially different test plans.
|
||||
|
||||
## Discovery Workflow
|
||||
|
||||
1. Establish repo context.
|
||||
- Check `git status --short`.
|
||||
- Identify stack, test runners, package scripts, CI checks, test file naming, fixture style, mocks, factories, browser tools, API test conventions, and monorepo boundaries.
|
||||
- Read relevant manifests, CI workflows, test configs, and nearby tests.
|
||||
|
||||
2. Map the behavior under review.
|
||||
- For full-repo audits, inventory major app surfaces, packages, routes, APIs, services, jobs, CLIs, schemas, integrations, and shared libraries before choosing the highest-risk gaps to inspect deeply.
|
||||
- For PRs, inspect changed files, changed tests, and adjacent unchanged code.
|
||||
- For features, locate routes, components, services, models, schemas, jobs, permissions, integrations, and user-facing states.
|
||||
- Identify happy paths, failure paths, edge cases, data boundaries, auth/authorization boundaries, migration/config behavior, and external integration behavior.
|
||||
|
||||
3. Map existing coverage.
|
||||
- Run the bundled `scripts/coverage_map.py` first when it is available. It detects the test framework and naming convention, then matches every source file against the tests by name, mirrored path, and what the test files actually import, and returns the unmatched files ranked with risk keywords plus test files that have cases but almost no assertions. The path is relative to this skill's own directory, which varies by host. Use `python` if `python3` is not on PATH.
|
||||
- `python <skill-dir>/scripts/coverage_map.py --top 25`, or `--format json` to filter the results yourself.
|
||||
- The matcher is heuristic and cannot see coverage that arrives through fixtures, end-to-end tests, or indirection. Treat an unmatched file as a lead, and grep for the module name to confirm before reporting it as `P0` or `P1`. Report a gap as confirmed only after you have looked.
|
||||
- If the script is unavailable, compare production/source areas against test directories and test naming conventions manually to find untested or weakly tested portions of the repo.
|
||||
- Find direct tests for the changed or requested code.
|
||||
- Find indirect tests that cover the same behavior through a higher-level workflow.
|
||||
- Inspect assertions, fixtures, mocks, setup, and test names to see what is actually proven.
|
||||
- Note stale tests whose names or fixtures no longer match current behavior.
|
||||
|
||||
4. Identify gaps.
|
||||
- Entire routes, features, services, packages, commands, jobs, or integration boundaries with no tests.
|
||||
- Missing critical path tests.
|
||||
- Tests that only render or call code without meaningful assertions.
|
||||
- Tests that mock away the behavior they claim to cover.
|
||||
- Missing negative/error/permission tests.
|
||||
- Missing tenant/ownership/role boundary tests.
|
||||
- Missing validation, pagination, sorting, filtering, time zone, race/idempotency, retry, or empty-state tests.
|
||||
- Missing regression test for a fixed bug.
|
||||
- Missing contract tests for API/schema/client changes.
|
||||
- Missing docs/example tests when examples are part of the user contract.
|
||||
- Missing migration/backward-compatibility tests when data shape changes.
|
||||
|
||||
5. Verify safely.
|
||||
- Run focused test discovery or relevant existing tests when quick and repo-conventional.
|
||||
- Use test list commands, grep/search, typecheck, lint, or focused test files as appropriate.
|
||||
- Do not install dependencies, start long-running services, or run expensive full suites unless the user asks or the repo clearly expects it.
|
||||
- Never run a command that writes into the repository as a side effect. `python -m compileall` and `py_compile` emit `.pyc` files, formatters rewrite sources, and installers touch lockfiles. `.pyc` output is usually gitignored, so `git status` will look clean while the tree has in fact been modified. Prefer checks that write nothing, and if a language offers no read-only check, say so under checks skipped.
|
||||
- Record checks run and skipped.
|
||||
|
||||
## Severity Rubric
|
||||
|
||||
- `P0`: Missing tests for code that can cause data loss, security/privacy exposure, payment/billing errors, destructive actions, or production outage with no practical safety net.
|
||||
- `P1`: High-impact missing coverage for common user paths, auth/authorization, critical API contracts, migrations, background jobs, or release-blocking behavior.
|
||||
- `P2`: Meaningful regression risk around important edge cases, validation, error handling, state transitions, integrations, or stale/weak tests.
|
||||
- `P3`: Lower-risk test cleanup, naming drift, fixture improvement, redundant tests, or useful coverage polish.
|
||||
|
||||
## Evidence Standards
|
||||
|
||||
- Verify every citation before you write it. Re-read the exact range and confirm it contains what you are describing. When citing a named symbol, function, CTE, or block, cite the line where the name is defined, not a line inside a neighbouring block. When quoting text, cite the file the quote is actually in. Prefer a single anchor line containing a distinctive token over a hand-counted range.
|
||||
- When you attribute a finding to a tool's output, quote the path and line the tool itself reported. Never infer which lines a linter or type checker fired on by reading the code. If the tool's output does not name the line, report the pattern without claiming the tool flagged it.
|
||||
- Never restate a count from a grep, a script, or a tool without the raw output in front of you. If you cannot re-derive the number, describe the pattern instead of counting it.
|
||||
- Before reporting that something is absent -- undocumented config, an unused dependency, a missing control, a variable nothing reads -- check every plausible location, not the first one. For a config variable that means the README, env sample files, deploy manifests, comments, and the transitive callers of whatever helper reads it. For a dependency it means whether it is a documented transitive requirement of something you do use. A negative claim from a single grep is not evidence.
|
||||
- Cite the behavior or changed code and the existing/missing test area.
|
||||
- Include file and line references whenever possible.
|
||||
- Explain what current tests prove and what they do not prove.
|
||||
- For inferred gaps, include `Confidence: high/medium/low`.
|
||||
- Recommend the smallest reliable test level that proves the behavior.
|
||||
- Include suggested test names or scenarios precise enough for implementation.
|
||||
|
||||
## Report Format
|
||||
|
||||
Use this structure unless the user asks otherwise:
|
||||
|
||||
```markdown
|
||||
**Test Gap Audit: <scope>**
|
||||
|
||||
No code changed. I reviewed <brief scope>, existing tests, and repo test conventions. <verification summary>. No P0s found / P0s found: <count>.
|
||||
|
||||
1. **P1: <gap title>.**
|
||||
Gap: <behavior or risk not covered>.
|
||||
Current coverage: <what existing tests cover or why none were found>.
|
||||
Evidence: code `<path>:<line>`; tests `<path>:<line>` or "no direct tests found in <area>".
|
||||
Suggested test: <specific test level, file/location, scenario, and key assertions>.
|
||||
|
||||
2. **P2: <gap title>.**
|
||||
Gap: <missing or weak coverage>.
|
||||
Current coverage: <what is currently proven>.
|
||||
Evidence: code `<path>:<line>`; tests `<path>:<line>`.
|
||||
Confidence: <high/medium/low if inferred>.
|
||||
Suggested test: <specific recommendation>.
|
||||
|
||||
**Suggested Test Plan**
|
||||
- <ordered list of concrete tests to add first>
|
||||
|
||||
**Untested Or Weakly Tested Areas**
|
||||
- <for full-repo audits, list important routes/features/services/packages/workflows that lack proper tests, with brief evidence>
|
||||
|
||||
**Existing Coverage Worth Keeping**
|
||||
- <only include useful tests that already protect important behavior>
|
||||
|
||||
**Surveyed But Not Deeply Inspected**
|
||||
- <For full-repo audits only: surfaces that were inventoried but not inspected deeply this pass, and which to run next. Omit this section entirely for scoped audits.>
|
||||
|
||||
**Checks Run**
|
||||
- `<command>`: <result>
|
||||
|
||||
**Not Tested**
|
||||
- <test suites, services, browsers, credentials, or dependency gaps and why>
|
||||
|
||||
**Assumptions**
|
||||
- <only include if useful>
|
||||
```
|
||||
|
||||
If no meaningful gaps are found, say that clearly, name the strongest coverage observed, and list any residual risk.
|
||||
|
||||
## Post-Audit Test Implementation
|
||||
|
||||
When the user asks to add tests:
|
||||
|
||||
- Implement the highest-priority gaps first.
|
||||
- Follow existing test style, factories, mocks, helpers, naming, and file placement.
|
||||
- Prefer focused tests that prove behavior with clear assertions.
|
||||
- Avoid broad snapshot tests unless snapshots are already the right local convention.
|
||||
- Update fixtures, test data, or contract examples only when needed for the selected tests.
|
||||
- Run the new tests and the closest existing related tests.
|
||||
- Final response should map gaps to added tests and list checks run.
|
||||
|
||||
## Related Skills
|
||||
|
||||
This skill is one of seven review skills that share a single report contract:
|
||||
every finding carries a `P0`-`P3` severity and a `path:line` you can open. The
|
||||
other five cover launch readiness, security, repo structure, improvement ideas,
|
||||
and pull request communication. They are at https://github.com/specialone0007/review-skills.
|
||||
|
||||
## Agent Portability Notes
|
||||
|
||||
- Use available shell, search, git, browser, CI, coverage, or MCP tools as appropriate.
|
||||
- If test execution is unavailable, continue with source and test inspection and state the limitation.
|
||||
- If the host supports inline review comments, emit them only for confirmed actionable test gaps and keep ranges tight.
|
||||
@@ -0,0 +1,416 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Map a repository's source files against its tests, so coverage gaps are evidence, not guesses.
|
||||
|
||||
Read-only. Standard library only. Writes nothing. Runs no tests.
|
||||
|
||||
python coverage_map.py # text summary of the current repo
|
||||
python coverage_map.py --repo ../other # a different repo
|
||||
python coverage_map.py --format json # machine-readable
|
||||
python coverage_map.py --top 40 # more rows per section
|
||||
|
||||
Detects the test framework and naming convention, then matches each source file to
|
||||
tests by basename, mirrored path, and — most importantly — by scanning what the test
|
||||
files actually import. The import scan is what makes "no tests found for X" worth
|
||||
reporting instead of merely plausible.
|
||||
|
||||
HEURISTIC. It cannot see coverage through indirection, fixtures, or end-to-end tests
|
||||
that exercise a file without naming it. Treat an `untested` entry as a lead to confirm
|
||||
by hand, never as a finding on its own.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
if hasattr(sys.stdout, "reconfigure"):
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
GIT_TIMEOUT = 30
|
||||
MAX_READ_BYTES = 400_000 # skip pathological files rather than stalling
|
||||
|
||||
IGNORED_DIRS = {
|
||||
".git", ".hg", ".svn", "node_modules", "vendor", "venv", ".venv", "env",
|
||||
"__pycache__", ".pytest_cache", ".mypy_cache", ".ruff_cache", ".tox",
|
||||
"dist", "build", "target", "out", ".next", ".nuxt", ".svelte-kit",
|
||||
".idea", ".vscode", ".gradle", "Pods", ".terraform", "coverage",
|
||||
}
|
||||
|
||||
SOURCE_EXTS = {
|
||||
".py", ".js", ".jsx", ".ts", ".tsx", ".mjs", ".cjs", ".vue", ".svelte",
|
||||
".go", ".rs", ".rb", ".php", ".java", ".kt", ".swift", ".scala", ".cs", ".ex", ".exs", ".dart",
|
||||
}
|
||||
|
||||
# Files that are configuration, generated, or entry-point glue rather than logic worth testing.
|
||||
NON_LOGIC = re.compile(
|
||||
r"(^|/)(setup|conftest|__init__|index|main|migrations?|__generated__)\.[A-Za-z]+$"
|
||||
r"|\.(config|d)\.[A-Za-z]+$"
|
||||
r"|(^|/)(migrations|__generated__|generated|\.storybook)/",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
TEST_PATH = re.compile(
|
||||
r"(^|/)(tests?|spec|specs|__tests__|e2e|integration|cypress|playwright)(/|$)"
|
||||
r"|(^|/)[^/]*[._-](test|spec)s?\.[A-Za-z0-9]+$"
|
||||
r"|(^|/)test_[^/]*\.[A-Za-z0-9]+$"
|
||||
r"|_test\.[A-Za-z0-9]+$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Framework marker -> label. Searched in manifests and config filenames.
|
||||
FRAMEWORKS = {
|
||||
"vitest": "Vitest", "jest": "Jest", "mocha": "Mocha", "jasmine": "Jasmine",
|
||||
"@playwright/test": "Playwright", "cypress": "Cypress", "ava": "AVA",
|
||||
"@testing-library": "Testing Library", "karma": "Karma", "node:test": "node:test",
|
||||
"pytest": "pytest", "unittest": "unittest", "nose": "nose", "tox": "tox",
|
||||
"rspec": "RSpec", "minitest": "Minitest", "phpunit": "PHPUnit", "pest": "Pest",
|
||||
"junit": "JUnit", "testng": "TestNG", "go test": "go test", "cargo test": "cargo test",
|
||||
"xunit": "xUnit", "nunit": "NUnit", "exunit": "ExUnit",
|
||||
}
|
||||
|
||||
ASSERTION_TOKENS = re.compile(
|
||||
r"\b(expect|assert|assert_|assertEqual|assertTrue|assertRaises|should|"
|
||||
r"toBe|toEqual|toThrow|toHaveBeenCalled|require\.Equal|assert\.|"
|
||||
r"refute|is_a|must_equal|shouldBe|verify)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
# Actual test cases only. `describe`, `context` and `class Test` are grouping
|
||||
# constructs, and counting them inflates the case count -- a file with one it()
|
||||
# inside one describe() would report two cases, which then skews the
|
||||
# assertions-per-case ratio below.
|
||||
TEST_CASE_TOKENS = re.compile(
|
||||
r"^\s*(it|test|def test_|func Test|scenario|@Test|it\.each|test\.each)\b",
|
||||
re.MULTILINE,
|
||||
)
|
||||
|
||||
RISK_KEYWORDS = (
|
||||
"auth", "login", "session", "token", "password", "permission", "role", "admin",
|
||||
"billing", "payment", "invoice", "subscription", "checkout", "charge", "refund",
|
||||
"migration", "delete", "destroy", "export", "import", "webhook", "crypto", "wallet",
|
||||
"security", "secret", "upload",
|
||||
)
|
||||
|
||||
# Import forms across the languages we handle. Group 1 is always the module reference.
|
||||
IMPORT_RES = [
|
||||
re.compile(r"""(?:from|import)\s+['"]([^'"]+)['"]"""), # JS/TS
|
||||
re.compile(r"""require\(\s*['"]([^'"]+)['"]\s*\)"""), # CJS
|
||||
re.compile(r"""^\s*from\s+([A-Za-z0-9_.]+)\s+import""", re.M), # Python from-import
|
||||
re.compile(r"""^\s*import\s+([A-Za-z0-9_.]+)""", re.M), # Python/Java/Go import
|
||||
re.compile(r"""use\s+([A-Za-z0-9_:]+)"""), # Rust
|
||||
re.compile(r"""require(?:_relative)?\s+['"]([^'"]+)['"]"""), # Ruby
|
||||
]
|
||||
|
||||
warnings: list[str] = []
|
||||
|
||||
|
||||
def run_git(args: list[str], repo: Path) -> str | None:
|
||||
git = shutil.which("git")
|
||||
if git is None:
|
||||
return None
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[git, *args], cwd=str(repo), text=True, timeout=GIT_TIMEOUT,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
encoding="utf-8", errors="replace",
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError) as exc:
|
||||
warnings.append(f"git {' '.join(args)} failed: {exc}")
|
||||
return None
|
||||
return r.stdout if r.returncode == 0 else None
|
||||
|
||||
|
||||
def list_files(repo: Path) -> list[str]:
|
||||
out = run_git(["ls-files", "--cached", "--other", "--exclude-standard"], repo)
|
||||
if out is not None:
|
||||
files = [ln.strip() for ln in out.splitlines() if ln.strip()]
|
||||
if files:
|
||||
return sorted(files)
|
||||
warnings.append("git unavailable or empty index; using a filesystem walk (ignore rules approximated)")
|
||||
files = []
|
||||
for root, dirnames, filenames in os.walk(repo):
|
||||
dirnames[:] = [d for d in dirnames if d not in IGNORED_DIRS and not d.startswith(".")]
|
||||
for fn in filenames:
|
||||
files.append(Path(root, fn).relative_to(repo).as_posix())
|
||||
return sorted(files)
|
||||
|
||||
|
||||
def read(path: Path) -> str:
|
||||
try:
|
||||
if path.stat().st_size > MAX_READ_BYTES:
|
||||
return ""
|
||||
return path.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def detect_frameworks(repo: Path, files: list[str]) -> list[str]:
|
||||
found: set[str] = set()
|
||||
haystacks: list[str] = []
|
||||
for name in ("package.json", "pyproject.toml", "requirements.txt", "Gemfile",
|
||||
"composer.json", "pom.xml", "build.gradle", "mix.exs", "Cargo.toml"):
|
||||
for rel in [f for f in files if Path(f).name == name][:5]:
|
||||
haystacks.append(read(repo / rel).lower())
|
||||
config_names = " ".join(Path(f).name.lower() for f in files)
|
||||
haystacks.append(config_names)
|
||||
blob = "\n".join(haystacks)
|
||||
for marker, label in FRAMEWORKS.items():
|
||||
m = marker.lower()
|
||||
# Short bare words such as "ava", "nose" and "pest" substring-match inside
|
||||
# "available", "javascript" and so on, so require word boundaries for them.
|
||||
# Markers carrying punctuation ("@playwright/test", "node:test") are already
|
||||
# specific enough, and \b would not behave around those characters anyway.
|
||||
if m.isalnum():
|
||||
if re.search(rf"\b{re.escape(m)}\b", blob):
|
||||
found.add(label)
|
||||
elif m in blob:
|
||||
found.add(label)
|
||||
# Language-implied runners that need no manifest entry.
|
||||
if any(f.endswith("_test.go") for f in files):
|
||||
found.add("go test")
|
||||
if any(f.endswith(".rs") for f in files) and any("#[test]" in read(repo / f) for f in
|
||||
[x for x in files if x.endswith(".rs")][:20]):
|
||||
found.add("cargo test")
|
||||
return sorted(found)
|
||||
|
||||
|
||||
def infer_convention(test_files: list[str]) -> list[str]:
|
||||
patterns: Counter[str] = Counter()
|
||||
for rel in test_files:
|
||||
name = Path(rel).name
|
||||
if re.match(r"^test_.*\.py$", name):
|
||||
patterns["test_*.py"] += 1
|
||||
elif re.search(r"_test\.go$", name):
|
||||
patterns["*_test.go"] += 1
|
||||
elif re.search(r"\.(test|spec)\.[jt]sx?$", name):
|
||||
patterns[f"*.{'test' if '.test.' in name else 'spec'}.[jt]s(x)"] += 1
|
||||
elif re.search(r"_spec\.rb$", name):
|
||||
patterns["*_spec.rb"] += 1
|
||||
elif re.search(r"Test\.(java|kt|cs)$", name):
|
||||
patterns["*Test.{java,kt,cs}"] += 1
|
||||
if "__tests__/" in rel:
|
||||
patterns["__tests__/ directory"] += 1
|
||||
elif re.match(r"^tests?/", rel):
|
||||
patterns["tests/ directory"] += 1
|
||||
elif re.match(r"^spec/", rel):
|
||||
patterns["spec/ directory"] += 1
|
||||
return [f"{p} ({n})" for p, n in patterns.most_common()]
|
||||
|
||||
|
||||
def module_tokens(rel: str) -> set[str]:
|
||||
"""Identifiers by which a test might refer to this source file."""
|
||||
p = Path(rel)
|
||||
stem = p.stem
|
||||
tokens = {stem.lower()}
|
||||
# A component at foo/Button/index.tsx is referred to as "Button".
|
||||
if stem.lower() in ("index", "__init__", "mod"):
|
||||
tokens.add(p.parent.name.lower())
|
||||
tokens.add(rel.lower())
|
||||
tokens.add(p.with_suffix("").as_posix().lower())
|
||||
return {t for t in tokens if t and t not in ("", ".")}
|
||||
|
||||
|
||||
def build(repo: Path, files: list[str]) -> dict:
|
||||
test_files = [f for f in files if TEST_PATH.search(f) and Path(f).suffix in SOURCE_EXTS]
|
||||
test_set = set(test_files)
|
||||
source_files = [
|
||||
f for f in files
|
||||
if Path(f).suffix in SOURCE_EXTS and f not in test_set and not NON_LOGIC.search(f)
|
||||
]
|
||||
|
||||
# Index what the tests import, plus every bare identifier they mention.
|
||||
imported: set[str] = set()
|
||||
mentioned: set[str] = set()
|
||||
weak: list[dict] = []
|
||||
for rel in test_files:
|
||||
text = read(repo / rel)
|
||||
if not text:
|
||||
continue
|
||||
for regex in IMPORT_RES:
|
||||
for m in regex.findall(text):
|
||||
ref = m.strip()
|
||||
imported.add(ref.lower())
|
||||
imported.add(Path(ref).name.lower())
|
||||
imported.add(Path(ref).stem.lower())
|
||||
for part in re.split(r"[./:\\]", ref):
|
||||
if part and part not in (".", "..", "src", "lib", "app"):
|
||||
mentioned.add(part.lower())
|
||||
assertions = len(ASSERTION_TOKENS.findall(text))
|
||||
cases = len(TEST_CASE_TOKENS.findall(text))
|
||||
lines = text.count("\n") + 1
|
||||
# A test file with cases but almost no assertions is usually asserting nothing useful.
|
||||
if cases and assertions <= max(1, cases // 4):
|
||||
weak.append({"path": rel, "test_cases": cases, "assertions": assertions, "lines": lines})
|
||||
|
||||
# Test files whose filename itself marks them as a test, as opposed to files that
|
||||
# merely live under tests/ or e2e/ (fixtures, helpers, page objects, factories).
|
||||
named_test_re = re.compile(r"([._-](test|spec)s?$)|(^test_)|(^test$)", re.IGNORECASE)
|
||||
named_tests = [t for t in test_files if named_test_re.search(Path(t).stem)]
|
||||
|
||||
matched: dict[str, list[str]] = {}
|
||||
untested: list[dict] = []
|
||||
for rel in source_files:
|
||||
tokens = module_tokens(rel)
|
||||
how: list[str] = []
|
||||
|
||||
stem = Path(rel).stem.lower()
|
||||
mirror = Path(rel).with_suffix("").as_posix().lower()
|
||||
|
||||
# 1. a test file whose *own name* carries a test marker and embeds this file's name.
|
||||
# Requiring the marker matters: a helper like e2e/lib/api.mjs is classified as a
|
||||
# test file because of its directory, and would otherwise "cover" every api.js
|
||||
# in the repo. A false match here hides a real gap, so keep this rule strict.
|
||||
for t in named_tests:
|
||||
tl = Path(t).stem.lower()
|
||||
if tl in {f"{stem}test", f"test{stem}", f"{stem}spec", f"{stem}_test",
|
||||
f"test_{stem}", f"{stem}.test", f"{stem}.spec"} or \
|
||||
re.sub(r"[._-]?(test|spec)s?$", "", tl) == stem:
|
||||
how.append(f"name match: {t}")
|
||||
break
|
||||
# 2. mirrored directory layout, e.g. src/a/b.ts -> tests/a/b.test.ts
|
||||
if not how:
|
||||
tail = "/".join(mirror.split("/")[1:]) if "/" in mirror else mirror
|
||||
if tail and any(tail in t.lower() for t in test_files):
|
||||
how.append("mirrored path match")
|
||||
# 3. a test actually imports it
|
||||
if not how and (tokens & imported):
|
||||
how.append("imported by a test")
|
||||
# 4. weakest signal: a test mentions the identifier
|
||||
if not how and stem in mentioned and len(stem) > 3:
|
||||
how.append("mentioned in a test (weak signal)")
|
||||
|
||||
if how:
|
||||
matched[rel] = how
|
||||
else:
|
||||
lines = read(repo / rel).count("\n") + 1
|
||||
hits = [k for k in RISK_KEYWORDS if k in rel.lower()]
|
||||
untested.append({
|
||||
"path": rel,
|
||||
"lines": lines,
|
||||
"dir": str(Path(rel).parent).replace("\\", "/"),
|
||||
"risk_keywords": hits,
|
||||
})
|
||||
|
||||
untested.sort(key=lambda x: (not x["risk_keywords"], -x["lines"]))
|
||||
weak.sort(key=lambda x: -x["test_cases"])
|
||||
|
||||
by_dir: dict[str, dict] = defaultdict(lambda: {"untested": 0, "lines": 0})
|
||||
for u in untested:
|
||||
by_dir[u["dir"]]["untested"] += 1
|
||||
by_dir[u["dir"]]["lines"] += u["lines"]
|
||||
|
||||
return {
|
||||
"repo": str(repo),
|
||||
"frameworks": detect_frameworks(repo, files),
|
||||
"conventions": infer_convention(test_files),
|
||||
"totals": {
|
||||
"source_files": len(source_files),
|
||||
"test_files": len(test_files),
|
||||
"matched": len(matched),
|
||||
"untested": len(untested),
|
||||
"coverage_ratio": round(len(matched) / len(source_files), 3) if source_files else None,
|
||||
},
|
||||
"untested": untested,
|
||||
"untested_by_directory": sorted(
|
||||
({"dir": d, **v} for d, v in by_dir.items()),
|
||||
key=lambda x: -x["untested"],
|
||||
),
|
||||
"weak_tests": weak,
|
||||
"matched_sample": [{"path": p, "why": w} for p, w in list(matched.items())[:15]],
|
||||
"warnings": warnings,
|
||||
}
|
||||
|
||||
|
||||
def truncate(items: list, limit: int, label: str, out: list[str]) -> list:
|
||||
if len(items) > limit:
|
||||
out.append(f" TRUNCATED: showing top {limit} of {len(items)} {label}")
|
||||
return items[:limit]
|
||||
return items
|
||||
|
||||
|
||||
def render(d: dict, top: int) -> str:
|
||||
t = d["totals"]
|
||||
L = ["# Test Coverage Map", "", f"Repo: {d['repo']}"]
|
||||
ratio = f"{t['coverage_ratio']:.0%}" if t["coverage_ratio"] is not None else "n/a"
|
||||
L.append(f"Source files: {t['source_files']} Test files: {t['test_files']} "
|
||||
f"Matched: {t['matched']} ({ratio}) Unmatched: {t['untested']}")
|
||||
L.append("")
|
||||
L.append(f"Frameworks detected: {', '.join(d['frameworks']) or 'none detected'}")
|
||||
L.append(f"Naming conventions: {', '.join(d['conventions']) or 'none inferred'}")
|
||||
L.append("")
|
||||
|
||||
L.append("## Unmatched source files (risk-flagged first, then largest)")
|
||||
if d["untested"]:
|
||||
for u in truncate(d["untested"], top, "unmatched files", L):
|
||||
flag = f" [risk: {', '.join(u['risk_keywords'])}]" if u["risk_keywords"] else ""
|
||||
L.append(f"- {u['path']}: {u['lines']} lines{flag}")
|
||||
else:
|
||||
L.append("- none; every source file matched at least one test signal")
|
||||
L.append("")
|
||||
|
||||
L.append("## Unmatched by directory")
|
||||
for row in truncate(d["untested_by_directory"], top, "directories", L):
|
||||
L.append(f"- {row['dir']}: {row['untested']} files, {row['lines']} lines")
|
||||
L.append("")
|
||||
|
||||
L.append("## Test files with test cases but few assertions")
|
||||
if d["weak_tests"]:
|
||||
for w in truncate(d["weak_tests"], top, "test files", L):
|
||||
L.append(f"- {w['path']}: {w['test_cases']} cases, {w['assertions']} assertions")
|
||||
else:
|
||||
L.append("- none flagged")
|
||||
L.append("")
|
||||
|
||||
if d["warnings"]:
|
||||
L.append("## Warnings")
|
||||
L.extend(f"- {w}" for w in d["warnings"])
|
||||
L.append("")
|
||||
|
||||
L.append("HEURISTIC MATCHER. An unmatched file is a lead, not a finding: coverage through")
|
||||
L.append("fixtures, end-to-end tests, or indirection is invisible here. Confirm by grepping for")
|
||||
L.append("the module name before reporting anything as P0 or P1.")
|
||||
return "\n".join(L)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="Map source files against tests. Read-only.")
|
||||
ap.add_argument("--repo", default=".", help="Path inside the repository.")
|
||||
ap.add_argument("--format", choices=["text", "json"], default="text", help="Output format.")
|
||||
ap.add_argument("--top", type=int, default=25, help="Rows per section. Default 25.")
|
||||
ap.add_argument(
|
||||
"--no-git-root",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Treat --repo literally instead of expanding to the enclosing git repository root. "
|
||||
"Use this to scope the survey to one package or subdirectory of a monorepo."
|
||||
),
|
||||
)
|
||||
args = ap.parse_args()
|
||||
|
||||
repo = Path(args.repo).resolve()
|
||||
if not repo.is_dir():
|
||||
print(f"error: not a directory: {repo}", file=sys.stderr)
|
||||
return 2
|
||||
if not args.no_git_root:
|
||||
root = run_git(["rev-parse", "--show-toplevel"], repo)
|
||||
if root and root.strip():
|
||||
repo = Path(root.strip()).resolve()
|
||||
|
||||
files = list_files(repo)
|
||||
if not files:
|
||||
print(f"error: no files found under {repo}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
data = build(repo, files)
|
||||
print(json.dumps(data, indent=2) if args.format == "json" else render(data, args.top))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user