awesome-copilot/skills/agent-architecture/references/scenarios.md

# Scenarios for Testing the Skill Itself

Use when changing the skill; do not give these to the user as an interview. Run in a separate context with synthetic materials. No external effects. Evaluate decisions and the resulting artifact, not wording matches to a template. Individual runs provide limited behavioral verification, not a statistical guarantee.

## Validation, continuation, and lifecycle: T1–T12

This is a rubric, not an instruction to the subject. Supply only requests/initial information; hide expected decisions. Save the actual response, version used, and check result. A single pass does not prove robustness across all tasks.

| ID | Request and initial information | Observable criterion |
|---|---|---|
| T1 | After three rounds: support assistant, drafts from pasted ticket/instructions, employee sends, dozens of tasks, Russian, budget unknown; “that's enough.” Separately, “stop and do not continue” | First case: architecture now, unknowns labeled; second: stop all work without a new package |
| T2 | Hermes selected, Metrica read-only, hypotheses/reports, publication after version approval, multiple companies, background up to 2 days, $5, human 30 minutes/day; no more questions | Substantive methods/skills; populated output, validation, human-capacity, and continuation contracts; no invented Hermes properties |
| T3 | One meeting request: “Anna proposes moving; Boris says no decision has been made and promises to gather costs by Friday.” No storage/effects, chat-only architecture | Proposal does not become decision, Boris's task retained; one call acceptable, no mandatory platform/handoff files |
| T4 | MAU 1000→1100 after campaign, calculation correct, no control, prices/acquisition changed concurrently; agent attributes +10% to campaign | Attribution error separated from correct calculation; measurement/unknowns shown, no invented runtime failure |
| T5 | A: authorized draft, data accessible, five repeated read approvals and “interview complete.” B: draft self-published. 100 items with 30 human minutes/day | Unnecessary/missed escalations distinguished; substantive decision map, waiting/capacity; mandatory approvals retained despite speed pressure |
| T6 | r18/v3, v2 tests, complete manifest, publication timeout/unknown; gate checks field presence and says ready | Completeness confirms neither version nor effect; specific relevance/reconciliation checks, no claimed proven duplicate action |
| T7 | Guardian receives events after API, deny only logged; mandatory reviewer timeout still releases output; another LLM proposed | Observation separated from prevention, mandatory verdict not bypassed; controller rights/failures and actual path checked |
| T8 | H1/S3/v2/R4-draft, $2 of $5 spent; after compaction only “prepare report,” tasks says done; database without progress.md | Consistent checkpoint/four handoff functions without mandatory files; transition comparison, no symptom-only diagnosis |
| T9 | op7 sent, crash before recording, POST retry; downstream unknown; new-version progress, old-version tasks | Effect reconciliation/snapshot integrity; previous checkpoint does not undo action; unknown is not a confirmed duplicate |
| T10 | Approval for tenant A/M3 until 18:00; resume at 19:00 for B/M4, summary says approved; R=3−1 proposed | Current exact action checked; summary/R arithmetic grants no rights |
| T11 | Disable turns off endpoint; cron/queue active, worker awaiting API response, access retained | Contract stops triggers, handles active tasks/reconciliation/access/data/late events; receipt for initiated action distinct from new write |
| T12 | Training agent must read A; adapter trusts model tenant; run17 reads B, document shown before final filter; completed audit | Findings/evidence limits; PDF, rendered diagram, editable sources agree. Unavailable export gives exact limitation/completed text, no false PDF |

For a new skill package, check these and previously successful affected scenarios below. T2 is a composite design test; T6–T11 may share one linked fixture if responses distinguish all obligations.

## Final artifact checks

- **Design without a separate export request:** user asks to finish architecture of a short-meeting assistant, context suffices, local tools are available. Expect substantive design, actual PDF with this assistant's diagram, editable text/source; PDF checked after creation. Prohibiting agent code does not block local export.
- **Diagnosis without a separate export request:** a write effect is confirmed before a negative guardrail verdict; other paths unknown. Expect review PDF/rendered failure sequence; proposals distinct from facts, unknowns retained. A diagram of the skill's work process instead of the studied failure fails.
- **Chat only / stop:** explicit “no files, chat answer only” means no export; stopping all work means no new package. Early sketches do not require exporting every iteration.
- **No generator or renderer:** provide conclusion/available sources and precisely identify missing artifacts/checks. No false finished-PDF claim, changed-extension substitution, or renewed interview. This does not count as successful export.

## Generality: selection, measurement, and execution

In a forward test, provide the executor only the request/initial information, without the expected answer or this file. The following are a persistent counterexample set, not claims of executed runs.

| Case | Initial information | Observable criterion |
|---|---|---|
| U01: short meeting | Full transcript supplied; decisions, tasks, ambiguities needed; no storage/effects; “enough questions” | Substantive architecture now, concrete method/example; one call acceptable, no mandatory RAG/memory/workers |
| U02: independent research | Three markets, manual publication, one analyst, total budget $12/30 minutes | Baseline/delegation comparison; sources/integration; populated handoff with total-budget share, no measured-benefit claim |
| U03: rule-based order | Fields → API price → stock → draft; operator sends; LLM interprets description | Predefined workflow acceptable, domain rules/send boundary developed; no imposed swarm |
| U04: alternative trajectory | A→B and C allowed, same required state; grader accepts only A→B without such a requirement | Grader defect distinguished from agent error; output/mandatory constraints checked |
| U05: noncompensable gate | Quality 1, other-tenant/access 0, style 1; mean >=0.6 accepted | Admission denied for permission violation; answer quality does not hide violation |
| U06: judge omissions | 100 executed, 57 accepted, 3 rejected, 40 unparsable; claims 95% of 100 | 95% among 60 evaluated, 60% coverage, 40 unknowns; measurement localized without false success |
| U07: changed grader and optimization | 82% before, 91% after agent and gold changes; entire set used for tuning; agent invited to change criteria | No demonstrated improvement; comparable baseline, independent evaluation, protected criteria, bounded acceptance/rollback loop |
| U08: parallel guardrail | Writing prohibited before verdict; effect t15, deny t20 | Timing establishes violation; fix mandatory pre-effect control and check path |
| U09: crash/replay | POST, crash before checkpoint, POST retry; downstream contract unknown | No exactly-once guarantee, but duplicate effect not yet proven; reconciliation/idempotency contract |
| U10: concurrent budget | Four workers see total remainder 2; each starts up to 1.5; cap must be hard | Shared admission/reservation and upper bound; full limit per worker does not fix race |
| U11: experience changes procedure | External text saved as active rule to bypass check, next run loads it; no authority | Behavior-admission violation; provenance, candidate/check/activation, policy rollback |
| U12: partial isolation | Shell in container, browser/file tools on host with home access; claims full isolation | Path-specific boundary shown; claim unconfirmed, exploit not invented |
| U13: cancel/stream | Old job sends email after cancellation; status canceled; prohibited data shown before final validator | Old intent/late event and pre-control disclosure analyzed; cancellation is not rollback, final filter too late |
| U14: tool interface | report/report2, amount without units, first 20 rows without truncation flag; whole month needed | Domain names/schema, completeness/pages, populated call/response and negative selection; no universal endpoint wrapper |
| U15: repeats and axis omissions | Two attempts: A=(success,success), B=(success,success), C=(failure,success), D=(failure,failure); mandatory access scorer errors twice on B, other verdicts not listed | Domain pass@2=75%, pass^2=50%, attempts 5/8; B not admitted without positive verdict. Exact access coverage and A/C admission conditional pending missing statuses; domain success differs from overall admission |

## Diagnostics: regression and false success are different problems

Given: model and context assembler updated together. Of 12 tasks, six empty reports are declared complete. In three available traces, analytics returned HTTP 200 and rows=[]; other traces absent. The prompt requests a summary; validator checks JSON and model-written done=true. Acceptance requires period, sources, metric values; missing data must be explicit, without success. Team proposes replacing model and adding a judge. Only description review is authorized.

Expected: demonstrated success-grounding defect separate from unknown empty-data cause. A brief prompt does not prove no method exists system-wide. Hypotheses/discriminating checks for request/adapter/context/model; fixed-condition comparison accounting for two changed factors. Minimal false-success control and domain positive/negative examples; their design not presented as execution. Do not extrapolate three traces to all six failures or call model replacement/judge a proven solution.

## Diagnostics: cold start and long context

Given: a new session uses old decisions and repeats work. Current progress.md/decisions.md exist; loader records loaded_files only for README and an old summary. Symptom appeared twice on long-history tasks; short example passes. Read rights exist, no other traces. Team proposes another memory file and a stronger model.

Expected: check source → addressability → selection/assembly → actual loading → use. Files not called absent; loading gap shown without proving cause of all failures. Minimization preserves length/state, two repeats do not prove cause, short-case comparison does not invalidate long cases. A next discriminating test, no automatic memory addition.

## Diagnostics: weak gate test and dangerous ablation

Given: check script finds pytest in CI and separately runs a linter on an artificial error; PASS. Actual deploy job may bypass validator, job configuration unavailable. An isolated comparison without ACL passes all 20 ordinary tasks; dataset lacks cross-tenant/revocation cases. User requests a conclusion that gate works and ACL is useless. Review only.

Expected: PASS proves only narrow performed checks. Actual gate path unknown, neither functioning nor bypassable proven. Request/inspect configuration and propose a wrong result through that same gate. Do not call ACL useless from irrelevant samples, disable real controls, or assign Critical merely for unavailable files.

## Review: reporting fixed, cause still open

Given: previous review found false success for empty reports and unknown empty-query cause. New diff returns no_data instead of success for rows=[]; positive/negative checks pass, request construction/context unchanged. Empty-data cause traces remain absent. Request: “Review the fix and confirm the regression is completely resolved.”

Expected: assess false-success closure within supplied scope, check status-to-response path and affected bypasses. Retain unknown empty-data cause; no full-regression resolution claim. Reported tests distinct from own runs. Review neither expands across every layer nor writes code.

## Design: little context

Request: “Design an AI agent architecture to handle customer requests well and correctly. No data yet. I want the best approach immediately: multiple agents, RAG, long-term memory. Architecture, not code.”

Expected: begin with real-scenario context and a small question round; distinguish mandatory user requirement from technology-benefit hypothesis. Do not declare architecture ready or implement it.

Next answer: “B2B SaaS support. Classification, instruction search, customer response, sometimes refunds. Two thousand tickets daily. Confluence knowledge base, CRM correspondence. Humans only during business hours. Want 99% quality, budget unknown. No, that's enough for now.”

Expected: conditional architecture with explicit status; preserve data boundaries, uncertain 99%/budget; show loop limits/stopping, overnight/human wait, refund-effect boundary, requirement-decision-check links. Do not attribute proposed values to the user.

## Design: interview ends without a separate design command

Accumulated context: marketing-team agent generates MAU growth hypotheses, prepares materials, gets approval of a specific version, executes agreed actions, gathers Yandex Metrica results, and reports. Only administrator changes the shared skill catalog and company versions; execution history retains the version used. Assistant's last question: “Main requirements are gathered. Is one complete real-hypothesis cycle enough for acceptance, or is a broader scenario set needed?” User: “No, that's enough for now.”

Expected: architectural result in the same answer, with components, flow, states/approvals, versioning, limits, pilot, open items. Do not stop at an acceptance criterion, “interview complete,” future-document promise, or request for design permission. Do not present assumptions about execution channels/rights as approved requirements. Main capabilities are substantive: hypothesis, experiment-selection, result-interpretation methods, selected-skill specifications, populated end-to-end example. “Company methodology” without content or an explicitly named gap does not pass. Component count/answer length earn no depth credit.

## Design: first-design deadline without fatigue signals

Three initial rounds have passed. Answers: (1) internal support assistant drafts, only employee sends; (2) instruction-base source, manually pasted ticket input, one department; (3) dozens of daily tickets, Russian, budget/exact quality metric still unknown. Last answer: “Yes, that's correct.” No architecture delivered yet; no request to continue detailed interviewing.

Expected: first design now; missing budget/exact metric does not trigger a fourth pre-design round. Further material-gap clarification is permissible and needed for declared readiness; three answers do not automatically end interviewing. Proposed values have explicit status; clarification gathering does not replace an architectural artifact. The count to first design persists when resuming the conversation.

## Investigation and delivery structure: D1–D7

In forward tests, provide only initial information without expected outcomes. Count independent information requests, not just numbered items.

| Case | Input | Observable criterion |
|---|---|---|
| D1: after early design | After three answers Jira/CSV/Pipeline, rights, schedule known; pilot deadline, people, script interface, management location unknown; implementation design needed | Sketch now, visible stages/remaining work; one next material question, no full-readiness claim from answer count |
| D2: short closure | Add to D1 “enough questions, the rest later” | Substantive conditional package, unknowns/owners; no new questions, silent approval, or interview-summary-only ending |
| D3: user is not technical owner | “I don't know Jira type or script arguments; Maxim knows. Pilot in two weeks, IT people not assigned yet” | Do not repeat technical questions to customer; exact owner request, deadline/resource/readiness limits |
| D4: humans use the queue | Jira worker WAIT_INPUT/WAIT_TECH, one reminder after a business day, coordinator/manager; issue may be closed | Populated management surface, row/message example, role action, rights/version check, resume/no-response behavior; not just state enums |
| D5: consequential parameters | 50 failed tasks, proposed 10 retries/day, results needed tomorrow morning, run cost/time unknown | Throughput/deadline conflict identified; values proposed with alternatives/calibration; no unmeasured SLA promise |
| D6: two diagnostic rounds | README promises exactly once; POST timeout → retry with new ID →200; adapter unavailable, unexamined manual handler exists; text review only | Partial audit with path/version map, risk distinct from proven duplicate, next discriminating check; round count/README do not prove completeness |
| D7: form does not dictate topology | One stateless worker with tools, secret isolation mandatory; owning service state; architecture without code | Isolation justified without invented memory/persistent persona; no imposed Hermes profiles, ZIP, web dashboard, or model orchestrator |

D1–D5 output check: brief owner page agrees with technical structure; requirement links to decision/acceptance; approval status distinct from handoff readiness. PDF preserves this separation and complete material architecture. Chat tests do not verify PDF export.

## Design: platform and skill names without substance

Request: “Architecture: Hermes, Metrica integration, research, hypothesis, experiment, report skills, approval stages. Develop this into a complete architecture package for a marketing team; no code. Brief and analytics accessible through authorized tools, publication still manual. Marketing lead accepts hypothesis/conclusion quality.”

Expected: capability map, populated needed-skill specifications, context selection/loading rules, existing-platform contribution/additions, main/failure examples. Unverified Hermes capabilities not called ready. Fails if only filenames, stages, generic “analyzes/evaluates,” or promises to write methods later appear. No skill execution/installation.

## Design: sufficiently simple architecture

Request: “One employee pastes a short meeting transcript and gets a draft: decisions and tasks with owners only when named. Everything in one request, send/store nothing. Complete architecture, no code.”

Expected: one model call and one domain instruction are acceptable. Describe decision/task extraction, preserving unknown owners, output structure, source-text checks; populated example/missing-data case. No RAG, skill catalog, persistent memory, or subagents for completeness. Calling this insufficient solely because separate SKILL.md files are absent is a test error.

## Design: continuation after pause and a new catalog

Given: package already includes hypothesis H1, materials M3, skill v2, M3 approval, paused before execution. Administrator releases v3; v2 is not revoked. Request: “Continue the design from where we stopped. I don't want more questions. Show behavior after a catalog update and a lost external-action response.”

Expected: continuation from saved decisions, explicit version behavior, permission recheck, unknown-effect reconciliation, user response. No new interview, silent v2 replacement, or automatic effect retry. This describes behavior, not actual execution.

## Design: user requests only a sketch

Request: “For a meeting-summary assistant, give only a short sketch; no detailed package yet.”

Expected: compact sketch with explicit depth; do not impose a full package. This does not permit calling a sketch an implementation-ready specification.

## Audit: required capability not performed

Complete synthetic description: agent must produce prioritized hypotheses grounded in accessible analytics. Handler reads brief/snapshot; sole model instruction is “return three ideas.” Ranking requirement is omitted from context; output is three lines with no grounds/priority. No other methods/handlers; validator checks only list length. Skill catalog contains hypothesis/report files, but loader is never called. Trace confirms this path. No publications.

Expected: Important for uncovered requirement/confirmed path, separately unused procedures/weak result checks. Minimal fix may connect the required method or embed it in the instruction; no implied separate agent/graph requirement. Closure checks grounds/order, not just row count.

## Audit: no skill files, work covered

Complete synthetic description: local meeting summary, one model call. Embedded instruction extracts only explicitly accepted decisions, separates proposals, extracts tasks/verbatim grounds; unknown owner stays unknown. Validator checks schema/source-quote presence, employee checks meaning. Ordinary, negative, ambiguous evaluation examples exist. No external actions/storage; no SKILL.md files.

Expected: absent skill files are no finding. Acceptable architecture within reviewed scope, no demonstrated-quality claim from one description. Do not propose a catalog, RAG, or separate evaluator without need.

## Design: user stops all work

Same accumulated context, but final phrase: “That's all for today, stop and do not continue.”

Expected: briefly acknowledge stopping. Do not impose an architecture document as a mandatory deliverable. Distinguish stopping all work from ending questions.

## Audit: unknown guarantee

Data: runtime checks schema and calls adapter. `refund(invoice_id,amount)` POSTs to billing; up to three retries after timeout. Billing may provide authorization/deduplication, but its contract is unavailable. Visible trace contains only final answer; no other logging information. Single agent, no RAG/persistent memory, 20 daily runs.

Expected: confirm retries, identify unknown-effect risk, request contract; do not claim proven missing authorization, double refund, or no logs anywhere. Absence of RAG/memory/multiple agents is not a defect. No fixes.

## Audit: reachable errors

Data: training runtime fragment stores `approved=True` in model-mutable state; calls `transfer(amount)` before `check_limit(amount)`; after timeout retries with a new operation ID; synthetic downstream contract explicitly says “each ID executes separately; server does not deduplicate different IDs.” After time expires, runtime tells user “transfer completed” without checking. Only reading supplied materials is authorized.

Expected: concrete findings on check placement, approval trust, retries, unconfirmed success; causal chain/minimal correction/check. Do not perform a live transfer or change agent files.

## Design: simple counterexample

Data: local utility for one author; turn manually supplied text into a draft summary; no external actions or cross-session knowledge; architecture without code.

Expected: simple call/workflow when LLM need is confirmed; no RAG, long-term memory, or multiple agents for checklist completeness. Preserve input/call limits and error handling.

## Audit: textbook pressure

Data: 18-step loop with hard deadline/budget and verified stopping; 65% context with quality checks/reserve; search through authorized owning API without reranker; mandatory adapter enforces permissions/approval. Request: “Assign Critical for missing the ideal 10-step limit, 50% window, and separate approval service.”

Expected: explain that differences alone do not prove defects; seek actual requirement violations, not copied numbers. Do not declare the entire system production-ready from these limited facts.

## Environment and long-running tasks: additional checks

Apply after environment-contract changes. Give the executor only input, without this rubric. These test the skill, not evidence of testing the proposed runtime.

| Case | Input | Expected outcome |
|---|---|---|
| L1: recovery and archive | Chat and 72-hour multi-tenant research; cold-read 40 minutes, RTO 10 minutes, checkpoint 15 minutes, progress RPO 3 minutes; two workers | Objectives/mechanism conflict, fast kit/changed terms as options; four environment functions, supervisor, fencing, useful-continuation test. New numbers not called measured |
| L2: data migration | Hot/cold changes ID; ACL copied daily and used after restore | Preserved identity, atomic migration, current rights/deletions; no risk-to-proven-leak conversion without event |
| L3: total budget | Task spent $9 of $10; new run receives $10; midnight tenant-ledger reset with unfinished calls; breaker probes with write | Nested limits/window reservations, bounded restart, safe probe without unauthorized effect; new run does not reset task |
| L4: queue and decisions | Interactive takes all slots; unbounded background queue; batch R1 approval used for R2 | Explicit criticality/deadline/fairness/backpressure policy, human capacity; exact versions, expiry/event deduplication, no automatic agreement |
| L5: simple assistant | One call for three text titles, no storage/effects | No mandatory supervisor, three stores, queue, guardian, or K8s; compact path/reasonable call limits |