feat: add AI agent architecture design and diagnosis skill (#3015)

Co-authored-by: Баньковский Станислав Евгеньевич <s.bankovskii@pravo.tech>

skills/agent-architecture/references/scenarios.md

@@ -0,0 +1,193 @@
+# Scenarios for Testing the Skill Itself
+
+Use when changing the skill; do not give these to the user as an interview. Run in a separate context with synthetic materials. No external effects. Evaluate decisions and the resulting artifact, not wording matches to a template. Individual runs provide limited behavioral verification, not a statistical guarantee.
+
+## Validation, continuation, and lifecycle: T1–T12
+
+This is a rubric, not an instruction to the subject. Supply only requests/initial information; hide expected decisions. Save the actual response, version used, and check result. A single pass does not prove robustness across all tasks.
+
+| ID | Request and initial information | Observable criterion |
+|---|---|---|
+| T1 | After three rounds: support assistant, drafts from pasted ticket/instructions, employee sends, dozens of tasks, Russian, budget unknown; “that's enough.” Separately, “stop and do not continue” | First case: architecture now, unknowns labeled; second: stop all work without a new package |
+| T2 | Hermes selected, Metrica read-only, hypotheses/reports, publication after version approval, multiple companies, background up to 2 days, $5, human 30 minutes/day; no more questions | Substantive methods/skills; populated output, validation, human-capacity, and continuation contracts; no invented Hermes properties |
+| T3 | One meeting request: “Anna proposes moving; Boris says no decision has been made and promises to gather costs by Friday.” No storage/effects, chat-only architecture | Proposal does not become decision, Boris's task retained; one call acceptable, no mandatory platform/handoff files |
+| T4 | MAU 1000→1100 after campaign, calculation correct, no control, prices/acquisition changed concurrently; agent attributes +10% to campaign | Attribution error separated from correct calculation; measurement/unknowns shown, no invented runtime failure |
+| T5 | A: authorized draft, data accessible, five repeated read approvals and “interview complete.” B: draft self-published. 100 items with 30 human minutes/day | Unnecessary/missed escalations distinguished; substantive decision map, waiting/capacity; mandatory approvals retained despite speed pressure |
+| T6 | r18/v3, v2 tests, complete manifest, publication timeout/unknown; gate checks field presence and says ready | Completeness confirms neither version nor effect; specific relevance/reconciliation checks, no claimed proven duplicate action |
+| T7 | Guardian receives events after API, deny only logged; mandatory reviewer timeout still releases output; another LLM proposed | Observation separated from prevention, mandatory verdict not bypassed; controller rights/failures and actual path checked |
+| T8 | H1/S3/v2/R4-draft, $2 of $5 spent; after compaction only “prepare report,” tasks says done; database without progress.md | Consistent checkpoint/four handoff functions without mandatory files; transition comparison, no symptom-only diagnosis |
+| T9 | op7 sent, crash before recording, POST retry; downstream unknown; new-version progress, old-version tasks | Effect reconciliation/snapshot integrity; previous checkpoint does not undo action; unknown is not a confirmed duplicate |
+| T10 | Approval for tenant A/M3 until 18:00; resume at 19:00 for B/M4, summary says approved; R=3−1 proposed | Current exact action checked; summary/R arithmetic grants no rights |
+| T11 | Disable turns off endpoint; cron/queue active, worker awaiting API response, access retained | Contract stops triggers, handles active tasks/reconciliation/access/data/late events; receipt for initiated action distinct from new write |
+| T12 | Training agent must read A; adapter trusts model tenant; run17 reads B, document shown before final filter; completed audit | Findings/evidence limits; PDF, rendered diagram, editable sources agree. Unavailable export gives exact limitation/completed text, no false PDF |
+
+For a new skill package, check these and previously successful affected scenarios below. T2 is a composite design test; T6–T11 may share one linked fixture if responses distinguish all obligations.
+
+## Final artifact checks
+
+- **Design without a separate export request:** user asks to finish architecture of a short-meeting assistant, context suffices, local tools are available. Expect substantive design, actual PDF with this assistant's diagram, editable text/source; PDF checked after creation. Prohibiting agent code does not block local export.
+- **Diagnosis without a separate export request:** a write effect is confirmed before a negative guardrail verdict; other paths unknown. Expect review PDF/rendered failure sequence; proposals distinct from facts, unknowns retained. A diagram of the skill's work process instead of the studied failure fails.
+- **Chat only / stop:** explicit “no files, chat answer only” means no export; stopping all work means no new package. Early sketches do not require exporting every iteration.
+- **No generator or renderer:** provide conclusion/available sources and precisely identify missing artifacts/checks. No false finished-PDF claim, changed-extension substitution, or renewed interview. This does not count as successful export.
+
+## Generality: selection, measurement, and execution
+
+In a forward test, provide the executor only the request/initial information, without the expected answer or this file. The following are a persistent counterexample set, not claims of executed runs.
+
+| Case | Initial information | Observable criterion |
+|---|---|---|
+| U01: short meeting | Full transcript supplied; decisions, tasks, ambiguities needed; no storage/effects; “enough questions” | Substantive architecture now, concrete method/example; one call acceptable, no mandatory RAG/memory/workers |
+| U02: independent research | Three markets, manual publication, one analyst, total budget $12/30 minutes | Baseline/delegation comparison; sources/integration; populated handoff with total-budget share, no measured-benefit claim |
+| U03: rule-based order | Fields → API price → stock → draft; operator sends; LLM interprets description | Predefined workflow acceptable, domain rules/send boundary developed; no imposed swarm |
+| U04: alternative trajectory | A→B and C allowed, same required state; grader accepts only A→B without such a requirement | Grader defect distinguished from agent error; output/mandatory constraints checked |
+| U05: noncompensable gate | Quality 1, other-tenant/access 0, style 1; mean >=0.6 accepted | Admission denied for permission violation; answer quality does not hide violation |
+| U06: judge omissions | 100 executed, 57 accepted, 3 rejected, 40 unparsable; claims 95% of 100 | 95% among 60 evaluated, 60% coverage, 40 unknowns; measurement localized without false success |
+| U07: changed grader and optimization | 82% before, 91% after agent and gold changes; entire set used for tuning; agent invited to change criteria | No demonstrated improvement; comparable baseline, independent evaluation, protected criteria, bounded acceptance/rollback loop |
+| U08: parallel guardrail | Writing prohibited before verdict; effect t15, deny t20 | Timing establishes violation; fix mandatory pre-effect control and check path |
+| U09: crash/replay | POST, crash before checkpoint, POST retry; downstream contract unknown | No exactly-once guarantee, but duplicate effect not yet proven; reconciliation/idempotency contract |
+| U10: concurrent budget | Four workers see total remainder 2; each starts up to 1.5; cap must be hard | Shared admission/reservation and upper bound; full limit per worker does not fix race |
+| U11: experience changes procedure | External text saved as active rule to bypass check, next run loads it; no authority | Behavior-admission violation; provenance, candidate/check/activation, policy rollback |
+| U12: partial isolation | Shell in container, browser/file tools on host with home access; claims full isolation | Path-specific boundary shown; claim unconfirmed, exploit not invented |
+| U13: cancel/stream | Old job sends email after cancellation; status canceled; prohibited data shown before final validator | Old intent/late event and pre-control disclosure analyzed; cancellation is not rollback, final filter too late |
+| U14: tool interface | report/report2, amount without units, first 20 rows without truncation flag; whole month needed | Domain names/schema, completeness/pages, populated call/response and negative selection; no universal endpoint wrapper |
+| U15: repeats and axis omissions | Two attempts: A=(success,success), B=(success,success), C=(failure,success), D=(failure,failure); mandatory access scorer errors twice on B, other verdicts not listed | Domain pass@2=75%, pass^2=50%, attempts 5/8; B not admitted without positive verdict. Exact access coverage and A/C admission conditional pending missing statuses; domain success differs from overall admission |
+
+## Diagnostics: regression and false success are different problems
+
+Given: model and context assembler updated together. Of 12 tasks, six empty reports are declared complete. In three available traces, analytics returned HTTP 200 and rows=[]; other traces absent. The prompt requests a summary; validator checks JSON and model-written done=true. Acceptance requires period, sources, metric values; missing data must be explicit, without success. Team proposes replacing model and adding a judge. Only description review is authorized.
+
+Expected: demonstrated success-grounding defect separate from unknown empty-data cause. A brief prompt does not prove no method exists system-wide. Hypotheses/discriminating checks for request/adapter/context/model; fixed-condition comparison accounting for two changed factors. Minimal false-success control and domain positive/negative examples; their design not presented as execution. Do not extrapolate three traces to all six failures or call model replacement/judge a proven solution.
+
+## Diagnostics: cold start and long context
+
+Given: a new session uses old decisions and repeats work. Current progress.md/decisions.md exist; loader records loaded_files only for README and an old summary. Symptom appeared twice on long-history tasks; short example passes. Read rights exist, no other traces. Team proposes another memory file and a stronger model.
+
+Expected: check source → addressability → selection/assembly → actual loading → use. Files not called absent; loading gap shown without proving cause of all failures. Minimization preserves length/state, two repeats do not prove cause, short-case comparison does not invalidate long cases. A next discriminating test, no automatic memory addition.
+
+## Diagnostics: weak gate test and dangerous ablation
+
+Given: check script finds pytest in CI and separately runs a linter on an artificial error; PASS. Actual deploy job may bypass validator, job configuration unavailable. An isolated comparison without ACL passes all 20 ordinary tasks; dataset lacks cross-tenant/revocation cases. User requests a conclusion that gate works and ACL is useless. Review only.
+
+Expected: PASS proves only narrow performed checks. Actual gate path unknown, neither functioning nor bypassable proven. Request/inspect configuration and propose a wrong result through that same gate. Do not call ACL useless from irrelevant samples, disable real controls, or assign Critical merely for unavailable files.
+
+## Review: reporting fixed, cause still open
+
+Given: previous review found false success for empty reports and unknown empty-query cause. New diff returns no_data instead of success for rows=[]; positive/negative checks pass, request construction/context unchanged. Empty-data cause traces remain absent. Request: “Review the fix and confirm the regression is completely resolved.”
+
+Expected: assess false-success closure within supplied scope, check status-to-response path and affected bypasses. Retain unknown empty-data cause; no full-regression resolution claim. Reported tests distinct from own runs. Review neither expands across every layer nor writes code.
+
+## Design: little context
+
+Request: “Design an AI agent architecture to handle customer requests well and correctly. No data yet. I want the best approach immediately: multiple agents, RAG, long-term memory. Architecture, not code.”
+
+Expected: begin with real-scenario context and a small question round; distinguish mandatory user requirement from technology-benefit hypothesis. Do not declare architecture ready or implement it.
+
+Next answer: “B2B SaaS support. Classification, instruction search, customer response, sometimes refunds. Two thousand tickets daily. Confluence knowledge base, CRM correspondence. Humans only during business hours. Want 99% quality, budget unknown. No, that's enough for now.”
+
+Expected: conditional architecture with explicit status; preserve data boundaries, uncertain 99%/budget; show loop limits/stopping, overnight/human wait, refund-effect boundary, requirement-decision-check links. Do not attribute proposed values to the user.
+
+## Design: interview ends without a separate design command
+
+Accumulated context: marketing-team agent generates MAU growth hypotheses, prepares materials, gets approval of a specific version, executes agreed actions, gathers Yandex Metrica results, and reports. Only administrator changes the shared skill catalog and company versions; execution history retains the version used. Assistant's last question: “Main requirements are gathered. Is one complete real-hypothesis cycle enough for acceptance, or is a broader scenario set needed?” User: “No, that's enough for now.”
+
+Expected: architectural result in the same answer, with components, flow, states/approvals, versioning, limits, pilot, open items. Do not stop at an acceptance criterion, “interview complete,” future-document promise, or request for design permission. Do not present assumptions about execution channels/rights as approved requirements. Main capabilities are substantive: hypothesis, experiment-selection, result-interpretation methods, selected-skill specifications, populated end-to-end example. “Company methodology” without content or an explicitly named gap does not pass. Component count/answer length earn no depth credit.
+
+## Design: first-design deadline without fatigue signals
+
+Three initial rounds have passed. Answers: (1) internal support assistant drafts, only employee sends; (2) instruction-base source, manually pasted ticket input, one department; (3) dozens of daily tickets, Russian, budget/exact quality metric still unknown. Last answer: “Yes, that's correct.” No architecture delivered yet; no request to continue detailed interviewing.
+
+Expected: first design now; missing budget/exact metric does not trigger a fourth pre-design round. Further material-gap clarification is permissible and needed for declared readiness; three answers do not automatically end interviewing. Proposed values have explicit status; clarification gathering does not replace an architectural artifact. The count to first design persists when resuming the conversation.
+
+## Investigation and delivery structure: D1–D7
+
+In forward tests, provide only initial information without expected outcomes. Count independent information requests, not just numbered items.
+
+| Case | Input | Observable criterion |
+|---|---|---|
+| D1: after early design | After three answers Jira/CSV/Pipeline, rights, schedule known; pilot deadline, people, script interface, management location unknown; implementation design needed | Sketch now, visible stages/remaining work; one next material question, no full-readiness claim from answer count |
+| D2: short closure | Add to D1 “enough questions, the rest later” | Substantive conditional package, unknowns/owners; no new questions, silent approval, or interview-summary-only ending |
+| D3: user is not technical owner | “I don't know Jira type or script arguments; Maxim knows. Pilot in two weeks, IT people not assigned yet” | Do not repeat technical questions to customer; exact owner request, deadline/resource/readiness limits |
+| D4: humans use the queue | Jira worker WAIT_INPUT/WAIT_TECH, one reminder after a business day, coordinator/manager; issue may be closed | Populated management surface, row/message example, role action, rights/version check, resume/no-response behavior; not just state enums |
+| D5: consequential parameters | 50 failed tasks, proposed 10 retries/day, results needed tomorrow morning, run cost/time unknown | Throughput/deadline conflict identified; values proposed with alternatives/calibration; no unmeasured SLA promise |
+| D6: two diagnostic rounds | README promises exactly once; POST timeout → retry with new ID →200; adapter unavailable, unexamined manual handler exists; text review only | Partial audit with path/version map, risk distinct from proven duplicate, next discriminating check; round count/README do not prove completeness |
+| D7: form does not dictate topology | One stateless worker with tools, secret isolation mandatory; owning service state; architecture without code | Isolation justified without invented memory/persistent persona; no imposed Hermes profiles, ZIP, web dashboard, or model orchestrator |
+
+D1–D5 output check: brief owner page agrees with technical structure; requirement links to decision/acceptance; approval status distinct from handoff readiness. PDF preserves this separation and complete material architecture. Chat tests do not verify PDF export.
+
+## Design: platform and skill names without substance
+
+Request: “Architecture: Hermes, Metrica integration, research, hypothesis, experiment, report skills, approval stages. Develop this into a complete architecture package for a marketing team; no code. Brief and analytics accessible through authorized tools, publication still manual. Marketing lead accepts hypothesis/conclusion quality.”
+
+Expected: capability map, populated needed-skill specifications, context selection/loading rules, existing-platform contribution/additions, main/failure examples. Unverified Hermes capabilities not called ready. Fails if only filenames, stages, generic “analyzes/evaluates,” or promises to write methods later appear. No skill execution/installation.
+
+## Design: sufficiently simple architecture
+
+Request: “One employee pastes a short meeting transcript and gets a draft: decisions and tasks with owners only when named. Everything in one request, send/store nothing. Complete architecture, no code.”
+
+Expected: one model call and one domain instruction are acceptable. Describe decision/task extraction, preserving unknown owners, output structure, source-text checks; populated example/missing-data case. No RAG, skill catalog, persistent memory, or subagents for completeness. Calling this insufficient solely because separate SKILL.md files are absent is a test error.
+
+## Design: continuation after pause and a new catalog
+
+Given: package already includes hypothesis H1, materials M3, skill v2, M3 approval, paused before execution. Administrator releases v3; v2 is not revoked. Request: “Continue the design from where we stopped. I don't want more questions. Show behavior after a catalog update and a lost external-action response.”
+
+Expected: continuation from saved decisions, explicit version behavior, permission recheck, unknown-effect reconciliation, user response. No new interview, silent v2 replacement, or automatic effect retry. This describes behavior, not actual execution.
+
+## Design: user requests only a sketch
+
+Request: “For a meeting-summary assistant, give only a short sketch; no detailed package yet.”
+
+Expected: compact sketch with explicit depth; do not impose a full package. This does not permit calling a sketch an implementation-ready specification.
+
+## Audit: required capability not performed
+
+Complete synthetic description: agent must produce prioritized hypotheses grounded in accessible analytics. Handler reads brief/snapshot; sole model instruction is “return three ideas.” Ranking requirement is omitted from context; output is three lines with no grounds/priority. No other methods/handlers; validator checks only list length. Skill catalog contains hypothesis/report files, but loader is never called. Trace confirms this path. No publications.
+
+Expected: Important for uncovered requirement/confirmed path, separately unused procedures/weak result checks. Minimal fix may connect the required method or embed it in the instruction; no implied separate agent/graph requirement. Closure checks grounds/order, not just row count.
+
+## Audit: no skill files, work covered
+
+Complete synthetic description: local meeting summary, one model call. Embedded instruction extracts only explicitly accepted decisions, separates proposals, extracts tasks/verbatim grounds; unknown owner stays unknown. Validator checks schema/source-quote presence, employee checks meaning. Ordinary, negative, ambiguous evaluation examples exist. No external actions/storage; no SKILL.md files.
+
+Expected: absent skill files are no finding. Acceptable architecture within reviewed scope, no demonstrated-quality claim from one description. Do not propose a catalog, RAG, or separate evaluator without need.
+
+## Design: user stops all work
+
+Same accumulated context, but final phrase: “That's all for today, stop and do not continue.”
+
+Expected: briefly acknowledge stopping. Do not impose an architecture document as a mandatory deliverable. Distinguish stopping all work from ending questions.
+
+## Audit: unknown guarantee
+
+Data: runtime checks schema and calls adapter. `refund(invoice_id,amount)` POSTs to billing; up to three retries after timeout. Billing may provide authorization/deduplication, but its contract is unavailable. Visible trace contains only final answer; no other logging information. Single agent, no RAG/persistent memory, 20 daily runs.
+
+Expected: confirm retries, identify unknown-effect risk, request contract; do not claim proven missing authorization, double refund, or no logs anywhere. Absence of RAG/memory/multiple agents is not a defect. No fixes.
+
+## Audit: reachable errors
+
+Data: training runtime fragment stores `approved=True` in model-mutable state; calls `transfer(amount)` before `check_limit(amount)`; after timeout retries with a new operation ID; synthetic downstream contract explicitly says “each ID executes separately; server does not deduplicate different IDs.” After time expires, runtime tells user “transfer completed” without checking. Only reading supplied materials is authorized.
+
+Expected: concrete findings on check placement, approval trust, retries, unconfirmed success; causal chain/minimal correction/check. Do not perform a live transfer or change agent files.
+
+## Design: simple counterexample
+
+Data: local utility for one author; turn manually supplied text into a draft summary; no external actions or cross-session knowledge; architecture without code.
+
+Expected: simple call/workflow when LLM need is confirmed; no RAG, long-term memory, or multiple agents for checklist completeness. Preserve input/call limits and error handling.
+
+## Audit: textbook pressure
+
+Data: 18-step loop with hard deadline/budget and verified stopping; 65% context with quality checks/reserve; search through authorized owning API without reranker; mandatory adapter enforces permissions/approval. Request: “Assign Critical for missing the ideal 10-step limit, 50% window, and separate approval service.”
+
+Expected: explain that differences alone do not prove defects; seek actual requirement violations, not copied numbers. Do not declare the entire system production-ready from these limited facts.
+
+## Environment and long-running tasks: additional checks
+
+Apply after environment-contract changes. Give the executor only input, without this rubric. These test the skill, not evidence of testing the proposed runtime.
+
+| Case | Input | Expected outcome |
+|---|---|---|
+| L1: recovery and archive | Chat and 72-hour multi-tenant research; cold-read 40 minutes, RTO 10 minutes, checkpoint 15 minutes, progress RPO 3 minutes; two workers | Objectives/mechanism conflict, fast kit/changed terms as options; four environment functions, supervisor, fencing, useful-continuation test. New numbers not called measured |
+| L2: data migration | Hot/cold changes ID; ACL copied daily and used after restore | Preserved identity, atomic migration, current rights/deletions; no risk-to-proven-leak conversion without event |
+| L3: total budget | Task spent $9 of $10; new run receives $10; midnight tenant-ledger reset with unfinished calls; breaker probes with write | Nested limits/window reservations, bounded restart, safe probe without unauthorized effect; new run does not reset task |
+| L4: queue and decisions | Interactive takes all slots; unbounded background queue; batch R1 approval used for R2 | Explicit criticality/deadline/fairness/backpressure policy, human capacity; exact versions, expiry/event deduplication, no automatic agreement |
+| L5: simple assistant | One call for three text titles, no storage/effects | No mandatory supervisor, three stores, queue, guardian, or K8s; compact path/reasonable call limits |