mirror of
https://github.com/github/awesome-copilot.git
synced 2026-08-19 07:28:43 +00:00
Merge upstream main and migrate daily focus board plugin
Adopt the Agent Plugins v1.0.0 namespaced composition model from #2546. Bundle the reusable daily-focus-board canvas into Ember only and remove its obsolete standalone catalog manifest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52be9c67-3ae4-4610-93d0-fe0b7ab95ccb
This commit is contained in:
Jenny Ferries
@@ -0,0 +1,134 @@
|
||||
---
|
||||
name: azure-developer-cli
|
||||
description: 'Design, create, review, migrate, or troubleshoot Azure Developer CLI (azd) projects using current Microsoft guidance. Use for azd, azure.yaml, AZD templates, Bicep or Terraform under infra, AZD environments and secrets, hooks, deployment workflows, and azd-managed CI/CD.'
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Azure Developer CLI best practices
|
||||
|
||||
Use this skill to produce maintainable, secure, environment-aware `azd` projects. Prefer repository conventions when they are already coherent, and make the smallest complete change that improves the project.
|
||||
|
||||
## Start with repository discovery
|
||||
|
||||
Before editing:
|
||||
|
||||
1. Find `azure.yaml`, the configured `infra.path`, source projects, deployment scripts, `.gitignore`, and pipeline definitions.
|
||||
2. Read `azure.yaml` before inferring services or the IaC provider.
|
||||
3. Identify whether the task is to create, migrate, review, deploy, or troubleshoot.
|
||||
4. Identify the active environment only when an environment-specific operation is required.
|
||||
5. Read the relevant reference:
|
||||
- Repository layout or `azure.yaml`: [references/project-structure.md](references/project-structure.md)
|
||||
- Bicep, Terraform, parameters, outputs, or environments: [references/iac-and-environments.md](references/iac-and-environments.md)
|
||||
- Secrets, hooks, CI/CD, deployment, or troubleshooting: [references/security-cicd-operations.md](references/security-cicd-operations.md)
|
||||
- Product details that may have changed: [references/official-docs.md](references/official-docs.md)
|
||||
|
||||
Do not assume the default `infra` path, the default Bicep provider, or a single service when `azure.yaml` says otherwise.
|
||||
|
||||
## Apply safety guardrails
|
||||
|
||||
- Never commit `.azure`, environment `.env` files, credentials, deployment outputs containing secrets, local Terraform state, or generated deployment artifacts.
|
||||
- Never put literal secrets in `azure.yaml`, IaC parameter files, hooks, source control, command arguments that will be logged, or IaC outputs.
|
||||
- Prefer managed identities and RBAC. Use Key Vault references and `azd env set-secret` when a secret is unavoidable.
|
||||
- Before a command that can create, modify, or delete Azure resources, confirm the target environment, subscription, tenant, region, and expected scope.
|
||||
- Treat an explicit user request to deploy, provision, destroy, or configure a pipeline as approval for that named action. Otherwise, ask before running `azd up`, `azd provision`, `azd deploy`, `azd down`, or `azd pipeline config`.
|
||||
- Do not replace Bicep with Terraform, Terraform with Bicep, or an established hosting service unless the user requests that architectural change.
|
||||
- Preserve resources and state owned outside the current `azd` project.
|
||||
|
||||
## Use these defaults
|
||||
|
||||
| Concern | Preferred default |
|
||||
| --- | --- |
|
||||
| Project manifest | One `azure.yaml` at the repository root |
|
||||
| Application code | `src/<service-name>` per independently deployable service |
|
||||
| Infrastructure | `infra` with a thin entry point and reusable modules |
|
||||
| IaC provider | Bicep unless the repository or user chooses Terraform |
|
||||
| Deployment environments | Separate named environments for dev, test, staging, and production |
|
||||
| Local AZD state | `.azure/<environment-name>` and excluded from source control |
|
||||
| Shared environment state | AZD remote environments backed by Azure Blob Storage |
|
||||
| Secrets | Managed identity/RBAC first, then Key Vault references |
|
||||
| Automation scripts | Short, idempotent scripts under `scripts/azd` |
|
||||
| CI authentication | Workload identity federation/OIDC where supported |
|
||||
| Routine development | `azd up` for simple workflows; separate phases for controlled workflows |
|
||||
|
||||
## Implementation workflow
|
||||
|
||||
### 1. Model the application
|
||||
|
||||
- Define one `services` entry for each independently deployable component.
|
||||
- Keep service keys stable because they participate in resource discovery and deployment.
|
||||
- Map each service to its actual `project`, `language`, and `host`.
|
||||
- Keep shared infrastructure in IaC rather than inventing a fake deployable service.
|
||||
- Declare dependencies with supported `azure.yaml` fields instead of relying on file order.
|
||||
|
||||
### 2. Model infrastructure
|
||||
|
||||
- Keep `main.bicep` or `main.tf` as the orchestration entry point.
|
||||
- Split reusable or independently understandable infrastructure into modules.
|
||||
- Parameterize environment-specific values; do not fork the IaC tree per environment.
|
||||
- Output only stable, nonsecret values required by deployment or application configuration.
|
||||
- Use deterministic naming and consistent tags that include the project and environment.
|
||||
- Add role assignments to identities rather than distributing service keys.
|
||||
- Use infrastructure layers only when separate scopes or lifecycle dependencies justify them.
|
||||
|
||||
### 3. Model environments
|
||||
|
||||
- Use predictable names such as `<project>-dev` for shared environments and `<alias>-dev` for personal environments.
|
||||
- Use `azd env set`, `azd env unset`, and `azd env set-secret` rather than editing `.env` directly.
|
||||
- Use `-e` or `--environment` in scripts and automation so the target is explicit.
|
||||
- Use `azd env refresh` to synchronize deployment outputs after another actor changes an environment.
|
||||
- Configure AZD remote state when a team shares environment state.
|
||||
|
||||
### 4. Add hooks only for lifecycle gaps
|
||||
|
||||
- Prefer declarative IaC and native service configuration over hooks.
|
||||
- Use root hooks for project-wide behavior and service hooks for service-specific behavior.
|
||||
- Keep nontrivial hook logic in versioned scripts under `scripts/azd`.
|
||||
- Set `shell` explicitly. Provide `windows` and `posix` variants when necessary.
|
||||
- Make hooks idempotent, noninteractive in CI, and fail on errors unless failure is intentionally nonblocking.
|
||||
- Test a hook independently with `azd hooks run <hook-name>`.
|
||||
|
||||
### 5. Build CI/CD deliberately
|
||||
|
||||
- Keep the pipeline definition with the template and review generated changes from `azd pipeline config`.
|
||||
- Use short-lived federated credentials where the provider supports them.
|
||||
- Run tests and IaC validation before provisioning.
|
||||
- Use explicit environments and `--no-prompt` in automation.
|
||||
- Add protected production environments and approval gates.
|
||||
- For Terraform, configure protected remote state before pipeline setup and account for current AZD authentication limitations.
|
||||
|
||||
## Validate before finishing
|
||||
|
||||
Run only checks applicable to the repository:
|
||||
|
||||
```text
|
||||
Application: existing formatter, linter, type-check, build, and tests
|
||||
Bicep: az bicep build --file infra/main.bicep
|
||||
Terraform: terraform fmt -check -recursive
|
||||
terraform init -backend=false
|
||||
terraform validate
|
||||
AZD hooks: azd hooks run <hook-name>
|
||||
Packaging: azd package
|
||||
```
|
||||
|
||||
For a Bicep what-if or Terraform plan, choose the correct deployment scope and environment. These checks can authenticate to Azure or read remote state, so follow the safety guardrails.
|
||||
|
||||
Verify that:
|
||||
|
||||
- `azure.yaml` paths exist and service settings match the source projects.
|
||||
- The IaC entry point and provider agree with `azure.yaml`.
|
||||
- Required deployment outputs match the variables consumed by services, hooks, and pipelines.
|
||||
- `.gitignore` excludes `.azure`, secrets, local state, and generated artifacts.
|
||||
- No secret appears in tracked content or command output.
|
||||
- Documentation explains prerequisites, environment creation, deployment, verification, and cleanup.
|
||||
|
||||
## Report the result
|
||||
|
||||
State:
|
||||
|
||||
- The files and behavior changed.
|
||||
- The IaC provider and environment assumptions.
|
||||
- The checks performed.
|
||||
- Any cloud-changing command deliberately not run.
|
||||
- Any beta or preview feature the solution relies on.
|
||||
|
||||
Do not claim deployment success unless the target environment was actually deployed and verified.
|
||||
@@ -0,0 +1,33 @@
|
||||
# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-dev/main/schemas/v1.0/azure.yaml.json
|
||||
name: sample-app
|
||||
|
||||
infra:
|
||||
provider: bicep
|
||||
path: ./infra
|
||||
module: main
|
||||
|
||||
services:
|
||||
api:
|
||||
project: ./src/api
|
||||
language: ts
|
||||
host: appservice
|
||||
web:
|
||||
project: ./src/web
|
||||
dist: dist
|
||||
language: ts
|
||||
host: staticwebapp
|
||||
|
||||
# Add hooks only when the default lifecycle cannot express the requirement.
|
||||
# Keep nontrivial commands in scripts/azd and provide both OS variants.
|
||||
hooks:
|
||||
preprovision:
|
||||
windows:
|
||||
shell: pwsh
|
||||
run: ./scripts/azd/validate.ps1
|
||||
interactive: false
|
||||
continueOnError: false
|
||||
posix:
|
||||
shell: sh
|
||||
run: ./scripts/azd/validate.sh
|
||||
interactive: false
|
||||
continueOnError: false
|
||||
@@ -0,0 +1,212 @@
|
||||
# Infrastructure as code and environments
|
||||
|
||||
## Choose the provider deliberately
|
||||
|
||||
### Bicep
|
||||
|
||||
Use Bicep when:
|
||||
|
||||
- The project is Azure-only.
|
||||
- Native Azure resource coverage and immediate API support matter.
|
||||
- The team wants a stateless deployment model.
|
||||
- Azure Verified Modules cover common resource patterns.
|
||||
|
||||
Bicep is AZD's default IaC provider.
|
||||
|
||||
### Terraform
|
||||
|
||||
Use Terraform when:
|
||||
|
||||
- The repository already uses Terraform.
|
||||
- The team has established Terraform module, state, policy, and review practices.
|
||||
- Cross-provider infrastructure is a real requirement.
|
||||
|
||||
Current Microsoft documentation marks AZD Terraform support as beta. Surface this constraint and do not migrate a project to Terraform merely for familiarity.
|
||||
|
||||
## Bicep structure
|
||||
|
||||
Keep `main.bicep` as an orchestration layer:
|
||||
|
||||
```text
|
||||
infra/
|
||||
|-- main.bicep
|
||||
|-- main.parameters.json
|
||||
|-- modules/
|
||||
| |-- core/
|
||||
| |-- data/
|
||||
| |-- identity/
|
||||
| |-- observability/
|
||||
| |-- services/
|
||||
```
|
||||
|
||||
### Bicep practices
|
||||
|
||||
- Declare the deployment `targetScope` intentionally.
|
||||
- Use modules for cohesive capabilities and repeated patterns.
|
||||
- Prefer Azure Verified Modules when they meet the requirement and the team accepts their versioning model.
|
||||
- Pin module versions; review upgrades rather than floating automatically.
|
||||
- Add descriptions and validation decorators to parameters.
|
||||
- Pass parameters down through modules instead of reading AZD environment variables inside every module.
|
||||
- Use deterministic names that respect each resource type's length and character constraints.
|
||||
- Use `uniqueString` with stable scope inputs where global uniqueness is required.
|
||||
- Apply consistent project, environment, owner, and cost tags when policy allows.
|
||||
- Use managed identities and narrowly scoped role assignments.
|
||||
- Avoid keys and connection strings when identity-based access is available.
|
||||
- Output resource IDs, names, and endpoints required by later phases.
|
||||
- Never output secret values. Deployment outputs are copied into the AZD environment.
|
||||
|
||||
### Parameter flow
|
||||
|
||||
Use `main.parameters.json` to map AZD environment values into Bicep:
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
|
||||
"contentVersion": "1.0.0.0",
|
||||
"parameters": {
|
||||
"environmentName": {
|
||||
"value": "${AZURE_ENV_NAME}"
|
||||
},
|
||||
"location": {
|
||||
"value": "${AZURE_LOCATION}"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Match those values in the entry point:
|
||||
|
||||
```bicep
|
||||
@description('Stable name of the AZD deployment environment.')
|
||||
@minLength(1)
|
||||
param environmentName string
|
||||
|
||||
@description('Primary Azure region for this deployment.')
|
||||
param location string
|
||||
```
|
||||
|
||||
Use outputs as the contract between provisioning and later AZD phases:
|
||||
|
||||
```bicep
|
||||
output SERVICE_API_ENDPOINT_URL string = api.outputs.endpoint
|
||||
```
|
||||
|
||||
Choose stable output names because services, hooks, and pipelines may consume them as environment variables.
|
||||
|
||||
When using AZD environment secrets with Bicep:
|
||||
|
||||
- Mark the Bicep input with `@secure()`.
|
||||
- Map the AZD secret reference through `main.parameters.json`.
|
||||
- Do not output the secure value.
|
||||
- Be aware that current AZD documentation says environment secrets are not supported with `.bicepparam` files.
|
||||
|
||||
## Terraform structure and state
|
||||
|
||||
### Terraform practices
|
||||
|
||||
- Set `infra.provider: terraform` explicitly in `azure.yaml`.
|
||||
- Keep all AZD-managed `.tf` files under the configured infrastructure path.
|
||||
- Pin Terraform and provider versions and commit the dependency lock file.
|
||||
- Use modules with clear inputs and outputs.
|
||||
- Mark sensitive variables and outputs as `sensitive`, but remember that sensitive values can still exist in state.
|
||||
- Do not commit `.tfstate`, plan files, crash logs, or provider credentials.
|
||||
- Avoid splitting ownership of the same Azure resource between AZD and an unrelated Terraform root module.
|
||||
|
||||
### Authentication
|
||||
|
||||
Terraform's Azure provider uses Azure CLI authentication by default and does not use the AZD credential cache. Prefer the documented single-sign-in configuration:
|
||||
|
||||
```text
|
||||
azd config set auth.useAzCliAuth true
|
||||
az login
|
||||
```
|
||||
|
||||
Otherwise, both `azd auth login` and `az login` are required.
|
||||
|
||||
### Remote state
|
||||
|
||||
Configure a protected remote backend before `azd pipeline config` or collaborative deployments:
|
||||
|
||||
- Use a dedicated storage account and private container where appropriate.
|
||||
- Restrict access with RBAC and network controls.
|
||||
- Enable platform protections such as versioning, soft delete, and resource locks according to organizational policy.
|
||||
- Use a distinct state key per project and environment.
|
||||
- Treat state as sensitive data.
|
||||
- Do not store backend access keys in source control.
|
||||
|
||||
AZD reads Terraform backend settings from `infra/provider.conf.json` when configured according to the official Terraform integration.
|
||||
|
||||
## Environment strategy
|
||||
|
||||
AZD stores local environment state under:
|
||||
|
||||
```text
|
||||
.azure/
|
||||
|-- config.json
|
||||
|-- <environment-name>/
|
||||
|-- .env
|
||||
|-- config.json
|
||||
```
|
||||
|
||||
The entire `.azure` directory should remain out of source control.
|
||||
|
||||
### Naming
|
||||
|
||||
Use names that make ownership and lifecycle clear:
|
||||
|
||||
- Shared: `<project>-dev`, `<project>-test`, `<project>-prod`
|
||||
- Personal: `<alias>-<purpose>` or `<alias>-dev`
|
||||
- Ephemeral: `<project>-pr-<number>` when automation also guarantees cleanup
|
||||
|
||||
Keep the name short enough to support resources with restrictive naming limits.
|
||||
|
||||
### Management
|
||||
|
||||
Use AZD commands rather than manual file editing:
|
||||
|
||||
```text
|
||||
azd env new <name>
|
||||
azd env list
|
||||
azd env select <name>
|
||||
azd env set <key> <value>
|
||||
azd env get-value <key>
|
||||
azd env unset <key>
|
||||
azd env refresh
|
||||
```
|
||||
|
||||
In automation and potentially destructive operations, target the environment explicitly:
|
||||
|
||||
```text
|
||||
azd provision -e <environment> --no-prompt
|
||||
azd deploy -e <environment> --no-prompt
|
||||
```
|
||||
|
||||
### Configuration rules
|
||||
|
||||
- Keep one IaC codebase and vary behavior through parameters.
|
||||
- Keep nonsecret defaults in reviewed configuration or IaC, not in committed `.azure` files.
|
||||
- Use `azd env set` for deployment-specific nonsecret settings.
|
||||
- Allow IaC outputs to populate computed resource names and endpoints.
|
||||
- Avoid environment-name conditionals scattered across modules. Prefer explicit feature or SKU parameters.
|
||||
- Use `azd env refresh` after another actor changes deployment outputs.
|
||||
- Do not assume the currently selected environment in scripts.
|
||||
|
||||
## Shared and remote environments
|
||||
|
||||
Configure `state.remote` when teammates or automation need a shared AZD environment:
|
||||
|
||||
```yaml
|
||||
state:
|
||||
remote:
|
||||
backend: AzureBlobStorage
|
||||
config:
|
||||
accountName: <storage-account-name>
|
||||
containerName: <project-container-name>
|
||||
```
|
||||
|
||||
Remote AZD state synchronizes `.env` and AZD `config.json`; it is separate from Terraform remote state. A Terraform project that collaborates through AZD can require both:
|
||||
|
||||
- AZD remote state for environment configuration.
|
||||
- Terraform remote state for managed infrastructure state.
|
||||
|
||||
Protect both stores with least-privilege RBAC and appropriate data-protection settings.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Official references
|
||||
|
||||
Use Microsoft Learn as the source of truth for AZD behavior and schema details. These references were reviewed on 2026-08-05.
|
||||
|
||||
## Core concepts and structure
|
||||
|
||||
- [Azure Developer CLI documentation](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/)
|
||||
- [What is the Azure Developer CLI?](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/overview)
|
||||
- [Azure Developer CLI templates overview](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/azd-templates)
|
||||
- [Create Azure Developer CLI templates overview](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/make-azd-compatible)
|
||||
- [Azure Developer CLI schema reference](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/azd-schema)
|
||||
- [`azure.yaml` JSON schema](https://aka.ms/azure.yaml.json)
|
||||
- [Explore the `azd up` workflow](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/azd-up-workflow)
|
||||
- [Full-stack deployment with Azure Developer CLI](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/full-stack-deployment)
|
||||
|
||||
## Infrastructure as code
|
||||
|
||||
- [Use Terraform as an infrastructure as code tool for Azure Developer CLI](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/use-terraform-for-azd)
|
||||
- [Azure Verified Modules](https://azure.github.io/Azure-Verified-Modules/)
|
||||
- [Bicep documentation](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/)
|
||||
- [Terraform on Azure documentation](https://learn.microsoft.com/en-us/azure/developer/terraform/)
|
||||
|
||||
## Environments and secrets
|
||||
|
||||
- [Azure Developer CLI environments overview](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/environments-overview)
|
||||
- [Work with Azure Developer CLI environments](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/work-with-environments)
|
||||
- [Work with Azure Developer CLI environment variables](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/manage-environment-variables)
|
||||
- [Remote environments support](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/remote-environments-support)
|
||||
- [Use environment secrets with Azure Developer CLI](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/environment-secrets)
|
||||
|
||||
## Hooks, pipelines, and operations
|
||||
|
||||
- [Customize Azure Developer CLI workflows using hooks](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/azd-extensibility)
|
||||
- [Explore Azure Developer CLI support for CI/CD pipelines](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/configure-devops-pipeline)
|
||||
- [Create a GitHub Actions CI/CD pipeline using Azure Developer CLI](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/pipeline-github-actions)
|
||||
- [Advanced pipeline features and configurations](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/pipeline-advanced-features)
|
||||
- [Azure Developer CLI command reference](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/reference)
|
||||
- [Troubleshoot Azure Developer CLI](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/troubleshoot)
|
||||
|
||||
## Skill format
|
||||
|
||||
- [Adding agent skills for GitHub Copilot](https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-cloud-agent/add-skills)
|
||||
- [About agent skills](https://docs.github.com/en/copilot/concepts/agents/about-agent-skills)
|
||||
|
||||
When a field, command flag, host type, preview status, or authentication behavior is uncertain, consult the relevant current reference before changing code. Do not rely on remembered syntax for fast-moving preview features.
|
||||
@@ -0,0 +1,134 @@
|
||||
# Project structure and `azure.yaml`
|
||||
|
||||
## Recommended repository layout
|
||||
|
||||
Use this as a default, not as a reason to reorganize an already coherent repository:
|
||||
|
||||
```text
|
||||
.
|
||||
|-- .azure/ # Generated local AZD environment state; ignored
|
||||
|-- .devcontainer/ # Optional reproducible developer environment
|
||||
|-- .github/
|
||||
| |-- workflows/
|
||||
| |-- azure-dev.yml # Optional GitHub Actions pipeline
|
||||
|-- infra/
|
||||
| |-- main.bicep # Bicep orchestration entry point
|
||||
| |-- main.parameters.json # AZD environment-to-Bicep parameter mapping
|
||||
| |-- modules/
|
||||
| |-- core/ # Shared platform resources
|
||||
| |-- app/ # Application-specific resources
|
||||
|-- scripts/
|
||||
| |-- azd/ # Hook and deployment helper scripts
|
||||
|-- src/
|
||||
| |-- api/ # Independently deployable service
|
||||
| |-- web/ # Independently deployable service
|
||||
|-- tests/
|
||||
|-- .gitignore
|
||||
|-- azure.yaml
|
||||
|-- README.md
|
||||
```
|
||||
|
||||
For Terraform, use a conventional `infra` layout:
|
||||
|
||||
```text
|
||||
infra/
|
||||
|-- main.tf
|
||||
|-- providers.tf
|
||||
|-- variables.tf
|
||||
|-- outputs.tf
|
||||
|-- provider.conf.json # AZD remote backend configuration, when used
|
||||
|-- modules/
|
||||
```
|
||||
|
||||
### Structure rules
|
||||
|
||||
- Place `azure.yaml` at the project root.
|
||||
- Keep application source independent from deployment assets.
|
||||
- Keep the IaC entry point small; move resource details into modules.
|
||||
- Organize modules by responsibility or lifecycle, not one arbitrary file per resource.
|
||||
- Keep hook scripts outside `infra` unless a script belongs exclusively to an infrastructure layer.
|
||||
- Avoid committed environment-specific source trees such as `infra/dev`, `infra/test`, and `infra/prod`. Use parameters.
|
||||
- Keep tests near their normal language conventions; do not move them merely to fit this example.
|
||||
- Include `.devcontainer` only when it is maintained and tested.
|
||||
|
||||
## `azure.yaml` baseline
|
||||
|
||||
Add the schema directive for editor validation:
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-dev/main/schemas/v1.0/azure.yaml.json
|
||||
name: sample-app
|
||||
|
||||
infra:
|
||||
provider: bicep
|
||||
path: ./infra
|
||||
module: main
|
||||
|
||||
services:
|
||||
api:
|
||||
project: ./src/api
|
||||
language: ts
|
||||
host: appservice
|
||||
web:
|
||||
project: ./src/web
|
||||
dist: dist
|
||||
language: ts
|
||||
host: staticwebapp
|
||||
```
|
||||
|
||||
The explicit `infra` block is useful when clarity matters, even though Bicep, `infra`, and `main` are defaults.
|
||||
|
||||
## Manifest design checklist
|
||||
|
||||
### Top-level configuration
|
||||
|
||||
- `name` is lowercase, starts and ends with an alphanumeric character, and uses only alphanumerics and hyphens.
|
||||
- `metadata.template` identifies the source template and version when the repository is distributed as a template.
|
||||
- `infra.provider`, `infra.path`, and `infra.module` match the actual repository.
|
||||
- `requiredVersions` is used when the project depends on a minimum AZD or extension version.
|
||||
- `workflows` overrides defaults only when deployment ordering genuinely requires it.
|
||||
- `state.remote` is configured at project scope when teams share AZD environments.
|
||||
|
||||
### Services
|
||||
|
||||
- A service represents deployable application code, not a database, Key Vault, or other shared resource.
|
||||
- Service names are short, meaningful, and stable.
|
||||
- `project` points to the service root and uses a relative path.
|
||||
- `language`, `host`, `dist`, container, and remote-build settings match how the service is built.
|
||||
- A Container Apps service uses either `project` or `image`, not both.
|
||||
- `resourceName` is set only when standard AZD discovery through the `azd-service-name` tag is unavailable or intentionally bypassed.
|
||||
- Dependencies use supported `uses` relationships rather than implicit assumptions.
|
||||
- Environment variables use substitutions or IaC outputs rather than hard-coded environment values.
|
||||
|
||||
### Resources and infrastructure
|
||||
|
||||
- Shared Azure resources stay in IaC.
|
||||
- Service modules and AZD service names align so resource discovery is predictable.
|
||||
- Custom resource group names include environment identity and comply with Azure naming constraints.
|
||||
- Infrastructure layers are reserved for independently provisioned units, different scopes, or hook-mediated dependencies.
|
||||
- Layer dependencies are explicit with `dependsOn` when AZD cannot infer them.
|
||||
|
||||
### Pipelines and hooks
|
||||
|
||||
- `pipeline.variables` contains nonsecret configuration.
|
||||
- `pipeline.secrets` is used only when the pipeline must store the resolved value instead of a Key Vault reference.
|
||||
- Root hooks handle project-wide work; service hooks handle one service.
|
||||
- Hook scripts use explicit shells and portable paths.
|
||||
- Hooks do not duplicate application tests or declarative IaC behavior.
|
||||
|
||||
## README requirements for a reusable AZD project
|
||||
|
||||
Document:
|
||||
|
||||
1. Architecture and deployed Azure services.
|
||||
2. Local prerequisites, including AZD and provider-specific tools.
|
||||
3. Authentication requirements.
|
||||
4. How to create or select an environment.
|
||||
5. Required nonsecret variables and how to set them.
|
||||
6. How secrets are supplied without exposing their values.
|
||||
7. How to run, test, provision, deploy, monitor, and troubleshoot.
|
||||
8. Expected cost-bearing resources.
|
||||
9. How to clean up safely.
|
||||
10. Any beta or preview dependencies, including Terraform or pipeline features when applicable.
|
||||
|
||||
Do not put actual subscription IDs, tenant IDs, secret names that reveal sensitive systems, or production endpoints in reusable documentation.
|
||||
@@ -0,0 +1,185 @@
|
||||
# Security, hooks, CI/CD, and operations
|
||||
|
||||
## Identity and secret handling
|
||||
|
||||
Use this order of preference:
|
||||
|
||||
1. Managed identity with least-privilege RBAC.
|
||||
2. Workload identity federation for CI/CD.
|
||||
3. Key Vault reference through `azd env set-secret`.
|
||||
4. Short-lived secret material only when no identity-based option exists.
|
||||
|
||||
Never:
|
||||
|
||||
- Store a plaintext secret in `.azure/<environment>/.env`.
|
||||
- Commit environment files, credentials, certificates, or Terraform state.
|
||||
- Put secrets in IaC outputs.
|
||||
- Echo environment values indiscriminately in hooks or pipelines.
|
||||
- Pass a secret directly on a command line when the shell or CI system can record it.
|
||||
- Grant broad subscription roles when resource-group or resource scope is enough.
|
||||
|
||||
`azd env set-secret <name>` stores a Key Vault reference in the AZD environment. Resolve it only where needed:
|
||||
|
||||
- Map it to an `@secure()` Bicep parameter.
|
||||
- Use a hook `secrets` mapping for a hook process.
|
||||
- Choose between a pipeline variable containing the Key Vault reference or a pipeline secret containing the resolved value.
|
||||
|
||||
Prefer the reference approach when the pipeline identity can read Key Vault because rotation does not require republishing a resolved pipeline secret.
|
||||
|
||||
## Hooks
|
||||
|
||||
Use hooks for validation, generated runtime configuration, data preparation, smoke checks, or lifecycle coordination that IaC and native AZD behavior cannot express.
|
||||
|
||||
### Hook rules
|
||||
|
||||
- Prefer external scripts over long inline commands.
|
||||
- Store scripts under `scripts/azd`.
|
||||
- Set `shell: sh` or `shell: pwsh` explicitly.
|
||||
- Supply `windows` and `posix` implementations when syntax differs.
|
||||
- Use paths relative to the documented hook working directory.
|
||||
- Make scripts idempotent and safe to rerun.
|
||||
- Keep `continueOnError` false unless the operation is observability-only or genuinely optional.
|
||||
- Use noninteractive behavior in CI.
|
||||
- Do not install unpinned dependencies on every run if a reproducible tool setup can do it once.
|
||||
- Do not log secret values or all environment variables.
|
||||
- Test with `azd hooks run <hook-name>` before coupling the hook to a complete deployment.
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
hooks:
|
||||
preprovision:
|
||||
windows:
|
||||
shell: pwsh
|
||||
run: ./scripts/azd/validate.ps1
|
||||
interactive: false
|
||||
continueOnError: false
|
||||
posix:
|
||||
shell: sh
|
||||
run: ./scripts/azd/validate.sh
|
||||
interactive: false
|
||||
continueOnError: false
|
||||
```
|
||||
|
||||
Use root hooks for the whole project. Put service-specific hooks under that service's `azure.yaml` entry.
|
||||
|
||||
## Deployment workflow
|
||||
|
||||
The normal AZD lifecycle is:
|
||||
|
||||
1. Package application artifacts.
|
||||
2. Provision or update infrastructure.
|
||||
3. Deploy application artifacts.
|
||||
|
||||
`azd up` is the convenient combined workflow and is appropriate for routine development and simple deployments.
|
||||
|
||||
Use separate commands when:
|
||||
|
||||
- Infrastructure review or approval must happen before deployment.
|
||||
- The application is redeployed frequently without infrastructure changes.
|
||||
- Troubleshooting requires isolating package, provision, or deploy failures.
|
||||
- A complex dependency requires a custom order.
|
||||
|
||||
```text
|
||||
azd package
|
||||
azd provision -e <environment>
|
||||
azd deploy -e <environment>
|
||||
```
|
||||
|
||||
Customize `workflows.up.steps` only when a real dependency requires another order, such as provisioning before a build that needs a generated endpoint. Do not customize the workflow merely to mirror a pipeline's naming conventions.
|
||||
|
||||
## Full-stack and multi-service dependencies
|
||||
|
||||
- Map service dependencies before implementation.
|
||||
- Let Bicep or Terraform handle one-directional infrastructure dependencies.
|
||||
- Use provisioning outputs for endpoints and names needed during deployment.
|
||||
- Use runtime configuration, such as Azure App Configuration or a generated config file, when settings must change without rebuilding.
|
||||
- Avoid circular compile-time dependencies between front-end and back-end services.
|
||||
- Use hooks or a custom workflow only when outputs and runtime configuration cannot resolve the dependency.
|
||||
- Test the strategy independently in development, test, and production-like environments.
|
||||
|
||||
## CI/CD
|
||||
|
||||
### Pipeline design
|
||||
|
||||
A robust pipeline separates:
|
||||
|
||||
1. Application format, lint, build, and tests.
|
||||
2. IaC format and static validation.
|
||||
3. What-if or plan review at the correct scope.
|
||||
4. Provisioning with an explicit AZD environment.
|
||||
5. Deployment.
|
||||
6. Smoke or health verification.
|
||||
7. Production approval and rollback/cleanup procedures.
|
||||
|
||||
Use:
|
||||
|
||||
- `--no-prompt` in automation.
|
||||
- A fixed `-e` or `--environment`.
|
||||
- Protected environments and required reviewers for production.
|
||||
- Concurrency controls to prevent simultaneous writes to one environment.
|
||||
- Least-privilege identities scoped to the target environment.
|
||||
- Pinned action and tool versions with a managed update process.
|
||||
|
||||
### `azd pipeline config`
|
||||
|
||||
Current Microsoft documentation marks `azd pipeline config` as beta. Before running it:
|
||||
|
||||
- Review the pipeline definition bundled with the template.
|
||||
- Confirm repository, organization, environment, subscription, and authentication mode.
|
||||
- Expect repository, identity, variable, secret, commit, push, and pipeline side effects.
|
||||
- Review generated workflow and permission changes before production use.
|
||||
- Rerun it when `pipeline.variables` or `pipeline.secrets` changes.
|
||||
|
||||
For GitHub Actions, AZD configures OIDC/federated credentials by default for supported scenarios. Current documentation says the AZD Terraform pipeline flow does not support OIDC, so evaluate the authentication tradeoff explicitly rather than silently falling back to a long-lived credential.
|
||||
|
||||
For Terraform, configure protected remote state before pipeline setup.
|
||||
|
||||
## Validation and preview
|
||||
|
||||
Run local checks before Azure-changing commands:
|
||||
|
||||
### Bicep
|
||||
|
||||
```text
|
||||
az bicep build --file infra/main.bicep
|
||||
```
|
||||
|
||||
Use an Azure deployment what-if at the scope declared by the template. Do not assume resource-group scope.
|
||||
|
||||
### Terraform
|
||||
|
||||
```text
|
||||
terraform fmt -check -recursive
|
||||
terraform init -backend=false
|
||||
terraform validate
|
||||
```
|
||||
|
||||
Use `terraform plan` only after confirming the backend, workspace/state key, variables, and Azure identity.
|
||||
|
||||
### AZD and application
|
||||
|
||||
- Run existing application checks.
|
||||
- Run relevant hooks independently.
|
||||
- Run `azd package` to verify service paths and packaging.
|
||||
- Confirm IaC outputs match variables consumed during deployment.
|
||||
- Inspect the environment name before provision, deploy, or down.
|
||||
|
||||
## Troubleshooting sequence
|
||||
|
||||
1. Identify whether the failure is package, provision, deploy, hook, authentication, or resource discovery.
|
||||
2. Re-run the smallest failing phase rather than `azd up`.
|
||||
3. Check the selected environment and expected subscription, tenant, and region.
|
||||
4. Check `azure.yaml` paths, provider, service names, host types, and resource discovery tags.
|
||||
5. Refresh environment outputs with `azd env refresh` when Azure state changed elsewhere.
|
||||
6. For Terraform, verify both AZD and Azure CLI authentication and the correct remote state.
|
||||
7. For hooks, run the hook directly and verify its shell, working directory, and environment dependencies.
|
||||
8. Use debug logging only when needed, and redact sensitive values before sharing logs.
|
||||
|
||||
## Cleanup
|
||||
|
||||
- Confirm the exact environment before `azd down`.
|
||||
- Explain that cleanup can delete data-bearing resources.
|
||||
- Preserve externally owned or shared resources.
|
||||
- For ephemeral environments, automate cleanup and include a fallback for failed pipeline runs.
|
||||
- Verify deletion rather than assuming command success.
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
name: build-evidence-map
|
||||
description: 'Build an auditable evidence map for a contested technical choice, research synthesis, proposal review, or consequential decision. Use when Copilot must preserve supporting, contradicting, qualifying, and missing evidence with exact source regions instead of collapsing disagreement into prose.'
|
||||
---
|
||||
|
||||
# Build Evidence Map
|
||||
|
||||
Turn one contested question into a portable decision artifact that shows what
|
||||
supports the current position, what pushes against it, and what remains unknown.
|
||||
Do not use a graph to decorate an answer that has not been sourced.
|
||||
|
||||
For a simple factual claim or a general fact-checking request, use a verification
|
||||
workflow such as `doublecheck` instead. Use this skill when the relationships
|
||||
between evidence, intermediate claims, trade-offs, and missing facts matter.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Frame one decision.** Write one falsifiable question and one provisional
|
||||
position. Narrow the question until a reader can identify what action or
|
||||
belief the map is testing.
|
||||
2. **Collect bounded source regions.** Prefer direct observations and primary
|
||||
sources. Record the URL or absolute local path, publisher, publication date,
|
||||
retrieval date, section/page/line/timestamp locator, and a short checkable
|
||||
excerpt. Read [references/evidence-ladder.md](references/evidence-ladder.md)
|
||||
when source quality is disputed.
|
||||
3. **Atomize the reasoning.** Create only four node types:
|
||||
- `position`: the single current verdict;
|
||||
- `claim`: an intermediate proposition;
|
||||
- `evidence`: a faithful statement of one source region;
|
||||
- `unknown`: a specific missing fact that could change the verdict.
|
||||
4. **Type every edge.** Use `supports`, `contradicts`, `qualifies`, or
|
||||
`missing`. Add a plain-language note explaining why the source node bears on
|
||||
the target. Topical similarity is not support. Different scope, date, or
|
||||
population is not automatically a contradiction.
|
||||
5. **Preserve counterevidence.** Do not delete contrary evidence because the
|
||||
provisional verdict survives it. Represent scope differences with
|
||||
`qualifies` edges.
|
||||
6. **Express uncertainty structurally.** Do not invent confidence percentages.
|
||||
Add an `unknown`, narrow the position, or qualify a claim.
|
||||
7. **Write UTF-8 JSON** with a `.doubt.json` suffix. Follow
|
||||
[references/map-schema.md](references/map-schema.md). Keep IDs short,
|
||||
stable, and semantic.
|
||||
8. **Validate fail-closed.** Resolve
|
||||
`scripts/validate.mjs` relative to this `SKILL.md`, then run it with Node.js
|
||||
18 or newer:
|
||||
|
||||
```bash
|
||||
node <skill-directory>/scripts/validate.mjs decision.doubt.json
|
||||
```
|
||||
|
||||
The bundled validator uses only Node.js built-ins and does not require npm or
|
||||
network access. Fix every finding before reporting success. Only say the map
|
||||
is valid when the command exits `0` and prints `VALID` followed by a
|
||||
64-character receipt. A file hash, node count, JSON parse, or manual schema
|
||||
review is not a Doubt receipt. If deterministic validation cannot run, report
|
||||
that block instead of inventing success.
|
||||
|
||||
Render the validated map only when the user has already installed
|
||||
`doubt-ai@0.8.0`; do not install or execute a remote package implicitly:
|
||||
|
||||
```bash
|
||||
doubt map decision.doubt.json --out decision.html
|
||||
```
|
||||
9. **Verify source snapshots only with explicit network permission.** The
|
||||
following command retrieves each recorded HTTP(S) source and fails closed if
|
||||
an excerpt cannot be matched:
|
||||
|
||||
```bash
|
||||
doubt verify decision.doubt.json \
|
||||
--out decision.verified.doubt.json
|
||||
```
|
||||
|
||||
Never run this command implicitly. Local file verification does not use the
|
||||
network. Do not write a `verification` object by hand or hide a mismatch.
|
||||
10. **Inspect the deliverable.** Confirm that the question, verdict,
|
||||
counterevidence, unknowns, edge notes, and exact source regions remain
|
||||
readable. Treat JSON as the canonical editable artifact; HTML is a
|
||||
shareable view.
|
||||
|
||||
## Quality gates
|
||||
|
||||
A finished map must satisfy all of these:
|
||||
|
||||
- exactly one `position` has incoming reasoning;
|
||||
- every evidence node names one source and participates in an edge;
|
||||
- every source is used and has dates, a bounded locator, and a substantive
|
||||
excerpt;
|
||||
- every non-position node has a directed path to the position;
|
||||
- the reasoning graph has no duplicate edges or directed cycles;
|
||||
- contrary or qualifying evidence is present when the source set contains it;
|
||||
- each decision-changing gap is an explicit `unknown` node;
|
||||
- every edge note explains support, contradiction, qualification, or absence;
|
||||
- the verdict is no broader than the evidence.
|
||||
|
||||
## Deliver the result
|
||||
|
||||
Report:
|
||||
|
||||
- the current position in one sentence;
|
||||
- the strongest counterevidence or qualification;
|
||||
- the most important unresolved unknown;
|
||||
- paths to the canonical JSON and any rendered HTML;
|
||||
- whether deterministic validation and explicit source verification ran.
|
||||
|
||||
Never describe a structurally valid map as proven true. Validation establishes
|
||||
traceability and graph integrity; source quality and inference quality still
|
||||
require human review.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Evidence ladder
|
||||
|
||||
Use the strongest evidence practical for the decision. A higher class can still
|
||||
be stale, irrelevant, or too broad for the nearby claim.
|
||||
|
||||
1. **Direct current observation** — reproduced behavior, command output,
|
||||
inspected artifact, or measured result.
|
||||
2. **Authoritative primary source** — official specification, dataset, law,
|
||||
documentation, first-party repository, or original research.
|
||||
3. **Independent corroboration** — competent sources with distinct underlying
|
||||
evidence.
|
||||
4. **Explicit inference** — a conclusion whose premises and assumptions are
|
||||
visible in the map.
|
||||
5. **Weak proxy** — related metric, benchmark, anecdote, or test that does not
|
||||
exercise the exact claim.
|
||||
6. **Unsupported assertion** — confidence, repetition, or polished language
|
||||
without evidence.
|
||||
|
||||
## Source-region test
|
||||
|
||||
Before creating an evidence node, answer:
|
||||
|
||||
- What exact sentence, table, command output, page, section, or line range is
|
||||
being relied on?
|
||||
- Does it entail the node text, or merely discuss the same subject?
|
||||
- Is its date and version appropriate for the claim?
|
||||
- Is the evidence independent, or copied from another cited source?
|
||||
- What context would reverse or narrow the interpretation?
|
||||
|
||||
If the exact region cannot be located, create an `unknown` node instead of an
|
||||
evidence node.
|
||||
|
||||
## Edge test
|
||||
|
||||
| Relation | Use when | Common counterfeit |
|
||||
| --- | --- | --- |
|
||||
| `supports` | The source increases reason to accept the target | Topical similarity |
|
||||
| `contradicts` | Both cannot hold under the same scope and conditions | Different dates or populations |
|
||||
| `qualifies` | The source narrows scope, strength, or applicability | Hiding inconvenient evidence |
|
||||
| `missing` | A specific absent fact blocks or could reverse the target | Generic “more research needed” |
|
||||
@@ -0,0 +1,101 @@
|
||||
# Evidence map schema
|
||||
|
||||
The canonical artifact is UTF-8 JSON. Use a `.doubt.json` suffix when practical.
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "Short artifact title",
|
||||
"question": "One decision-changing question?",
|
||||
"updatedAt": "YYYY-MM-DD",
|
||||
"verdict": "A provisional, evidence-bounded answer.",
|
||||
"nodes": [
|
||||
{
|
||||
"id": "current-position",
|
||||
"type": "position",
|
||||
"label": "Current position",
|
||||
"text": "The proposition represented by this node."
|
||||
},
|
||||
{
|
||||
"id": "primary-observation",
|
||||
"type": "evidence",
|
||||
"label": "Observed result",
|
||||
"text": "A faithful statement of the source region.",
|
||||
"sourceId": "source-1"
|
||||
},
|
||||
{
|
||||
"id": "missing-baseline",
|
||||
"type": "unknown",
|
||||
"label": "Missing baseline",
|
||||
"text": "The exact absent fact and why it matters."
|
||||
}
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"from": "primary-observation",
|
||||
"to": "current-position",
|
||||
"relation": "supports",
|
||||
"note": "Why the observation increases reason to accept the position."
|
||||
},
|
||||
{
|
||||
"from": "missing-baseline",
|
||||
"to": "current-position",
|
||||
"relation": "missing",
|
||||
"note": "Why this missing baseline could reverse the position."
|
||||
}
|
||||
],
|
||||
"sources": [
|
||||
{
|
||||
"id": "source-1",
|
||||
"title": "Source title",
|
||||
"url": "https://example.com/source",
|
||||
"publisher": "Publisher",
|
||||
"date": "YYYY-MM-DD",
|
||||
"retrievedAt": "YYYY-MM-DD",
|
||||
"locator": "Section: Results, p. 7, § 2.1, L12-L18, or 00:04:31",
|
||||
"excerpt": "A short, checkable excerpt or bounded source-region description."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Invariants
|
||||
|
||||
- Allowed node types: `position`, `claim`, `evidence`, `unknown`.
|
||||
- Allowed relations: `supports`, `contradicts`, `qualifies`, `missing`.
|
||||
- Exactly one `position` node is required.
|
||||
- Evidence nodes require `sourceId`.
|
||||
- Every evidence node must be the `from` side of at least one edge.
|
||||
- Every non-position node must have a directed path to the position.
|
||||
- Duplicate reasoning edges and directed cycles are rejected.
|
||||
- Every source must be used by an evidence node.
|
||||
- Every edge needs a plain-language `note`.
|
||||
- Map and source dates are real ISO calendar dates; source dates cannot be later
|
||||
than `updatedAt`.
|
||||
- Every source records `retrievedAt`. Receipts cover that value and the recorded
|
||||
excerpt, not the mutable bytes currently served by the URL.
|
||||
- Locators identify a bounded section, page, line range, or timestamp.
|
||||
- Excerpts contain 40–500 characters of varied, checkable content; repeated
|
||||
filler is invalid.
|
||||
- `confidence` fields are invalid. Use an `unknown` node or a qualified claim.
|
||||
|
||||
## Optional verification record
|
||||
|
||||
Only a successful explicit source-verification command may add this object to a
|
||||
source:
|
||||
|
||||
```json
|
||||
{
|
||||
"verification": {
|
||||
"status": "verified",
|
||||
"method": "normalized-excerpt-match",
|
||||
"checkedAt": "YYYY-MM-DDTHH:mm:ss.sssZ",
|
||||
"contentSha256": "64 lowercase hexadecimal characters",
|
||||
"excerptSha256": "64 lowercase hexadecimal characters",
|
||||
"finalUrl": "The checked URL or absolute local path",
|
||||
"locatorStatus": "matched"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`locatorStatus` may be `not-machine-checked` for page, section, and timestamp
|
||||
locators. Do not treat it as proof that the region was manually confirmed.
|
||||
@@ -0,0 +1,517 @@
|
||||
export const NODE_TYPES = new Set(["position", "claim", "evidence", "unknown"]);
|
||||
export const RELATIONS = new Set(["supports", "contradicts", "qualifies", "missing"]);
|
||||
|
||||
const ISO_DATE = /^(\d{4})-(\d{2})-(\d{2})$/;
|
||||
const ISO_UTC_TIMESTAMP = /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.\d{1,3})?Z$/;
|
||||
const LOCATOR_PATTERNS = [
|
||||
/\bp(?:age)?\.?\s*\d+(?:\s*[-–]\s*\d+)?\b/i,
|
||||
/§\s*[\p{L}\p{N}][\p{L}\p{N}._-]*/u,
|
||||
/\bL\d+(?:\s*[-–]\s*L?\d+)?\b/i,
|
||||
/\blines?\s+\d+(?:\s*[-–]\s*\d+)?\b/i,
|
||||
/\b(?:\d{1,2}:)?\d{2}:\d{2}(?:\s*[-–]\s*(?:\d{1,2}:)?\d{2}:\d{2})?\b/,
|
||||
/^(?:section|chapter|heading)\s*(?::|§)\s*\S.{1,}$/i,
|
||||
];
|
||||
|
||||
export class MapValidationError extends Error {
|
||||
constructor(findings) {
|
||||
super(`Evidence map is invalid (${findings.length} ${findings.length === 1 ? "finding" : "findings"}).`);
|
||||
this.name = "MapValidationError";
|
||||
this.findings = findings;
|
||||
}
|
||||
}
|
||||
|
||||
function canonical(value) {
|
||||
if (Array.isArray(value)) return value.map(canonical);
|
||||
if (!value || typeof value !== "object") return value;
|
||||
return Object.fromEntries(
|
||||
Object.keys(value)
|
||||
.sort()
|
||||
.map((key) => [key, canonical(value[key])]),
|
||||
);
|
||||
}
|
||||
|
||||
export function canonicalJson(map) {
|
||||
return JSON.stringify(canonical(map));
|
||||
}
|
||||
|
||||
export function receiptPayload(map, sourceSnapshots) {
|
||||
return {
|
||||
contract: "doubt-evidence-receipt-v1",
|
||||
map,
|
||||
sourceSnapshots,
|
||||
};
|
||||
}
|
||||
|
||||
function finding(path, rule, message) {
|
||||
return { path, rule, message };
|
||||
}
|
||||
|
||||
function parseIsoDate(value) {
|
||||
if (typeof value !== "string") return null;
|
||||
const match = value.match(ISO_DATE);
|
||||
if (!match) return null;
|
||||
const [, year, month, day] = match.map(Number);
|
||||
const time = Date.UTC(year, month - 1, day);
|
||||
const date = new Date(time);
|
||||
if (
|
||||
date.getUTCFullYear() !== year
|
||||
|| date.getUTCMonth() !== month - 1
|
||||
|| date.getUTCDate() !== day
|
||||
) return null;
|
||||
return time;
|
||||
}
|
||||
|
||||
function retrievalDate(value) {
|
||||
const date = parseIsoDate(value);
|
||||
if (date !== null) return date;
|
||||
if (typeof value !== "string") return null;
|
||||
const match = value.match(ISO_UTC_TIMESTAMP);
|
||||
if (!match) return null;
|
||||
const [, year, month, day, hour, minute, second] = match.map(Number);
|
||||
if (hour > 23 || minute > 59 || second > 59) return null;
|
||||
const dayValue = parseIsoDate(
|
||||
`${String(year).padStart(4, "0")}-${String(month).padStart(2, "0")}-${String(day).padStart(2, "0")}`,
|
||||
);
|
||||
return dayValue === null ? null : dayValue;
|
||||
}
|
||||
|
||||
function validUtcTimestamp(value) {
|
||||
if (typeof value !== "string") return false;
|
||||
const match = value.match(ISO_UTC_TIMESTAMP);
|
||||
if (!match) return false;
|
||||
const [, year, month, day, hour, minute, second] = match.map(Number);
|
||||
if (hour > 23 || minute > 59 || second > 59) return false;
|
||||
return parseIsoDate(
|
||||
`${String(year).padStart(4, "0")}-${String(month).padStart(2, "0")}-${String(day).padStart(2, "0")}`,
|
||||
) !== null;
|
||||
}
|
||||
|
||||
function boundedLocator(value) {
|
||||
return typeof value === "string" && LOCATOR_PATTERNS.some((pattern) => pattern.test(value.trim()));
|
||||
}
|
||||
|
||||
function sourceLocation(value) {
|
||||
return typeof value === "string" && /^(?:https?:\/\/|file:\/\/|\.\.?[\\/]|[\\/]|[A-Za-z]:[\\/])/.test(value);
|
||||
}
|
||||
|
||||
function substantiveExcerpt(value) {
|
||||
if (typeof value !== "string") return false;
|
||||
const symbols = value.toLowerCase().match(/[\p{L}\p{N}]/gu) || [];
|
||||
return new Set(symbols).size >= 6;
|
||||
}
|
||||
|
||||
function reaches(start, target, adjacency, seen = new Set()) {
|
||||
if (start === target) return true;
|
||||
if (seen.has(start)) return false;
|
||||
seen.add(start);
|
||||
return (adjacency.get(start) || []).some((next) => reaches(next, target, adjacency, seen));
|
||||
}
|
||||
|
||||
export function inspectMapContract(map) {
|
||||
const findings = [];
|
||||
if (!map || typeof map !== "object" || Array.isArray(map)) {
|
||||
return {
|
||||
findings: [finding("$", "map-type", "The map must be a JSON object.")],
|
||||
metrics: { claims: 0, contradictions: 0, evidence: 0, sources: 0, unknowns: 0 },
|
||||
receipt: null,
|
||||
valid: false,
|
||||
};
|
||||
}
|
||||
|
||||
for (const key of ["title", "question", "verdict", "updatedAt"]) {
|
||||
if (!map[key] || typeof map[key] !== "string") {
|
||||
findings.push(finding(`$.${key}`, "required-field", `${key} must be a non-empty string.`));
|
||||
}
|
||||
}
|
||||
const updatedAt = parseIsoDate(map.updatedAt);
|
||||
if (typeof map.updatedAt === "string" && updatedAt === null) {
|
||||
findings.push(
|
||||
finding("$.updatedAt", "map-date", "updatedAt must be a real calendar date in YYYY-MM-DD format."),
|
||||
);
|
||||
}
|
||||
if (!Array.isArray(map.nodes) || map.nodes.length === 0) {
|
||||
findings.push(finding("$.nodes", "required-nodes", "nodes must be a non-empty array."));
|
||||
}
|
||||
if (!Array.isArray(map.edges)) {
|
||||
findings.push(finding("$.edges", "required-edges", "edges must be an array."));
|
||||
}
|
||||
if (!Array.isArray(map.sources)) {
|
||||
findings.push(finding("$.sources", "required-sources", "sources must be an array."));
|
||||
}
|
||||
|
||||
const nodes = Array.isArray(map.nodes) ? map.nodes : [];
|
||||
const edges = Array.isArray(map.edges) ? map.edges : [];
|
||||
const sources = Array.isArray(map.sources) ? map.sources : [];
|
||||
const nodeIds = new Set();
|
||||
const sourceIds = new Set();
|
||||
|
||||
for (const [index, node] of nodes.entries()) {
|
||||
const base = `$.nodes[${index}]`;
|
||||
if (!node || typeof node !== "object" || Array.isArray(node)) {
|
||||
findings.push(finding(base, "node-type", "Each node must be an object."));
|
||||
continue;
|
||||
}
|
||||
if (!node.id || typeof node.id !== "string") {
|
||||
findings.push(finding(`${base}.id`, "node-id", "Each node needs a string id."));
|
||||
} else if (nodeIds.has(node.id)) {
|
||||
findings.push(finding(`${base}.id`, "duplicate-node", `Duplicate node id: ${node.id}.`));
|
||||
} else {
|
||||
nodeIds.add(node.id);
|
||||
}
|
||||
if (!NODE_TYPES.has(node.type)) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${base}.type`,
|
||||
"node-type",
|
||||
`Node type must be one of: ${[...NODE_TYPES].join(", ")}.`,
|
||||
),
|
||||
);
|
||||
}
|
||||
for (const key of ["label", "text"]) {
|
||||
if (!node[key] || typeof node[key] !== "string") {
|
||||
findings.push(finding(`${base}.${key}`, "node-copy", `${key} must be a non-empty string.`));
|
||||
}
|
||||
}
|
||||
if (node.confidence != null) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${base}.confidence`,
|
||||
"false-precision",
|
||||
"Confidence percentages are not supported; use an unknown or a qualified claim instead.",
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [index, source] of sources.entries()) {
|
||||
const base = `$.sources[${index}]`;
|
||||
if (!source || typeof source !== "object" || Array.isArray(source)) {
|
||||
findings.push(finding(base, "source-type", "Each source must be an object."));
|
||||
continue;
|
||||
}
|
||||
if (!source.id || typeof source.id !== "string") {
|
||||
findings.push(finding(`${base}.id`, "source-id", "Each source needs a string id."));
|
||||
} else if (sourceIds.has(source.id)) {
|
||||
findings.push(finding(`${base}.id`, "duplicate-source", `Duplicate source id: ${source.id}.`));
|
||||
} else {
|
||||
sourceIds.add(source.id);
|
||||
}
|
||||
for (const key of ["title", "publisher", "date", "retrievedAt", "url", "locator", "excerpt"]) {
|
||||
if (!source[key] || typeof source[key] !== "string") {
|
||||
findings.push(
|
||||
finding(`${base}.${key}`, "source-field", `${key} must be a non-empty string.`),
|
||||
);
|
||||
}
|
||||
}
|
||||
const sourceDate = parseIsoDate(source.date);
|
||||
if (typeof source.date === "string" && sourceDate === null) {
|
||||
findings.push(
|
||||
finding(`${base}.date`, "source-date", "Source date must be a real calendar date in YYYY-MM-DD format."),
|
||||
);
|
||||
} else if (sourceDate !== null && updatedAt !== null && sourceDate > updatedAt) {
|
||||
findings.push(
|
||||
finding(`${base}.date`, "future-source-date", "Source date cannot be later than map.updatedAt."),
|
||||
);
|
||||
}
|
||||
const retrievedAt = retrievalDate(source.retrievedAt);
|
||||
if (typeof source.retrievedAt === "string" && retrievedAt === null) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${base}.retrievedAt`,
|
||||
"retrieval-date",
|
||||
"retrievedAt must be YYYY-MM-DD or an ISO UTC timestamp ending in Z.",
|
||||
),
|
||||
);
|
||||
} else if (retrievedAt !== null && updatedAt !== null && retrievedAt > updatedAt) {
|
||||
findings.push(
|
||||
finding(`${base}.retrievedAt`, "future-retrieval", "retrievedAt cannot be later than map.updatedAt."),
|
||||
);
|
||||
} else if (retrievedAt !== null && sourceDate !== null && retrievedAt < sourceDate) {
|
||||
findings.push(
|
||||
finding(`${base}.retrievedAt`, "retrieval-before-source", "retrievedAt cannot predate the source date."),
|
||||
);
|
||||
}
|
||||
if (typeof source.url === "string" && !sourceLocation(source.url)) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${base}.url`,
|
||||
"source-url",
|
||||
"Source location must be http(s), file://, or a relative or absolute local path.",
|
||||
),
|
||||
);
|
||||
}
|
||||
if (typeof source.locator === "string" && !boundedLocator(source.locator)) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${base}.locator`,
|
||||
"source-locator",
|
||||
"Locator must identify a bounded page, section, line range, or timestamp (for example p. 7, § 2.1, L12-L18, Section: Results, or 00:04:31).",
|
||||
),
|
||||
);
|
||||
}
|
||||
if (typeof source.excerpt === "string" && source.excerpt.trim().length < 40) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${base}.excerpt`,
|
||||
"thin-excerpt",
|
||||
"Source excerpt must contain at least 40 characters of checkable context.",
|
||||
),
|
||||
);
|
||||
}
|
||||
if (typeof source.excerpt === "string" && source.excerpt.trim().length > 500) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${base}.excerpt`,
|
||||
"oversized-excerpt",
|
||||
"Keep source excerpts under 500 characters and link to the full source.",
|
||||
),
|
||||
);
|
||||
}
|
||||
if (
|
||||
typeof source.excerpt === "string"
|
||||
&& source.excerpt.trim().length >= 40
|
||||
&& source.excerpt.trim().length <= 500
|
||||
&& !substantiveExcerpt(source.excerpt)
|
||||
) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${base}.excerpt`,
|
||||
"low-information-excerpt",
|
||||
"Source excerpt must contain varied, checkable content rather than repeated filler.",
|
||||
),
|
||||
);
|
||||
}
|
||||
if (source.verification != null) {
|
||||
const verification = source.verification;
|
||||
const verificationBase = `${base}.verification`;
|
||||
if (!verification || typeof verification !== "object" || Array.isArray(verification)) {
|
||||
findings.push(
|
||||
finding(verificationBase, "verification-type", "verification must be an object."),
|
||||
);
|
||||
} else {
|
||||
if (verification.status !== "verified") {
|
||||
findings.push(
|
||||
finding(`${verificationBase}.status`, "verification-status", "Verification status must be verified."),
|
||||
);
|
||||
}
|
||||
if (verification.method !== "normalized-excerpt-match") {
|
||||
findings.push(
|
||||
finding(
|
||||
`${verificationBase}.method`,
|
||||
"verification-method",
|
||||
"Verification method must be normalized-excerpt-match.",
|
||||
),
|
||||
);
|
||||
}
|
||||
if (!validUtcTimestamp(verification.checkedAt)) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${verificationBase}.checkedAt`,
|
||||
"verification-time",
|
||||
"Verification checkedAt must be an ISO UTC timestamp ending in Z.",
|
||||
),
|
||||
);
|
||||
} else if (source.retrievedAt !== verification.checkedAt.slice(0, 10)) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${verificationBase}.checkedAt`,
|
||||
"verification-retrieval-mismatch",
|
||||
"A verified source retrievedAt must equal the UTC date in verification.checkedAt.",
|
||||
),
|
||||
);
|
||||
}
|
||||
for (const key of ["contentSha256", "excerptSha256"]) {
|
||||
if (typeof verification[key] !== "string" || !/^[a-f0-9]{64}$/.test(verification[key])) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${verificationBase}.${key}`,
|
||||
"verification-digest",
|
||||
`${key} must be a lowercase SHA-256 digest.`,
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
if (!["matched", "not-machine-checked"].includes(verification.locatorStatus)) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${verificationBase}.locatorStatus`,
|
||||
"verification-locator",
|
||||
"locatorStatus must be matched or not-machine-checked.",
|
||||
),
|
||||
);
|
||||
}
|
||||
if (typeof verification.finalUrl !== "string" || !sourceLocation(verification.finalUrl)) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${verificationBase}.finalUrl`,
|
||||
"verification-url",
|
||||
"finalUrl must be an http(s), file://, or local path source location.",
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const incoming = new Map(nodes.filter((node) => node?.id).map((node) => [node.id, 0]));
|
||||
const adjacency = new Map(nodes.filter((node) => node?.id).map((node) => [node.id, []]));
|
||||
const uniqueEdges = new Set();
|
||||
for (const [index, edge] of edges.entries()) {
|
||||
const base = `$.edges[${index}]`;
|
||||
if (!edge || typeof edge !== "object" || Array.isArray(edge)) {
|
||||
findings.push(finding(base, "edge-type", "Each edge must be an object."));
|
||||
continue;
|
||||
}
|
||||
if (!nodeIds.has(edge.from)) {
|
||||
findings.push(finding(`${base}.from`, "unknown-node", `Unknown from node: ${edge.from}.`));
|
||||
}
|
||||
if (!nodeIds.has(edge.to)) {
|
||||
findings.push(finding(`${base}.to`, "unknown-node", `Unknown to node: ${edge.to}.`));
|
||||
}
|
||||
if (edge.from && edge.from === edge.to) {
|
||||
findings.push(finding(base, "self-edge", `Node ${edge.from} cannot point to itself.`));
|
||||
}
|
||||
const edgeKey = `${edge.from}\0${edge.to}\0${edge.relation}`;
|
||||
if (uniqueEdges.has(edgeKey)) {
|
||||
findings.push(
|
||||
finding(base, "duplicate-edge", "Duplicate from/to/relation edges are not allowed."),
|
||||
);
|
||||
} else {
|
||||
uniqueEdges.add(edgeKey);
|
||||
}
|
||||
if (!RELATIONS.has(edge.relation)) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${base}.relation`,
|
||||
"edge-relation",
|
||||
`Relation must be one of: ${[...RELATIONS].join(", ")}.`,
|
||||
),
|
||||
);
|
||||
}
|
||||
if (!edge.note || typeof edge.note !== "string") {
|
||||
findings.push(
|
||||
finding(`${base}.note`, "edge-note", "Each reasoning edge needs a plain-language note."),
|
||||
);
|
||||
}
|
||||
if (nodeIds.has(edge.to)) incoming.set(edge.to, (incoming.get(edge.to) || 0) + 1);
|
||||
if (nodeIds.has(edge.from) && nodeIds.has(edge.to) && edge.from !== edge.to) {
|
||||
adjacency.get(edge.from).push(edge.to);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [index, node] of nodes.entries()) {
|
||||
if (!node || typeof node !== "object") continue;
|
||||
const base = `$.nodes[${index}]`;
|
||||
if (node.type === "evidence" && !node.sourceId) {
|
||||
findings.push(
|
||||
finding(`${base}.sourceId`, "unsourced-evidence", "Evidence nodes require sourceId."),
|
||||
);
|
||||
}
|
||||
if (node.sourceId && !sourceIds.has(node.sourceId)) {
|
||||
findings.push(
|
||||
finding(
|
||||
`${base}.sourceId`,
|
||||
"unknown-source",
|
||||
`Node references unknown source: ${node.sourceId}.`,
|
||||
),
|
||||
);
|
||||
}
|
||||
if (
|
||||
node.type === "evidence" &&
|
||||
node.id &&
|
||||
!edges.some((edge) => edge?.from === node.id)
|
||||
) {
|
||||
findings.push(
|
||||
finding(base, "unused-evidence", "Evidence must participate in at least one reasoning edge."),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [index, source] of sources.entries()) {
|
||||
if (
|
||||
source?.id &&
|
||||
!nodes.some((node) => node?.type === "evidence" && node.sourceId === source.id)
|
||||
) {
|
||||
findings.push(
|
||||
finding(
|
||||
`$.sources[${index}]`,
|
||||
"unused-source",
|
||||
"Every source must be attached to at least one evidence node.",
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const positions = nodes.filter((node) => node?.type === "position");
|
||||
if (positions.length !== 1) {
|
||||
findings.push(
|
||||
finding("$.nodes", "position-count", "The map must contain exactly one position node."),
|
||||
);
|
||||
} else if (!incoming.get(positions[0].id)) {
|
||||
findings.push(
|
||||
finding(
|
||||
`$.nodes[${nodes.indexOf(positions[0])}]`,
|
||||
"unsupported-position",
|
||||
"The position needs at least one incoming reasoning edge.",
|
||||
),
|
||||
);
|
||||
} else {
|
||||
for (const [index, node] of nodes.entries()) {
|
||||
if (!node?.id || node.id === positions[0].id) continue;
|
||||
if (!reaches(node.id, positions[0].id, adjacency)) {
|
||||
findings.push(
|
||||
finding(
|
||||
`$.nodes[${index}]`,
|
||||
"disconnected-node",
|
||||
`Node ${node.id} must have a directed reasoning path to the position.`,
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const visitState = new Map();
|
||||
const cyclicNodes = new Set();
|
||||
function visit(nodeId, stack = []) {
|
||||
const state = visitState.get(nodeId) || 0;
|
||||
if (state === 1) {
|
||||
for (const member of stack.slice(stack.indexOf(nodeId))) cyclicNodes.add(member);
|
||||
return;
|
||||
}
|
||||
if (state === 2) return;
|
||||
visitState.set(nodeId, 1);
|
||||
for (const next of adjacency.get(nodeId) || []) visit(next, [...stack, nodeId]);
|
||||
visitState.set(nodeId, 2);
|
||||
}
|
||||
for (const nodeId of nodeIds) visit(nodeId);
|
||||
for (const nodeId of cyclicNodes) {
|
||||
const index = nodes.findIndex((node) => node?.id === nodeId);
|
||||
findings.push(
|
||||
finding(`$.nodes[${index}]`, "reasoning-cycle", `Node ${nodeId} participates in a reasoning cycle.`),
|
||||
);
|
||||
}
|
||||
|
||||
const metrics = {
|
||||
claims: nodes.filter((node) => node?.type === "claim").length,
|
||||
contradictions: new Set(
|
||||
edges
|
||||
.filter((edge) => edge?.relation === "contradicts")
|
||||
.map((edge) => `${edge.from}\0${edge.to}\0${edge.relation}`),
|
||||
).size,
|
||||
evidence: nodes.filter((node) => node?.type === "evidence").length,
|
||||
sources: sources.length,
|
||||
unknowns: nodes.filter((node) => node?.type === "unknown").length,
|
||||
};
|
||||
return {
|
||||
findings,
|
||||
metrics,
|
||||
receipt: null,
|
||||
valid: findings.length === 0,
|
||||
};
|
||||
}
|
||||
|
||||
export function validateMapContract(map) {
|
||||
const result = inspectMapContract(map);
|
||||
if (!result.valid) throw new MapValidationError(result.findings);
|
||||
return result;
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import {
|
||||
canonicalJson,
|
||||
inspectMapContract,
|
||||
receiptPayload,
|
||||
} from "./contract.mjs";
|
||||
|
||||
function receiptFor(map) {
|
||||
const sourceSnapshots = map.sources.map((source) => ({
|
||||
id: source.id,
|
||||
retrievedAt: source.retrievedAt,
|
||||
excerptSha256: createHash("sha256").update(source.excerpt).digest("hex"),
|
||||
}));
|
||||
return createHash("sha256")
|
||||
.update(canonicalJson(receiptPayload(map, sourceSnapshots)))
|
||||
.digest("hex");
|
||||
}
|
||||
|
||||
export function inspectOfflineMap(map) {
|
||||
const result = inspectMapContract(map);
|
||||
return {
|
||||
...result,
|
||||
receipt: result.valid ? receiptFor(map) : null,
|
||||
};
|
||||
}
|
||||
|
||||
export async function validateMapFile(file) {
|
||||
let map;
|
||||
try {
|
||||
const input = file instanceof URL ? file : resolve(file);
|
||||
map = JSON.parse(await readFile(input, "utf8"));
|
||||
} catch (error) {
|
||||
return {
|
||||
findings: [{
|
||||
path: "$",
|
||||
rule: "invalid-json",
|
||||
message: `Could not parse JSON: ${error.message}`,
|
||||
}],
|
||||
metrics: { claims: 0, contradictions: 0, evidence: 0, sources: 0, unknowns: 0 },
|
||||
receipt: null,
|
||||
valid: false,
|
||||
};
|
||||
}
|
||||
return inspectOfflineMap(map);
|
||||
}
|
||||
|
||||
function plural(count, singular, pluralForm = `${singular}s`) {
|
||||
return count === 1 ? singular : pluralForm;
|
||||
}
|
||||
|
||||
function printHuman(result) {
|
||||
if (!result.valid) {
|
||||
console.error(`INVALID ${result.findings.length} ${plural(result.findings.length, "finding")}`);
|
||||
for (const finding of result.findings) {
|
||||
console.error(` - ${finding.path} [${finding.rule}] ${finding.message}`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
console.log(`VALID ${result.receipt}`);
|
||||
console.log(
|
||||
` ✓ ${result.metrics.claims} ${plural(result.metrics.claims, "claim")} · ${result.metrics.evidence} evidence · ${result.metrics.sources} ${plural(result.metrics.sources, "source")}`,
|
||||
);
|
||||
console.log(
|
||||
` ↯ ${result.metrics.contradictions} ${plural(result.metrics.contradictions, "contradiction")} · ${result.metrics.unknowns} explicit ${plural(result.metrics.unknowns, "unknown")}`,
|
||||
);
|
||||
}
|
||||
|
||||
async function main(argv) {
|
||||
if (argv.includes("--help") || argv.includes("-h")) {
|
||||
console.log("Usage: node <skill-dir>/scripts/validate.mjs <map.doubt.json> [--json]");
|
||||
return;
|
||||
}
|
||||
const file = argv.find((value) => !value.startsWith("-"));
|
||||
if (!file) throw new Error("Pass a .doubt.json evidence map.");
|
||||
const result = await validateMapFile(file);
|
||||
if (argv.includes("--json")) console.log(JSON.stringify(result, null, 2));
|
||||
else printHuman(result);
|
||||
if (!result.valid) process.exitCode = 1;
|
||||
}
|
||||
|
||||
if (process.argv[1] === fileURLToPath(import.meta.url)) {
|
||||
main(process.argv.slice(2)).catch((error) => {
|
||||
console.error(error.message);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
@@ -11,14 +11,21 @@ Use the configured Codebase Memory graph as a discovery accelerator, not as the
|
||||
|
||||
1. Discover the Codebase Memory tools exposed by the current MCP client; clients may prefix or rename tool namespaces.
|
||||
2. Call `list_projects` when available and use the exact indexed project name. If the repository is not indexed, continue with local exploration or ask before calling `index_repository` when graph access is important.
|
||||
3. Before branch-sensitive or edit-sensitive conclusions, use `index_status` or `detect_changes` when available. After a branch switch, assume the index may be stale until checked. If freshness cannot be established, disclose that limitation and verify locally.
|
||||
4. Use `get_architecture` once for orientation in an unfamiliar repository or subsystem. Do not repeat it for narrow follow-up questions.
|
||||
5. Use `search_graph` for definitions, implementations, routes, classes, interfaces, callers, and related symbols. Prefer a natural-language query for discovery and a name or qualified-name pattern for known symbols. Narrow by label or path, set a result limit, and paginate or reduce scope when the response reports more results.
|
||||
3. Before branch-sensitive or edit-sensitive conclusions, use `index_status` and verify the actual version-control state. Use `detect_changes` only when its Git base and head are valid for the checkout. If it unexpectedly reports zero changes, or the checkout uses another VCS, inspect that VCS's status or diff before claiming no impact.
|
||||
4. Use `get_architecture` once for unfamiliar structure. Request `clusters` to discover de-facto module seams. Treat `cycles` as an opt-in whole-call-graph scan: `path` does not scope cycle detection, so verify relevant cycles before making module-local claims.
|
||||
5. Use `search_graph` for definitions, implementations, routes, classes, interfaces, and related symbols. Prefer a natural-language query for discovery and a name or qualified-name pattern for known symbols. Narrow by label or path and set a result limit. For exhaustive claims, increase `offset` by `limit` while `has_more` is true.
|
||||
6. Use `search_code` or normal repository search for literal strings, configuration keys, test identifiers, error messages, and non-code files. Do not turn a precise text lookup into a broad graph query.
|
||||
7. After graph search, use `get_code_snippet` with the returned qualified name. If source snippets are unavailable, open the local file before relying on the result.
|
||||
8. Use `trace_path` for callers, callees, dependency paths, data flow, cross-service paths, and impact analysis. Include tests only when test coverage is part of the question.
|
||||
9. Use `get_graph_schema` before `query_graph`. Reserve custom queries for multi-hop or aggregate questions that simpler tools cannot answer, and apply `LIMIT` or the tool's row limit.
|
||||
10. When graph and checked-out source disagree, treat source as current and report likely index drift.
|
||||
8. Use `trace_path` for callers, callees, dependency paths, data flow, cross-service paths, and impact analysis. Include tests when the claim covers them. While `truncated` is true, pass `next` back as `cursor` with every other argument unchanged.
|
||||
9. After identifying candidate files, call `check_index_coverage` for every cited path. Before negative or exhaustive claims, also check the relevant `scopes`; advance `scope_offset` to each `next_offset` while `has_more` is true. This metadata is best-effort, not proof of completeness. Inspect local source for partial, skipped, excluded, stale, or otherwise uncovered paths.
|
||||
10. Use `get_graph_schema` before custom `query_graph` calls. Reserve them for bounded multi-hop or aggregate questions, apply `LIMIT` or `max_rows`, and use `graph="missed"` to audit files the main graph did not fully index.
|
||||
11. Complete every relevant result stream before an exhaustive claim. For bounded discovery, stopping early is acceptable when the result states its limit or truncation. When graph and checked-out source disagree, treat source as current and report likely index drift.
|
||||
|
||||
## Indexing Modes
|
||||
|
||||
- Use `moderate` by default for normal indexing: it filters files while retaining similarity and semantic edges.
|
||||
- Use `fast` only for an explicitly requested smoke index, or when `moderate` is blocked and a degraded fallback is useful. Disclose that similarity and semantic edges are absent.
|
||||
- Use `full` when moderate-only discovery filters omit relevant supported files and the additional indexing cost is justified. Full still honors `.gitignore`, `.cbmignore`, and always-skip rules.
|
||||
|
||||
## Safety and Fallbacks
|
||||
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
name: gh-attach
|
||||
description: 'Uploads a local file (screenshot, image, PDF, zip, video) to GitHub user-attachments, downloads GitHub user-attachments, and embeds local files in a PR, issue, or comment. Use when asked to "attach a screenshot to the PR", "add an image to the issue", "embed before/after screenshots", "attach this file", or "download this GitHub attachment". Powered by `gh-attach`.'
|
||||
---
|
||||
|
||||
# gh-attach
|
||||
|
||||
`gh attach` uploads a file to GitHub's internal user-attachments endpoint (no public API exists) and prints the URL, which GitHub auto-renders (image/video/file) wherever it's pasted. The URL inherits the repo's visibility, so private-repo uploads stay private.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```sh
|
||||
gh extension list | grep -q 'gh attach' || gh extension install sudosubin/gh-attach
|
||||
```
|
||||
|
||||
Uploads use a GitHub browser session cookie, not the `gh` token. By default, `gh` must be authenticated so `gh-attach` can select the matching browser account. If the wrong account is selected, add `--browser <name> --profile <name>`. For headless use, set `GH_ATTACH_SESSION_TOKEN` to the bare `user_session` cookie value. Treat it as a full account credential.
|
||||
|
||||
## Steps
|
||||
|
||||
**1. Upload**: Use an absolute quoted path. `-R` is optional inside a repository. For GHES, use `-R host/owner/repo`. The command prints the URL on one line. GitHub auto-renders it (image/video/file), so use it as-is:
|
||||
|
||||
```sh
|
||||
URL=$(gh attach "$FILE" -R <owner>/<repo>)
|
||||
```
|
||||
|
||||
**2. Embed** (always `--body-file -`, e.g. `gh pr comment/edit`, `gh issue comment/edit`):
|
||||
|
||||
```sh
|
||||
printf '## Screenshots\n\n%s\n' "$URL" | gh pr comment <pr> -R <owner>/<repo> --body-file -
|
||||
```
|
||||
|
||||
**3. Download**: Specify the destination explicitly. Private attachments use the active `gh` token, with browser cookies as an authorization fallback:
|
||||
|
||||
```sh
|
||||
gh attach download "$URL" -O "$FILE"
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Private repo: URL renders only for authorized viewers. An anonymous fetch is expected to return 404 or 403.
|
||||
- Sizing: embed `<img width="800" src="$URL">` instead of the bare URL.
|
||||
- GitHub Cloud and GHES decide which file extensions and content types they accept.
|
||||
@@ -0,0 +1,296 @@
|
||||
---
|
||||
name: shopify-review-triage
|
||||
description: 'Use this skill when someone wants public Shopify App Store reviews, low-star reviews, or merchant feedback triaged, prioritized, clustered, or turned into a product or support brief. Trigger for prompts like "triage these app store reviews", "what should we fix first from this feedback", "cluster our 1-star reviews", or "write a weekly low-star review brief", for a single Shopify app or a portfolio plus watched competitors. Produces a P0-P3 brief covering incident risk, repeated friction, pricing confusion, feature requests, and an explicit needs-human-read bucket, where every item keeps its public source link and stays labeled first pass or human-checked. Do not trigger for support tickets, order data, or any other private merchant data, and never use it to reply to or contact a reviewer.'
|
||||
license: MIT
|
||||
compatibility: 'Cross-platform. Pure reasoning skill over review rows the user pastes - no network access, scripts, API keys, or system packages. Portable to any client that supports the Agent Skills SKILL.md format.'
|
||||
metadata:
|
||||
version: '1.0'
|
||||
author: 'Shopify App Review Brief - independent, not affiliated with or endorsed by Shopify Inc.'
|
||||
source: https://alfredtech2026.github.io/shopify-app-review-brief/guides/shopify-app-review-triage.html
|
||||
---
|
||||
|
||||
# Shopify review triage - public low-star reviews to a P0-P3 brief
|
||||
|
||||
## What this does
|
||||
|
||||
Takes rows of **public** Shopify App Store review text and produces one prioritized brief a
|
||||
product or support owner can act on: what kind of problem each review describes, how badly it
|
||||
can hurt, what to do first, and where the original wording came from.
|
||||
|
||||
It is built for independent Shopify app teams and the agencies that run their support - the
|
||||
case where low-star reviews arrive scattered across several listings plus a few watched
|
||||
competitors, and the failure mode is treating them all as equally urgent.
|
||||
|
||||
The rubric below is not invented here. It reproduces a publicly published rule set verbatim, so
|
||||
a manual pass and this skill sort the same row the same way. See [Provenance](#provenance) for
|
||||
the source.
|
||||
|
||||
## Hard rules
|
||||
|
||||
These are not style preferences. Breaking one makes the output worse than nothing.
|
||||
|
||||
1. **Public review text only.** Never accept, request, or copy support tickets, merchant emails,
|
||||
order data, personal contact details, internal telemetry, or anything else not already public
|
||||
on a listing page. If such data appears in the input, stop, say which rows are affected, and
|
||||
ask for them to be removed before continuing.
|
||||
2. **Never invent evidence.** Do not write a review, a rating, a date, an app name, or a source
|
||||
URL that was not supplied. A row with no link gets `source: not captured` - never a guessed one.
|
||||
3. **Keyword output is a sort, not a verdict.** Everything produced by the rubric alone is
|
||||
labeled *first pass - not human-checked*. Only a person who read the review and checked it
|
||||
against their own systems may relabel an item *human-checked*.
|
||||
4. **Reviews are customer reports, not verified defects.** Write "the reviewer reports the editor
|
||||
showed a blank screen", never "the editor is broken". The distinction survives into the brief.
|
||||
5. **No coverage claims.** The brief covers exactly the rows supplied and says so. Make no claim
|
||||
of exhaustive coverage of a listing, a period, or an app.
|
||||
6. **No promises.** No revenue impact, no outcome, no ranking effect, no legal or compliance
|
||||
advice. Suggest actions; do not predict results.
|
||||
7. **Draft only - never contact anyone.** Do not send email, post a developer reply, open a
|
||||
support ticket, message a reviewer, or publish anything. Hand the draft back to the team and
|
||||
let a person decide what to send.
|
||||
8. **Reviewers are people.** Refer to "the reviewer". Do not name, profile, or speculate about them.
|
||||
|
||||
## 1. Collect the rows
|
||||
|
||||
Ask for one review per line. The full form keeps the source link, which the brief needs:
|
||||
|
||||
```text
|
||||
rating | app name | review date | public reviews URL | review text
|
||||
```
|
||||
|
||||
A shorter three-field form is also accepted - treat field 1 as the rating when it is a bare 1-5
|
||||
(optionally followed by `star` or `stars`), otherwise as the app name:
|
||||
|
||||
```text
|
||||
rating | app name | review text
|
||||
```
|
||||
|
||||
Rules for this step:
|
||||
|
||||
- Lines starting with `#` are comments. Blank lines are skipped.
|
||||
- If a row lacks a source URL, carry `source: not captured` through to the brief. Do not drop
|
||||
the row and do not fabricate a link.
|
||||
- Do not go and fetch anything yourself. This skill needs no network access; the person you are
|
||||
helping pastes the public rows they already opened.
|
||||
- The trigger this rubric is tuned for is a **new 1-3-star review**. Higher-rated rows still
|
||||
classify correctly (a 5-star review often lands in feature requests or needs-human-read), so
|
||||
keep them if they were supplied, but never present them as low-star signal.
|
||||
|
||||
## 2. First pass - apply the rubric
|
||||
|
||||
Lower-case the review text and normalize curly apostrophes (`’` to `'`) before matching, so a
|
||||
pasted "won’t load" still matches `won't load`. Also match every keyword below with the
|
||||
apostrophe dropped entirely: merchants routinely type these contractions without one, and the
|
||||
apostrophe-free spelling must classify exactly the same as the contracted form.
|
||||
|
||||
Five buckets. Each row gets exactly **one primary** bucket - the first dimension below, in this
|
||||
order, with any matching keyword. Further matches are recorded as **secondary**, never as a
|
||||
second brief item.
|
||||
|
||||
### P0 - Incident risk
|
||||
|
||||
The purchase path, app activation, or merchant data may be at stake right now. Left alone it
|
||||
costs the merchant money and the team installs.
|
||||
|
||||
**Suggested action.** Try to reproduce on a test store today. If confirmed, treat it as an
|
||||
incident: fix or mitigate first, then reply to the reviewer with what changed.
|
||||
|
||||
**Signal keywords.** `won't load`, `won't open`, `won't close`, `can't close`, `cannot close`, `blank screen`, `broken`, `crash`, `stopped working`, `not working`, `doesn't work`, `does not work`, `checkout`, `losing sales`, `lost sales`, `error`
|
||||
|
||||
### P1 - Repeated friction
|
||||
|
||||
The product works, but the same struggle keeps showing up across reviews or against an open
|
||||
support theme. Repetition is the signal, not volume of adjectives.
|
||||
|
||||
**Suggested action.** Log it against the matching support theme. If the same complaint repeats
|
||||
across rows, schedule a UX fix ahead of new feature work.
|
||||
|
||||
**Signal keywords.** `confusing`, `unclear`, `hard to`, `difficult`, `complicated`, `clunky`, `slow`, `couldn't figure`, `could not figure`, `annoying`, `had to contact support`, `setup took`, `too many steps`
|
||||
|
||||
### P2 - Pricing confusion
|
||||
|
||||
What the merchant expected to pay and what happened diverged. Usually a copy problem in the
|
||||
listing, the plan limits, or the upgrade prompts - not a code problem.
|
||||
|
||||
**Suggested action.** Compare what the reviewer expected with the listing's pricing section and
|
||||
in-app upgrade prompts; clarify the copy where they diverge.
|
||||
|
||||
**Signal keywords.** `pricing`, `price`, `charged`, `charge`, `billing`, `billed`, `expensive`, `free plan`, `trial`, `refund`, `hidden fee`, `hidden cost`, `paywall`
|
||||
|
||||
### P3 - Feature request
|
||||
|
||||
The merchant wants something the app does not do, or could not find. Valuable as a log entry,
|
||||
rarely urgent on its own.
|
||||
|
||||
**Suggested action.** Add it to the feature-request log with a link to the review. If the
|
||||
capability already exists, reply to the reviewer with where to find it.
|
||||
|
||||
**Signal keywords.** `wish`, `would be great`, `would love`, `please add`, `feature request`, `missing`, `if only`, `would like`, `no option to`, `needs an option`, `hope you add`, `add support for`
|
||||
|
||||
### Needs human read
|
||||
|
||||
No keyword matched. Vague frustration, sarcasm, mixed praise, or a story that needs context.
|
||||
|
||||
**Suggested action.** No keyword matched. Read the full review yourself and file it manually -
|
||||
the heuristic makes no guess here.
|
||||
|
||||
**Priority.** The rubric labels this bucket `P2` and sorts it last. Treat that label as
|
||||
provisional placement in the queue, not as a severity judgment - nothing has been judged yet.
|
||||
|
||||
### Tie-breaks and escalation
|
||||
|
||||
1. **Most severe wins.** A row naming both a broken checkout and a billing surprise files under
|
||||
P0 with pricing noted as secondary. Never split one review across two brief items.
|
||||
2. **Repetition escalates.** If the same friction or pricing theme appears in three or more
|
||||
reviews within about 60 days, move it up one level and say how many rows drove the change.
|
||||
3. **Age discounts.** A review older than a year is background, not evidence of a current
|
||||
problem, unless a recent row corroborates it. Cite it as context, never as the headline.
|
||||
4. **Competitor reviews never create a P0 for you.** A competitor's incident is roadmap,
|
||||
positioning, or copy input - it belongs in the competitor watch section.
|
||||
5. **When unsure, choose needs human read.** The bucket exists so the rubric never launders
|
||||
uncertainty into a priority label.
|
||||
|
||||
## 3. Human pass - verify before you promote anything
|
||||
|
||||
The first pass is where this skill stops being able to help on its own. Before any item is
|
||||
presented as more than a keyword match, a person on the team has to:
|
||||
|
||||
- read the full original review at its source link;
|
||||
- for P0 candidates, attempt to reproduce on a development store and check the error tracker and
|
||||
support inbox for matching signals from the same period;
|
||||
- record the outcome as *reproduced*, *not reproduced*, or *attempted - notes attached*.
|
||||
|
||||
Ask for these outcomes rather than assuming them. Until you have them, every item stays labeled
|
||||
*first pass - not human-checked*, including in the summary line. An unverified P0 is a candidate,
|
||||
not an incident.
|
||||
|
||||
Known limits to state plainly when they apply: keyword matching is English-only, misses sarcasm
|
||||
and context, can misfile a review that mentions "checkout" in passing, and sees only the rows
|
||||
supplied.
|
||||
|
||||
## 4. Write the brief
|
||||
|
||||
One document per portfolio, sections in rubric order, every item carrying an owner, a next
|
||||
action, and a source link. An item without an owner is a note, not a brief entry.
|
||||
|
||||
```markdown
|
||||
# Low-star review brief - {portfolio or team name} - week of {YYYY-MM-DD}
|
||||
|
||||
Scope: {apps monitored} - {competitors watched} - {N} rows supplied, {date range}.
|
||||
Covers only the rows supplied - no claim of exhaustive coverage.
|
||||
Reviews are customer reports, not verified defects. Items marked "first pass" are
|
||||
unverified keyword matches; "human-checked" means a person read the review and checked it.
|
||||
|
||||
## P0 - Incident risk
|
||||
- **{App} - {signal in a few words}** ({rating} stars, {review date}, [source]({public reviews URL}))
|
||||
- Reviewer reports: {one sentence, in their words where possible}
|
||||
- Status: first pass - not human-checked / human-checked
|
||||
- Reproduced: {yes / no / attempted - notes}
|
||||
- Next action: {action} - owner {name}, due {date}
|
||||
|
||||
## P1 - Repeated friction
|
||||
- **{App} - {theme}** ({rating} stars, {date}, [source]({public reviews URL}); also seen: {where})
|
||||
- Status: first pass - not human-checked / human-checked
|
||||
- Next action: {UX or docs change} - owner {name}, due {date}
|
||||
|
||||
## P2 - Pricing confusion
|
||||
- **{App} - {signal}** ({rating} stars, {date}, [source]({public reviews URL}))
|
||||
- Expected vs. actual: {one line}
|
||||
- Status: first pass - not human-checked / human-checked
|
||||
- Next action: {copy or prompt change} - owner {name}, due {date}
|
||||
|
||||
## P3 - Feature requests
|
||||
- **{App} - {request}** ({rating} stars, {date}, [source]({public reviews URL})) - {log it, or already exists so reply with where to find it}
|
||||
|
||||
## Needs human read
|
||||
- **{App}** ({rating} stars, {date}, [source]({public reviews URL})) - {no keyword matched; what a human should look for}
|
||||
|
||||
## Competitor watch
|
||||
- **{Competitor} - {signal}**: {what it implies for our roadmap, copy, or positioning}
|
||||
|
||||
## Decisions this week
|
||||
- {one decision or experiment, with the rows that motivated it}
|
||||
```
|
||||
|
||||
Open the summary line with the counts, e.g. *"Triaged 8 rows supplied: 3 incident risk,
|
||||
2 repeated friction, 1 pricing confusion, 1 feature request, 1 needs human read - first pass,
|
||||
not human-checked."*
|
||||
|
||||
## 5. Self-check before you hand it over
|
||||
|
||||
Refuse to deliver until every line is true:
|
||||
|
||||
- [ ] Every item names its bucket and priority from the rubric above, and nothing else.
|
||||
- [ ] Every item carries a source link or an explicit `source: not captured`.
|
||||
- [ ] No review text, rating, date, app name, or URL appears that was not supplied.
|
||||
- [ ] Every unverified item says *first pass - not human-checked*; nothing claims a human check
|
||||
that did not happen.
|
||||
- [ ] Claims are phrased as reports ("the reviewer reports..."), not as findings about the code.
|
||||
- [ ] The scope line says how many rows were supplied and makes no coverage claim.
|
||||
- [ ] No promise about revenue, ratings, outcomes, or compliance appears anywhere.
|
||||
- [ ] No private data survived into the output.
|
||||
- [ ] Nothing was sent, posted, or published - the brief is a draft for the team.
|
||||
|
||||
## Worked example
|
||||
|
||||
These eight fictional rows exercise every bucket. Two of them are deliberately 4-star and
|
||||
5-star, to reach the feature-request and needs-human-read buckets.
|
||||
|
||||
```text
|
||||
1 | Example Popup App | The editor shows a blank screen and the popup won't load. We are losing sales every day.
|
||||
2 | Example Popup App | The overlay can't close on mobile and it blocks the checkout button.
|
||||
1 | Example Currency App | Conversion is broken at checkout and we were still billed for the month.
|
||||
3 | Example Currency App | Setup took hours and the settings screen is confusing. Support was slow to reply.
|
||||
3 | Example Reviews App | The widget looks fine but the template editor is confusing and hard to use on a tablet.
|
||||
2 | Example Currency App | We kept getting charged after uninstalling, and the pricing page never mentioned this.
|
||||
4 | Example Reviews App | Great app, but I wish it could export reviews to CSV. Please add filtering by country.
|
||||
5 | Example Reviews App | Does what it promises and support replied the same day.
|
||||
```
|
||||
|
||||
First pass over those rows:
|
||||
|
||||
```text
|
||||
row 1 -> P0 incident risk
|
||||
row 2 -> P0 incident risk
|
||||
row 3 -> P0 incident risk (secondary: pricing confusion)
|
||||
row 4 -> P1 repeated friction
|
||||
row 5 -> P1 repeated friction
|
||||
row 6 -> P2 pricing confusion
|
||||
row 7 -> P3 feature request
|
||||
row 8 -> needs human read
|
||||
```
|
||||
|
||||
Rows 4 and 5 both matched `confusing`, so they are flagged as a repeated theme - two rows, which
|
||||
is a cluster to watch, not yet the three that trigger escalation. Row 3 is a single P0 item with
|
||||
pricing recorded as secondary, never two items. Row 8 matched nothing and stays unjudged. None of
|
||||
these rows carried a source URL, so each item would read `source: not captured` until the team
|
||||
supplies the listing links.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **The Shopify App Store has no stable per-review permalink.** Cite the listing's public reviews
|
||||
page, keep the rating filter if one was used (`.../reviews?ratings%5B%5D=1`), and pin the item
|
||||
with the review date plus the reviewer's first few words so a human can find it again.
|
||||
- **Prefer the five-field input form.** It carries the review date and the source URL the brief
|
||||
needs. A three-field parser folds everything after the second `|` into the review text, so a
|
||||
row carrying a date and a URL still classifies but displays them inside the quoted review.
|
||||
- **`checkout` is the noisiest keyword in the set.** It fires on "we love the checkout upsell".
|
||||
A P0 whose only evidence is the word `checkout` is a needs-human-read row wearing a P0 badge -
|
||||
say so instead of promoting it.
|
||||
- **`missing` and `error` cross buckets.** "missing a dark mode" is P3; "settings page errors out"
|
||||
is P0. Primary-bucket order resolves the collision mechanically; the human pass fixes the ones
|
||||
where it guessed wrong.
|
||||
- **Non-English reviews will not match at all.** They land in needs human read. That is the
|
||||
correct outcome - do not translate and then classify as if the keyword had matched.
|
||||
- **A competitor's P0 is not yours.** It goes to competitor watch even when the wording is worse
|
||||
than anything on the team's own listings.
|
||||
- **One review, one item.** Secondary matches are annotations. Splitting a review across sections
|
||||
double-counts the same merchant and inflates every count in the summary line.
|
||||
|
||||
## Provenance
|
||||
|
||||
The dimensions, priorities, keyword lists, suggested actions, tie-break rules, and brief template
|
||||
reproduced above come from a publicly published manual triage guide:
|
||||
<https://alfredtech2026.github.io/shopify-app-review-brief/guides/shopify-app-review-triage.html>
|
||||
|
||||
That guide is maintained independently and is not affiliated with, endorsed by, or sponsored by
|
||||
Shopify Inc. or any app developer. Shopify is a trademark of Shopify Inc.
|
||||
@@ -0,0 +1,229 @@
|
||||
---
|
||||
name: verify-agent-action
|
||||
description: 'Review a proposed AI-agent action or human-approval packet before execution. Use when an agent wants to run a consequential tool, command, deployment, message, purchase, credential operation, or data mutation; when checking whether approval still matches the exact action; or when auditing action evidence for forged results, parameter swaps, replay, correlated reviewers, missing evidence, expiry, or stale monitoring. Produce an evidence-based review only—never execute or authorize the action.'
|
||||
---
|
||||
|
||||
# Verify Agent Action
|
||||
|
||||
Treat a plausible approval screen as a claim, not proof. Verify the complete
|
||||
decision path before a human or an external enforcement point decides whether
|
||||
to act.
|
||||
|
||||
## Preserve the safety boundary
|
||||
|
||||
- Never execute, approve, sign, send, purchase, deploy, or mutate anything.
|
||||
- Never convert this review into execution authority.
|
||||
- Never infer missing evidence, identities, timestamps, or parameters.
|
||||
- Treat a valid schema, checksum, or signature as insufficient by itself.
|
||||
- Treat signatures as evidence of attribution and integrity, not factual truth.
|
||||
- Keep supporting and refuting evidence separate; do not average conflict away.
|
||||
- Fail closed on a material mismatch. Use `INCONCLUSIVE` when required evidence
|
||||
is unavailable.
|
||||
|
||||
Set this field in every final result:
|
||||
|
||||
```json
|
||||
{"execution_authorized": false}
|
||||
```
|
||||
|
||||
## Collect the review packet
|
||||
|
||||
Request only the artifacts needed for the review:
|
||||
|
||||
1. The original user or system request.
|
||||
2. The exact proposed action:
|
||||
- operation or tool name
|
||||
- target resource
|
||||
- complete parameters
|
||||
- filesystem and network scope
|
||||
- maximum execution count
|
||||
- not-before and expiry times
|
||||
3. The assessment that claims the action is justified.
|
||||
4. The source evidence and policy used by that assessment.
|
||||
5. The approval record, including approver identity, role, action digest, nonce,
|
||||
audience, issue time, expiry, and use count.
|
||||
6. The latest monitoring events and expected heartbeat interval.
|
||||
7. The current trusted time and any prior nonce-use record.
|
||||
|
||||
List missing fields before analysis. Do not silently substitute defaults.
|
||||
|
||||
## Build the exact action identity
|
||||
|
||||
Create one normalized action object without dropping fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"operation": "git.push",
|
||||
"target": "owner/repository",
|
||||
"parameters": {
|
||||
"branch": "fix/example",
|
||||
"commit": "40-character-sha",
|
||||
"remote": "origin"
|
||||
},
|
||||
"filesystem_scope": [],
|
||||
"network_scope": ["github.com:443"],
|
||||
"execution_count": 1,
|
||||
"not_before": "RFC3339 timestamp",
|
||||
"expires_at": "RFC3339 timestamp"
|
||||
}
|
||||
```
|
||||
|
||||
Use a project-specified canonicalization and digest algorithm when provided.
|
||||
Otherwise, report that cryptographic identity cannot be independently verified;
|
||||
still compare every field structurally.
|
||||
|
||||
Never normalize away a security-relevant distinction such as:
|
||||
|
||||
- branch, commit, repository, environment, recipient, amount, currency, or host
|
||||
- recursive, force, overwrite, privileged, destructive, or dry-run flags
|
||||
- filesystem roots, CIDRs, ports, domains, execution counts, or expiry
|
||||
|
||||
## Run the six controls
|
||||
|
||||
Evaluate every control as `PASS`, `FAIL`, `INCONCLUSIVE`, or `NOT_APPLICABLE`.
|
||||
|
||||
### 1. Recompute the assessment
|
||||
|
||||
- Re-run the declared deterministic evaluator from the declared source inputs
|
||||
when its implementation is available.
|
||||
- Compare the complete canonical result, not selected fields.
|
||||
- Mark `FAIL` if the received result differs from recomputation.
|
||||
- Mark `INCONCLUSIVE` when only schema validation, an internal checksum, or an
|
||||
unverifiable evaluator claim is available.
|
||||
|
||||
### 2. Match the exact approved action
|
||||
|
||||
- Compare the proposed action with the action bound into the approval.
|
||||
- Compare the complete normalized object and its digest.
|
||||
- Mark `FAIL` if any material field changed after approval.
|
||||
- Treat a broad target or scope as a mismatch when the evidence justifies only
|
||||
a narrower action.
|
||||
|
||||
### 3. Reject replay and identity ambiguity
|
||||
|
||||
- Verify the nonce is unique and unused.
|
||||
- Verify subject, audience, issuer, approver role, issue time, not-before time,
|
||||
expiry, and maximum use count.
|
||||
- Mark `FAIL` for a reused nonce, wrong audience, expired approval, future-dated
|
||||
approval, excessive use count, revoked identity, or role mismatch.
|
||||
- Mark `INCONCLUSIVE` if no trustworthy replay store or time source exists.
|
||||
|
||||
### 4. Test reviewer independence
|
||||
|
||||
Build a dependence table for every reviewer or evaluator:
|
||||
|
||||
| Dimension | Compare |
|
||||
|---|---|
|
||||
| Model | family, version, fine-tune |
|
||||
| Provider | account and control plane |
|
||||
| Prompt | shared template or ancestry |
|
||||
| Retrieval | overlapping sources and indexes |
|
||||
| Tools | shared evaluator code and runtime |
|
||||
| Operator | common owner or approval authority |
|
||||
|
||||
Do not count correlated reviewers as independent quorum members. Mark `FAIL` if
|
||||
the policy requires independent approval and the remaining independent set is
|
||||
too small.
|
||||
|
||||
### 5. Preserve evidence and contradiction
|
||||
|
||||
- Inventory every evidence identifier referenced by the assessment.
|
||||
- Confirm each item is present, authenticatable, within its validity window,
|
||||
and relevant to the claim.
|
||||
- Record support and refutation independently:
|
||||
|
||||
| Support | Refutation | Epistemic state |
|
||||
|---|---|---|
|
||||
| absent | absent | `UNDETERMINED` |
|
||||
| present | absent | `SUPPORTED_ONLY` |
|
||||
| absent | present | `REFUTED_ONLY` |
|
||||
| present | present | `CONFLICTED` |
|
||||
|
||||
- Mark `FAIL` if evidence was removed, altered, expired, or concealed in a way
|
||||
that changes the result.
|
||||
- Never convert `CONFLICTED` into a numeric average that appears safe.
|
||||
|
||||
### 6. Verify lifecycle and monitoring
|
||||
|
||||
- Confirm the action is inside its validity window.
|
||||
- Verify monitoring-event signatures or integrity evidence when available.
|
||||
- Check sequence numbers, previous-event digests, and expected heartbeat
|
||||
cadence.
|
||||
- Treat missing, stale, reordered, or broken-chain telemetry as a failure when
|
||||
policy requires continuous monitoring.
|
||||
- Do not interpret silence as health.
|
||||
|
||||
## Challenge convenient conclusions
|
||||
|
||||
Before producing the final result, attempt these mutations mentally or with
|
||||
project-provided test fixtures:
|
||||
|
||||
1. Replace a blocked assessment with an allowed result.
|
||||
2. Change one approved target, parameter, scope, amount, or commit.
|
||||
3. Reuse an otherwise valid approval nonce.
|
||||
4. Replace independent reviewers with correlated copies.
|
||||
5. Remove one refuting evidence item.
|
||||
6. Stop the monitoring heartbeat after approval.
|
||||
|
||||
If any mutation would pass the reviewed controls, record the affected control
|
||||
as `FAIL`; do not merely recommend future hardening.
|
||||
|
||||
## Determine the review result
|
||||
|
||||
Use exactly one result:
|
||||
|
||||
- `ELIGIBLE_FOR_HUMAN_DECISION`: all required controls pass.
|
||||
- `ELIGIBLE_WITH_CONTROLS`: no required control fails, and explicit external
|
||||
controls can resolve the listed conditions before execution.
|
||||
- `BLOCKED`: at least one required control fails or the action exceeds the
|
||||
justified scope.
|
||||
- `INCONCLUSIVE`: no required control is proven false, but evidence needed for
|
||||
a safe decision is missing or unverifiable.
|
||||
|
||||
`ELIGIBLE_FOR_HUMAN_DECISION` is not approval. A human authority and a separate
|
||||
enforcement point remain responsible for any real action.
|
||||
|
||||
## Report in this format
|
||||
|
||||
```markdown
|
||||
# Agent Action Review
|
||||
|
||||
## Result
|
||||
- Review result: BLOCKED | INCONCLUSIVE | ELIGIBLE_WITH_CONTROLS |
|
||||
ELIGIBLE_FOR_HUMAN_DECISION
|
||||
- Execution authorized: false
|
||||
- Exact action digest: <verified value or NOT_VERIFIED>
|
||||
|
||||
## Action
|
||||
- Operation:
|
||||
- Target:
|
||||
- Material parameters:
|
||||
- Scope:
|
||||
- Validity window:
|
||||
- Maximum uses:
|
||||
|
||||
## Control matrix
|
||||
| Control | Status | Evidence | Reason |
|
||||
|---|---|---|---|
|
||||
| Recomputed assessment | PASS/FAIL/INCONCLUSIVE/N/A | ... | ... |
|
||||
| Exact action binding | ... | ... | ... |
|
||||
| Replay and identity | ... | ... | ... |
|
||||
| Reviewer independence | ... | ... | ... |
|
||||
| Evidence completeness | ... | ... | ... |
|
||||
| Monitoring freshness | ... | ... | ... |
|
||||
|
||||
## Supporting evidence
|
||||
- ...
|
||||
|
||||
## Refuting evidence and defeaters
|
||||
- ...
|
||||
|
||||
## Required next action
|
||||
- State the smallest concrete step that could change the result.
|
||||
|
||||
## Boundaries
|
||||
- State what this review did not prove.
|
||||
```
|
||||
|
||||
Lead with the result and the exact reason. Prefer a reproducible blocker over a
|
||||
confidence score.
|
||||
Reference in New Issue
Block a user