mirror of https://github.com/github/awesome-copilot.git synced 2026-08-16 14:16:56 +00:00

Files

T

Stephen Benjamin 4214189287 fix: repair broken skill references, fence nesting, and invisible unicode (#2568 )

While implementing the Agent Plugin v1 spec in https://github.com/stbenjam/skillsaw/pull/483, I used this repository as a canonical real-world source for tests. Along the way its checks surfaced genuine correctness issues worth fixing regardless of tooling:

- Internal links that do not resolve (22): wrong relative paths, links whose targets are runtime output paths rather than bundled files (now inline code), a schemeless external URL parsed as a relative path, and links to files that do not exist (removed).

- Bundled skill files never referenced from their SKILL.md (14): each is now cited from the section where it is used, so nothing ships unreachable from the skill's instructions.

- Unclosed or mis-nested code fences (11): markdown templates embedding code blocks now use four-backtick outer fences so inner fences nest instead of closing the template early and leaking example content as live markdown; also removed stray orphan fences and closed an unterminated block at end of file.

- Invisible unicode (1): removed a zero-width space from a code example, a copy-paste artifact invisible to reviewers.

2026-08-07 02:01:43 +00:00

4.5 KiB

Raw Blame History

name, description, license, compatibility, metadata

name

description

license

compatibility

metadata

phoenix-evals

Build and run evaluators for AI/LLM applications using Phoenix.

Apache-2.0

Requires Phoenix server. Python skills need phoenix and openai packages; TypeScript skills need @arizeai/phoenix-client.

author	version	languages
oss@arize.com	1.0.0	Python, TypeScript

Phoenix Evals

Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.

Quick Reference

Task	Files
Setup	setup-python, setup-typescript
Decide what to evaluate	evaluators-overview
Choose a judge model	fundamentals-model-selection
Use pre-built evaluators	evaluators-pre-built
Build code evaluator	evaluators-code-python, evaluators-code-typescript
Build LLM evaluator	evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates
Batch evaluate DataFrame	evaluate-dataframe-python
Understand experiments	experiments-overview
Run experiment	experiments-running-python, experiments-running-typescript
Create dataset	experiments-datasets-python, experiments-datasets-typescript
Generate synthetic data	experiments-synthetic-python, experiments-synthetic-typescript
Validate evaluator accuracy	validation, validation-evaluators-python, validation-evaluators-typescript
Sample traces for review	observe-sampling-python, observe-sampling-typescript
Analyze errors	error-analysis, error-analysis-multi-turn, axial-coding
RAG evals	evaluators-rag
Avoid common mistakes	common-mistakes-python, fundamentals-anti-patterns
Production	production-overview, production-guardrails, production-continuous

Workflows

Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview

Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}

RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)

Production: production-overview → production-guardrails → production-continuous

Reference Categories

Prefix	Description
`fundamentals-*`	Types, scores, anti-patterns
`observe-*`	Tracing, sampling
`error-analysis-*`	Finding failures
`axial-coding-*`	Categorizing failures
`evaluators-*`	Code, LLM, RAG evaluators
`experiments-*`	Datasets, running experiments
`validation-*`	Validating evaluator accuracy against human labels
`production-*`	CI/CD, monitoring

Key Principles

Principle	Action
Error analysis first	Can't automate what you haven't observed
Custom > generic	Build from your failures
Code first	Deterministic before LLM
Validate judges	>80% TPR/TNR
Binary > Likert	Pass/fail, not 1-5

4.5 KiB Raw Blame History