mirror of
https://github.com/github/awesome-copilot.git
synced 2026-05-15 11:11:48 +00:00
chore: sync Arize skills from arize-skills@6a622b6c962907f54ca3578cb2cabff161d8aae6 and phoenix@30ccbe6b38cc83719038bf30041335f29bae45e9 (#1690)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This commit is contained in:
Jim Bennett
GitHub
@@ -1,6 +1,10 @@
|
||||
---
|
||||
name: arize-instrumentation
|
||||
description: "INVOKE THIS SKILL when adding Arize AX tracing or observability to an app for the first time, or when the user wants to instrument their LLM app or get started with LLM observability. Follow the Agent-Assisted Tracing two-phase flow: analyze the codebase (read-only), then implement after user confirmation. When the app uses LLM tool/function calling, add manual CHAIN + TOOL spans. Leverages https://arize.com/docs/ax/alyx/tracing-assistant and https://arize.com/docs/PROMPT.md."
|
||||
description: Adds Arize AX tracing to an LLM application for the first time. Follows a two-phase agent-assisted flow to analyze the codebase then implement instrumentation after user confirmation. Use when the user wants to instrument their app, add tracing from scratch, set up LLM observability, integrate OpenTelemetry or openinference, or get started with Arize tracing.
|
||||
metadata:
|
||||
author: arize
|
||||
version: "1.0"
|
||||
compatibility: Python and TypeScript/JavaScript apps use openinference-instrumentation packages for auto-instrumentation. Java and Go apps use the OpenTelemetry SDK with manual OpenInference spans. See https://arize.com/docs/PROMPT.md for setup details.
|
||||
---
|
||||
|
||||
# Arize Instrumentation Skill
|
||||
@@ -46,6 +50,7 @@ Before changing code:
|
||||
- Python: `pyproject.toml`, `requirements.txt`, `setup.py`, `Pipfile`
|
||||
- TypeScript/JavaScript: `package.json`
|
||||
- Java: `pom.xml`, `build.gradle`, `build.gradle.kts`
|
||||
- Go: `go.mod`
|
||||
|
||||
2. **Scan import statements** in source files to confirm what is actually used.
|
||||
|
||||
@@ -57,8 +62,8 @@ Before changing code:
|
||||
|
||||
| Item | Examples |
|
||||
|------|----------|
|
||||
| Language | Python, TypeScript/JavaScript, Java |
|
||||
| Package manager | pip/poetry/uv, npm/pnpm/yarn, maven/gradle |
|
||||
| Language | Python, TypeScript/JavaScript, Java, Go |
|
||||
| Package manager | pip/poetry/uv, npm/pnpm/yarn, maven/gradle, go modules |
|
||||
| LLM providers | OpenAI, Anthropic, LiteLLM, Bedrock, etc. |
|
||||
| Frameworks | LangChain, LangGraph, LlamaIndex, Vercel AI SDK, Mastra, etc. |
|
||||
| Existing tracing | Any OTel or vendor setup |
|
||||
@@ -86,8 +91,9 @@ The **canonical list** of supported integrations and doc URLs is in the [Agent S
|
||||
- **Python frameworks:** [LangChain](https://arize.com/docs/ax/integrations/python-agent-frameworks/langchain), [LangGraph](https://arize.com/docs/ax/integrations/python-agent-frameworks/langgraph), [LlamaIndex](https://arize.com/docs/ax/integrations/python-agent-frameworks/llamaindex), [CrewAI](https://arize.com/docs/ax/integrations/python-agent-frameworks/crewai), [DSPy](https://arize.com/docs/ax/integrations/python-agent-frameworks/dspy), [AutoGen](https://arize.com/docs/ax/integrations/python-agent-frameworks/autogen), [Semantic Kernel](https://arize.com/docs/ax/integrations/python-agent-frameworks/semantic-kernel), [Pydantic AI](https://arize.com/docs/ax/integrations/python-agent-frameworks/pydantic), [Haystack](https://arize.com/docs/ax/integrations/python-agent-frameworks/haystack), [Guardrails AI](https://arize.com/docs/ax/integrations/python-agent-frameworks/guardrails-ai), [Hugging Face Smolagents](https://arize.com/docs/ax/integrations/python-agent-frameworks/hugging-face-smolagents), [Instructor](https://arize.com/docs/ax/integrations/python-agent-frameworks/instructor), [Agno](https://arize.com/docs/ax/integrations/python-agent-frameworks/agno), [Google ADK](https://arize.com/docs/ax/integrations/python-agent-frameworks/google-adk), [MCP](https://arize.com/docs/ax/integrations/python-agent-frameworks/model-context-protocol), [Portkey](https://arize.com/docs/ax/integrations/python-agent-frameworks/portkey), [Together AI](https://arize.com/docs/ax/integrations/python-agent-frameworks/together-ai), [BeeAI](https://arize.com/docs/ax/integrations/python-agent-frameworks/beeai), [AWS Bedrock Agents](https://arize.com/docs/ax/integrations/python-agent-frameworks/aws).
|
||||
- **TypeScript/JavaScript:** [LangChain JS](https://arize.com/docs/ax/integrations/ts-js-agent-frameworks/langchain), [Mastra](https://arize.com/docs/ax/integrations/ts-js-agent-frameworks/mastra), [Vercel AI SDK](https://arize.com/docs/ax/integrations/ts-js-agent-frameworks/vercel), [BeeAI JS](https://arize.com/docs/ax/integrations/ts-js-agent-frameworks/beeai).
|
||||
- **Java:** [LangChain4j](https://arize.com/docs/ax/integrations/java/langchain4j), [Spring AI](https://arize.com/docs/ax/integrations/java/spring-ai), [Arconia](https://arize.com/docs/ax/integrations/java/arconia).
|
||||
- **Go:** No first-party auto-instrumentation packages today — use the OpenTelemetry Go SDK with manual [OpenInference](https://github.com/Arize-ai/openinference) attributes per [Manual instrumentation](https://arize.com/docs/ax/instrument/manual-instrumentation).
|
||||
- **Platforms (UI-based):** [LangFlow](https://arize.com/docs/ax/integrations/platforms/langflow), [Flowise](https://arize.com/docs/ax/integrations/platforms/flowise), [Dify](https://arize.com/docs/ax/integrations/platforms/dify), [Prompt flow](https://arize.com/docs/ax/integrations/platforms/prompt-flow).
|
||||
- **Fallback:** [Manual instrumentation](https://arize.com/docs/ax/observe/tracing/setup/manual-instrumentation), [All integrations](https://arize.com/docs/ax/integrations).
|
||||
- **Fallback:** [Manual instrumentation](https://arize.com/docs/ax/instrument/manual-instrumentation), [All integrations](https://arize.com/docs/ax/integrations).
|
||||
|
||||
**Fetch the matched doc pages** from the [full routing table in PROMPT.md](https://arize.com/docs/PROMPT.md) for exact installation and code snippets. Use [llms.txt](https://arize.com/docs/llms.txt) as a fallback for doc discovery if needed.
|
||||
|
||||
@@ -104,13 +110,14 @@ Proceed **only after the user confirms** the Phase 1 analysis.
|
||||
- Python: `pip install arize-otel` plus `openinference-instrumentation-{name}` (hyphens in package name; underscores in import, e.g. `openinference.instrumentation.llama_index`).
|
||||
- TypeScript/JavaScript: `@opentelemetry/sdk-trace-node` plus the relevant `@arizeai/openinference-*` package.
|
||||
- Java: OpenTelemetry SDK plus `openinference-instrumentation-*` in pom.xml or build.gradle.
|
||||
- Go: `go get go.opentelemetry.io/otel go.opentelemetry.io/otel/sdk go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` — no auto-instrumentors yet, so the agent sets OpenInference attributes manually on spans. **Wire the exporter** with `otlptracehttp.WithEndpoint("otlp.arize.com")` (US) or `otlptracehttp.WithEndpoint("otlp.eu-west-1a.arize.com")` (EU) — pass the bare hostname, no `https://` scheme — and `otlptracehttp.WithHeaders(map[string]string{"space_id": ..., "api_key": ...})`. Recent OTel Go modules require Go ≥ 1.23 — `go mod tidy` may bump the toolchain.
|
||||
3. **Credentials** — User needs an **Arize API Key** and **Space ID**. Check existing `ax` profiles for `ARIZE_API_KEY` and `ARIZE_SPACE` — never read `.env` files:
|
||||
- Run `ax profiles show` to check for an existing profile.
|
||||
- If no profile exists, guide the user to run `ax profiles create` which provides an **interactive wizard** that walks through API key and space setup. See [CLI profiles docs](https://arize.com/docs/api-clients/cli/profiles) for details.
|
||||
- If the user needs to find their API key manually, direct them to **https://app.arize.com** and to navigate to the settings page (do not use organization-specific URLs with placeholder IDs — they won't resolve for new users).
|
||||
- If credentials are not set, instruct the user to set them as environment variables — never embed raw values in generated code. All generated instrumentation code must reference `os.environ["ARIZE_API_KEY"]` (Python) or `process.env.ARIZE_API_KEY` (TypeScript/JavaScript).
|
||||
- If credentials are not set, instruct the user to set them as environment variables — never embed raw values in generated code. All generated instrumentation code must reference `os.environ["ARIZE_API_KEY"]` (Python), `process.env.ARIZE_API_KEY` (TypeScript/JavaScript), or `os.Getenv("ARIZE_API_KEY")` (Go).
|
||||
- See references/ax-profiles.md for full profile setup and troubleshooting.
|
||||
4. **Centralized instrumentation** — Create a single module (e.g. `instrumentation.py`, `instrumentation.ts`) and initialize tracing **before** any LLM client is created.
|
||||
4. **Centralized instrumentation** — Create a single module (e.g. `instrumentation.py`, `instrumentation.ts`, `instrumentation.go`) and initialize tracing **before** any LLM client is created.
|
||||
5. **Existing OTel** — If there is already a TracerProvider, add Arize as an **additional** exporter (e.g. BatchSpanProcessor with Arize OTLP). Do not replace existing setup unless the user asks.
|
||||
|
||||
### Implementation rules
|
||||
@@ -119,8 +126,11 @@ Proceed **only after the user confirms** the Phase 1 analysis.
|
||||
- Prefer the repo's native integration surface before adding generic OpenTelemetry plumbing. If the framework ships an exporter or observability package, use that first unless there is a documented gap.
|
||||
- **Fail gracefully** if env vars are missing (warn, do not crash).
|
||||
- **Import order:** register tracer → attach instrumentors → then create LLM clients.
|
||||
- **Project name attribute (required):** Arize rejects spans with HTTP 500 if the project name is missing — `service.name` alone is not accepted. Set it as a **resource attribute** on the TracerProvider (recommended — one place, applies to all spans): Python: `register(project_name="my-app")` handles it automatically (sets `"openinference.project.name"` on the resource); TypeScript: Arize accepts both `"model_id"` (shown in the official TS quickstart) and `"openinference.project.name"` via `SEMRESATTRS_PROJECT_NAME` from `@arizeai/openinference-semantic-conventions` (shown in the manual instrumentation docs) — both work. For routing spans to different projects in Python, use `set_routing_context(space_id=..., project_name=...)` from `arize.otel`.
|
||||
- **CLI/script apps — flush before exit:** `provider.shutdown()` (TS) / `provider.force_flush()` then `provider.shutdown()` (Python) must be called before the process exits, otherwise async OTLP exports are dropped and no traces appear.
|
||||
- **Project name attribute (required):** Arize rejects spans with HTTP 500 if the project name is missing — `service.name` alone is not accepted. Set it as a **resource attribute** on the TracerProvider (recommended — one place, applies to all spans):
|
||||
- **Python:** `register(project_name="my-app")` handles it automatically (sets `"openinference.project.name"` on the resource). For routing spans to different projects, use `set_routing_context(space_id=..., project_name=...)` from `arize.otel`.
|
||||
- **TypeScript:** Arize accepts both `"model_id"` (shown in the official TS quickstart) and `"openinference.project.name"` via `SEMRESATTRS_PROJECT_NAME` from `@arizeai/openinference-semantic-conventions` (shown in the manual instrumentation docs) — both work.
|
||||
- **Go:** Pass `attribute.String("openinference.project.name", "my-app")` to `resource.New(...)` and apply via `sdktrace.WithResource(res)`. The Go SDK has no helper for this, so it must be set manually on every TracerProvider.
|
||||
- **CLI/script apps — flush before exit:** `provider.shutdown()` (TS) / `provider.force_flush()` then `provider.shutdown()` (Python) / `tp.Shutdown(ctx)` (Go) must be called before the process exits, otherwise async OTLP exports are dropped and no traces appear.
|
||||
- **When the app has tool/function execution:** add manual CHAIN + TOOL spans (see **Enriching traces** below) so the trace tree shows each tool call and its result — otherwise traces will look sparse (only LLM API spans, no tool input/output).
|
||||
|
||||
## Enriching traces: manual spans for tool use and agent loops
|
||||
@@ -151,10 +161,26 @@ To avoid sparse traces where tool inputs/outputs are missing:
|
||||
|
||||
| Attribute | Use |
|
||||
|-----------|-----|
|
||||
| `openinference.span.kind` | `"CHAIN"` or `"TOOL"` |
|
||||
| `openinference.span.kind` | Pick the right value: `"LLM"` for raw provider API calls (OpenAI, Anthropic, etc.); `"CHAIN"` for orchestration / agent-loop boundaries; `"TOOL"` for tool/function execution; `"RETRIEVER"` for vector-store / search lookups; `"EMBEDDING"` for embedding API calls; `"AGENT"` for an autonomous sub-agent run nested inside a larger chain; `"RERANKER"` for rerank API calls; `"GUARDRAIL"` for guardrail/policy checks; `"EVALUATOR"` for online eval calls. |
|
||||
| `input.value` | string (e.g. user message or JSON of tool args) |
|
||||
| `output.value` | string (e.g. final reply or JSON of tool result) |
|
||||
|
||||
**LLM-span attributes (set these in addition to the three above when the span is an actual LLM call):**
|
||||
|
||||
| Attribute | Use |
|
||||
|-----------|-----|
|
||||
| `llm.model_name` | model identifier (e.g. `"gpt-4o-mini"`) |
|
||||
| `llm.provider` / `llm.system` | provider name (e.g. `"openai"`, `"anthropic"`) |
|
||||
| `llm.input_messages.{i}.message.role` | `"system"` / `"user"` / `"assistant"` / `"tool"` for the i-th input message |
|
||||
| `llm.input_messages.{i}.message.content` | text content of the i-th input message |
|
||||
| `llm.output_messages.{i}.message.role` | role of the i-th output message |
|
||||
| `llm.output_messages.{i}.message.content` | text content of the i-th output message |
|
||||
| `llm.token_count.prompt` | int — prompt/input tokens |
|
||||
| `llm.token_count.completion` | int — completion/output tokens |
|
||||
| `llm.token_count.total` | int — total tokens |
|
||||
|
||||
In Python and TypeScript these names are exposed via `openinference-semantic-conventions` packages; in Go they must be hand-typed as the strings above.
|
||||
|
||||
**Python pattern:** Get the global tracer (same provider as Arize), then use context managers so tool spans are children of the CHAIN span and appear in the same trace as the LLM spans:
|
||||
|
||||
```python
|
||||
@@ -177,7 +203,51 @@ with tracer.start_as_current_span("run_agent") as chain_span:
|
||||
chain_span.set_attribute("output.value", final_reply)
|
||||
```
|
||||
|
||||
See [Manual instrumentation](https://arize.com/docs/ax/observe/tracing/setup/manual-instrumentation) for more span kinds and attributes.
|
||||
**Go pattern:** Get a tracer from the global TracerProvider (registered via `otel.SetTracerProvider`), then nest spans with `tracer.Start` so tool spans become children of the CHAIN span.
|
||||
|
||||
> **Critical for short-lived processes:** never call `log.Fatalf` / `os.Exit` after a span has started — they skip the deferred `tp.Shutdown(ctx)` and the in-flight CHAIN/LLM spans never flush. Use `log.Printf` + `return` from `main` instead, and keep `tp.Shutdown(ctx)` deferred at the top of `main`.
|
||||
|
||||
```go
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
)
|
||||
|
||||
var tracer = otel.Tracer("my-app")
|
||||
|
||||
func runAgent(ctx context.Context, userMessage string) string {
|
||||
ctx, chainSpan := tracer.Start(ctx, "run_agent")
|
||||
defer chainSpan.End()
|
||||
chainSpan.SetAttributes(
|
||||
attribute.String("openinference.span.kind", "CHAIN"),
|
||||
attribute.String("input.value", userMessage),
|
||||
)
|
||||
|
||||
// ... LLM call ...
|
||||
for _, toolUse := range toolUses {
|
||||
ctx, toolSpan := tracer.Start(ctx, toolUse.Name)
|
||||
argsJSON, err := json.Marshal(toolUse.Input)
|
||||
if err != nil {
|
||||
toolSpan.RecordError(err)
|
||||
}
|
||||
toolSpan.SetAttributes(
|
||||
attribute.String("openinference.span.kind", "TOOL"),
|
||||
attribute.String("input.value", string(argsJSON)),
|
||||
)
|
||||
result := runTool(toolUse.Name, toolUse.Input)
|
||||
toolSpan.SetAttributes(attribute.String("output.value", result))
|
||||
toolSpan.End()
|
||||
// ... append tool result to messages, call LLM again ...
|
||||
}
|
||||
|
||||
chainSpan.SetAttributes(attribute.String("output.value", finalReply))
|
||||
return finalReply
|
||||
}
|
||||
```
|
||||
|
||||
See [Manual instrumentation](https://arize.com/docs/ax/instrument/manual-instrumentation) for more span kinds and attributes.
|
||||
|
||||
## Verification
|
||||
|
||||
@@ -192,7 +262,7 @@ After implementation:
|
||||
|
||||
1. Run the application and trigger at least one LLM call.
|
||||
2. **Use the `arize-trace` skill** to confirm traces arrived. If empty, retry shortly. Verify spans have expected `openinference.span.kind`, `input.value`/`output.value`, and parent-child relationships.
|
||||
3. If no traces: verify `ARIZE_SPACE` and `ARIZE_API_KEY`, ensure tracer is initialized before instrumentors and clients, check connectivity to `otlp.arize.com:443`, and inspect app/runtime exporter logs so you can tell whether spans are being emitted locally but rejected remotely. For debug set `GRPC_VERBOSITY=debug` or pass `log_to_console=True` to `register()`. Common gotchas: (a) missing project name resource attribute causes HTTP 500 rejections — `service.name` alone is not enough; Python: pass `project_name` to `register()`; TypeScript: set `"model_id"` or `SEMRESATTRS_PROJECT_NAME` on the resource; (b) CLI/script processes exit before OTLP exports flush — call `provider.force_flush()` then `provider.shutdown()` before exit; (c) CLI-visible spaces/projects can disagree with a collector-targeted space ID — report the mismatch instead of silently rewriting credentials.
|
||||
3. If no traces: verify `ARIZE_SPACE` and `ARIZE_API_KEY`, ensure tracer is initialized before instrumentors and clients, check connectivity to `otlp.arize.com:443`, and inspect app/runtime exporter logs so you can tell whether spans are being emitted locally but rejected remotely. For debug set `GRPC_VERBOSITY=debug` or pass `log_to_console=True` to `register()`. Common gotchas: (a) missing project name resource attribute causes HTTP 500 rejections — `service.name` alone is not enough; Python: pass `project_name` to `register()`; TypeScript: set `"model_id"` or `SEMRESATTRS_PROJECT_NAME` on the resource; Go: add `attribute.String("openinference.project.name", "my-app")` to `resource.New(...)`; (b) CLI/script processes exit before OTLP exports flush — call `provider.force_flush()` then `provider.shutdown()` (Python/TS) or `tp.Shutdown(ctx)` (Go) before exit; (c) CLI-visible spaces/projects can disagree with a collector-targeted space ID — report the mismatch instead of silently rewriting credentials.
|
||||
4. If the app uses tools: confirm CHAIN and TOOL spans appear with `input.value` / `output.value` so tool calls and results are visible.
|
||||
|
||||
When verification is blocked by CLI or account issues, end with a concrete status:
|
||||
|
||||
Reference in New Issue
Block a user