mirror of
https://github.com/github/awesome-copilot.git
synced 2026-05-04 14:15:55 +00:00
chore: sync Arize skills from arize-skills@597d609bfe5f07fd7d24acfdb408a082911b18fc and phoenix@746247cbb07b0dc7803b87c69dd8c77811c33f59 (#1583)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: arize-instrumentation
|
||||
description: "INVOKE THIS SKILL when adding Arize AX tracing to an application. Follow the Agent-Assisted Tracing two-phase flow: analyze the codebase (read-only), then implement instrumentation after user confirmation. When the app uses LLM tool/function calling, add manual CHAIN + TOOL spans so traces show each tool's input and output. Leverages https://arize.com/docs/ax/alyx/tracing-assistant and https://arize.com/docs/PROMPT.md."
|
||||
description: "INVOKE THIS SKILL when adding Arize AX tracing or observability to an app for the first time, or when the user wants to instrument their LLM app or get started with LLM observability. Follow the Agent-Assisted Tracing two-phase flow: analyze the codebase (read-only), then implement after user confirmation. When the app uses LLM tool/function calling, add manual CHAIN + TOOL spans. Leverages https://arize.com/docs/ax/alyx/tracing-assistant and https://arize.com/docs/PROMPT.md."
|
||||
---
|
||||
|
||||
# Arize Instrumentation Skill
|
||||
@@ -104,7 +104,12 @@ Proceed **only after the user confirms** the Phase 1 analysis.
|
||||
- Python: `pip install arize-otel` plus `openinference-instrumentation-{name}` (hyphens in package name; underscores in import, e.g. `openinference.instrumentation.llama_index`).
|
||||
- TypeScript/JavaScript: `@opentelemetry/sdk-trace-node` plus the relevant `@arizeai/openinference-*` package.
|
||||
- Java: OpenTelemetry SDK plus `openinference-instrumentation-*` in pom.xml or build.gradle.
|
||||
3. **Credentials** — User needs **Arize Space ID** and **API Key** from [Space API Keys](https://app.arize.com/organizations/-/settings/space-api-keys). Check `.env` for `ARIZE_API_KEY` and `ARIZE_SPACE_ID`. If not found, instruct the user to set them as environment variables — never embed raw values in generated code. All generated instrumentation code must reference `os.environ["ARIZE_API_KEY"]` (Python) or `process.env.ARIZE_API_KEY` (TypeScript/JavaScript).
|
||||
3. **Credentials** — User needs an **Arize API Key** and **Space ID**. Check existing `ax` profiles for `ARIZE_API_KEY` and `ARIZE_SPACE` — never read `.env` files:
|
||||
- Run `ax profiles show` to check for an existing profile.
|
||||
- If no profile exists, guide the user to run `ax profiles create` which provides an **interactive wizard** that walks through API key and space setup. See [CLI profiles docs](https://arize.com/docs/api-clients/cli/profiles) for details.
|
||||
- If the user needs to find their API key manually, direct them to **https://app.arize.com** and to navigate to the settings page (do not use organization-specific URLs with placeholder IDs — they won't resolve for new users).
|
||||
- If credentials are not set, instruct the user to set them as environment variables — never embed raw values in generated code. All generated instrumentation code must reference `os.environ["ARIZE_API_KEY"]` (Python) or `process.env.ARIZE_API_KEY` (TypeScript/JavaScript).
|
||||
- See references/ax-profiles.md for full profile setup and troubleshooting.
|
||||
4. **Centralized instrumentation** — Create a single module (e.g. `instrumentation.py`, `instrumentation.ts`) and initialize tracing **before** any LLM client is created.
|
||||
5. **Existing OTel** — If there is already a TracerProvider, add Arize as an **additional** exporter (e.g. BatchSpanProcessor with Arize OTLP). Do not replace existing setup unless the user asks.
|
||||
|
||||
@@ -187,7 +192,7 @@ After implementation:
|
||||
|
||||
1. Run the application and trigger at least one LLM call.
|
||||
2. **Use the `arize-trace` skill** to confirm traces arrived. If empty, retry shortly. Verify spans have expected `openinference.span.kind`, `input.value`/`output.value`, and parent-child relationships.
|
||||
3. If no traces: verify `ARIZE_SPACE_ID` and `ARIZE_API_KEY`, ensure tracer is initialized before instrumentors and clients, check connectivity to `otlp.arize.com:443`, and inspect app/runtime exporter logs so you can tell whether spans are being emitted locally but rejected remotely. For debug set `GRPC_VERBOSITY=debug` or pass `log_to_console=True` to `register()`. Common gotchas: (a) missing project name resource attribute causes HTTP 500 rejections — `service.name` alone is not enough; Python: pass `project_name` to `register()`; TypeScript: set `"model_id"` or `SEMRESATTRS_PROJECT_NAME` on the resource; (b) CLI/script processes exit before OTLP exports flush — call `provider.force_flush()` then `provider.shutdown()` before exit; (c) CLI-visible spaces/projects can disagree with a collector-targeted space ID — report the mismatch instead of silently rewriting credentials.
|
||||
3. If no traces: verify `ARIZE_SPACE` and `ARIZE_API_KEY`, ensure tracer is initialized before instrumentors and clients, check connectivity to `otlp.arize.com:443`, and inspect app/runtime exporter logs so you can tell whether spans are being emitted locally but rejected remotely. For debug set `GRPC_VERBOSITY=debug` or pass `log_to_console=True` to `register()`. Common gotchas: (a) missing project name resource attribute causes HTTP 500 rejections — `service.name` alone is not enough; Python: pass `project_name` to `register()`; TypeScript: set `"model_id"` or `SEMRESATTRS_PROJECT_NAME` on the resource; (b) CLI/script processes exit before OTLP exports flush — call `provider.force_flush()` then `provider.shutdown()` before exit; (c) CLI-visible spaces/projects can disagree with a collector-targeted space ID — report the mismatch instead of silently rewriting credentials.
|
||||
4. If the app uses tools: confirm CHAIN and TOOL spans appear with `input.value` / `output.value` so tool calls and results are visible.
|
||||
|
||||
When verification is blocked by CLI or account issues, end with a concrete status:
|
||||
|
||||
@@ -54,7 +54,7 @@ ax profiles create work --api-key $ARIZE_API_KEY --region us-east-1b
|
||||
|
||||
To use a named profile with any `ax` command, add `-p NAME`:
|
||||
```bash
|
||||
ax spans export PROJECT_ID -p work
|
||||
ax spans export PROJECT -p work
|
||||
```
|
||||
|
||||
## 4. Getting the API key
|
||||
@@ -67,7 +67,7 @@ If `ARIZE_API_KEY` is not already set, instruct the user to export it in their s
|
||||
export ARIZE_API_KEY="..." # user pastes their key here in their own terminal
|
||||
```
|
||||
|
||||
They can find their key at https://app.arize.com/admin > API Keys. Recommend they create a **scoped service key** (not a personal user key) — service keys are not tied to an individual account and are safer for programmatic use. Keys are space-scoped — make sure they copy the key for the correct space.
|
||||
They can find their key at https://app.arize.com by navigating to the settings page. Recommend they create a **scoped service key** (not a personal user key) — service keys are not tied to an individual account and are safer for programmatic use. Keys are space-scoped — make sure they copy the key for the correct space.
|
||||
|
||||
Once the user confirms the variable is set, proceed with `ax profiles create --api-key $ARIZE_API_KEY` or `ax profiles update --api-key $ARIZE_API_KEY` as described above.
|
||||
|
||||
@@ -81,19 +81,19 @@ ax profiles show
|
||||
|
||||
Confirm the API key and region are correct, then retry the original command.
|
||||
|
||||
## Space ID
|
||||
## Space
|
||||
|
||||
There is no profile flag for space ID. Save it as an environment variable:
|
||||
There is no profile flag for space. Save it as an environment variable — accepts a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list -o json`.
|
||||
|
||||
**macOS/Linux** — add to `~/.zshrc` or `~/.bashrc`:
|
||||
```bash
|
||||
export ARIZE_SPACE_ID="U3BhY2U6..."
|
||||
export ARIZE_SPACE="my-workspace" # name or base64 ID
|
||||
```
|
||||
Then `source ~/.zshrc` (or restart terminal).
|
||||
|
||||
**Windows (PowerShell):**
|
||||
```powershell
|
||||
[System.Environment]::SetEnvironmentVariable('ARIZE_SPACE_ID', 'U3BhY2U6...', 'User')
|
||||
[System.Environment]::SetEnvironmentVariable('ARIZE_SPACE', 'my-workspace', 'User')
|
||||
```
|
||||
Restart terminal for it to take effect.
|
||||
|
||||
@@ -103,8 +103,8 @@ At the **end of the session**, if the user manually provided any credentials dur
|
||||
|
||||
**Skip this entirely if:**
|
||||
- The API key was already loaded from an existing profile or `ARIZE_API_KEY` env var
|
||||
- The space ID was already set via `ARIZE_SPACE_ID` env var
|
||||
- The user only used base64 project IDs (no space ID was needed)
|
||||
- The space was already set via `ARIZE_SPACE` env var
|
||||
- The user only used base64 project IDs (no space was needed)
|
||||
|
||||
**How to offer:** Use **AskQuestion**: *"Would you like to save your Arize credentials so you don't have to enter them next time?"* with options `"Yes, save them"` / `"No thanks"`.
|
||||
|
||||
@@ -112,4 +112,4 @@ At the **end of the session**, if the user manually provided any credentials dur
|
||||
|
||||
1. **API key** — Run `ax profiles show` to check the current state. Then run `ax profiles create --api-key $ARIZE_API_KEY` or `ax profiles update --api-key $ARIZE_API_KEY` (the key must already be exported as an env var — never pass a raw key value).
|
||||
|
||||
2. **Space ID** — See the Space ID section above to persist it as an environment variable.
|
||||
2. **Space** — See the Space section above to persist it as an environment variable.
|
||||
|
||||
Reference in New Issue
Block a user