Awesome/awesome-copilot
1
0
Fork 0
mirror of https://github.com/github/awesome-copilot.git synced 2026-05-04 22:25:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: sync Arize skills from arize-skills@597d609bfe5f07fd7d24acfdb408a082911b18fc and phoenix@746247cbb07b0dc7803b87c69dd8c77811c33f59 (#1583)

Browse Source 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This commit is contained in:
Jim Bennett
2026-05-03 18:05:44 -07:00
committed by GitHub
parent 82b58047e0
commit c7b2aecb94
40 changed files with 1316 additions and 423 deletions
Download Patch File Download Diff File Expand all files Collapse all files

76
skills/arize-ai-provider-integration/SKILL.md
View File

@@ -5,6 +5,9 @@ description: "INVOKE THIS SKILL when creating, reading, updating, or deleting Ar
# Arize AI Integration Skill
> **`SPACE`** — Most `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`.
> **Note:** `ai-integrations create` does **not** accept `--space` — AI integrations are account-scoped. Use `--space` only with `list`, `get`, `update`, and `delete`.
## Concepts
- **AI Integration** = stored LLM provider credentials registered in Arize; used by evaluators to call a judge model and by other Arize features that need to invoke an LLM on your behalf
@@ -19,9 +22,10 @@ Proceed directly with the task — run the `ax` command you need. Do NOT check v
If an `ax` command fails, troubleshoot based on the error:
- `command not found` or version error → see references/ax-setup.md
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via references/ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
- Space ID unknown → check `.env` for `ARIZE_SPACE_ID`, or run `ax spaces list -o json`, or ask the user
- LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → check `.env`, load if present, otherwise ask the user
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
- Space unknown → run `ax spaces list` to pick by name, or ask the user
- LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the **arize-ai-provider-integration** skill
- **Security:** Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. If credentials are not available through these channels, ask the user.
---
@@ -30,32 +34,32 @@ If an `ax` command fails, troubleshoot based on the error:
List all integrations accessible in a space:
```bash
ax ai-integrations list --space-id SPACE_ID
ax ai-integrations list --space SPACE
```
Filter by name (case-insensitive substring match):
```bash
ax ai-integrations list --space-id SPACE_ID --name "openai"
ax ai-integrations list --space SPACE --name "openai"
```
Paginate large result sets:
```bash
# Get first page
ax ai-integrations list --space-id SPACE_ID --limit 20 -o json
ax ai-integrations list --space SPACE --limit 20 -o json
# Get next page using cursor from previous response
ax ai-integrations list --space-id SPACE_ID --limit 20 --cursor CURSOR_TOKEN -o json
ax ai-integrations list --space SPACE --limit 20 --cursor CURSOR_TOKEN -o json
```
**Key flags:**
| Flag | Description |
|------|-------------|
| `--space-id` | Space to list integrations in |
| `--space` | Space name or ID to filter integrations |
| `--name` | Case-insensitive substring filter on integration name |
| `--limit` | Max results (1–100, default 50) |
| `--limit` | Max results (1–100, default 15) |
| `--cursor` | Pagination token from a previous response |
| `-o, --output` | Output format: `table` (default) or `json` |
@@ -77,8 +81,9 @@ ax ai-integrations list --space-id SPACE_ID --limit 20 --cursor CURSOR_TOKEN -o
## Get a Specific Integration
```bash
ax ai-integrations get INT_ID
ax ai-integrations get INT_ID -o json
ax ai-integrations get NAME_OR_ID
ax ai-integrations get NAME_OR_ID -o json
ax ai-integrations get NAME_OR_ID --space SPACE # required when using name instead of ID
```
Use this to inspect an integration's full configuration or to confirm its ID after creation.
@@ -90,7 +95,7 @@ Use this to inspect an integration's full configuration or to confirm its ID aft
Before creating, always list integrations first — the user may already have a suitable one:
```bash
ax ai-integrations list --space-id SPACE_ID
ax ai-integrations list --space SPACE
```
If no suitable integration exists, create one. The required flags depend on the provider.
@@ -125,25 +130,24 @@ ax ai-integrations create \
### AWS Bedrock
AWS Bedrock uses IAM role-based auth instead of an API key. Provide the ARN of the role Arize should assume:
AWS Bedrock uses IAM role-based auth. Provide the ARN of the role Arize should assume via `--provider-metadata`:
```bash
ax ai-integrations create \
--name "My Bedrock Integration" \
--provider awsBedrock \
--role-arn "arn:aws:iam::123456789012:role/ArizeBedrockRole"
--provider-metadata '{"role_arn": "arn:aws:iam::123456789012:role/ArizeBedrockRole"}'
```
### Vertex AI
Vertex AI uses GCP service account credentials. Provide the GCP project and region:
Vertex AI uses GCP service account credentials. Provide the GCP project and region via `--provider-metadata`:
```bash
ax ai-integrations create \
--name "My Vertex AI Integration" \
--provider vertexAI \
--project-id "my-gcp-project" \
--location "us-central1"
--provider-metadata '{"project_id": "my-gcp-project", "location": "us-central1"}'
```
### Gemini
@@ -182,8 +186,8 @@ ax ai-integrations create \
| `openAI` | `--api-key <key>` |
| `anthropic` | `--api-key <key>` |
| `azureOpenAI` | `--api-key <key>`, `--base-url <azure-endpoint>` |
| `awsBedrock` | `--role-arn <arn>` |
| `vertexAI` | `--project-id <gcp-project>`, `--location <region>` |
| `awsBedrock` | `--provider-metadata '{"role_arn": "<arn>"}'` |
| `vertexAI` | `--provider-metadata '{"project_id": "<gcp-project>", "location": "<region>"}'` |
| `gemini` | `--api-key <key>` |
| `nvidiaNim` | `--api-key <key>`, `--base-url <nim-endpoint>` |
| `custom` | `--base-url <endpoint>` |
@@ -192,18 +196,21 @@ ax ai-integrations create \
| Flag | Description |
|------|-------------|
| `--model-names` | Comma-separated list of allowed model names; omit to allow all models |
| `--enable-default-models` / `--no-default-models` | Enable or disable the provider's default model list |
| `--function-calling` / `--no-function-calling` | Enable or disable tool/function calling support |
| `--model-name` | Allowed model name (repeat for multiple, e.g. `--model-name gpt-4o --model-name gpt-4o-mini`); omit to allow all models |
| `--enable-default-models` | Enable the provider's default model list |
| `--function-calling-enabled` | Enable tool/function calling support |
| `--auth-type` | Authentication type: `default`, `proxy_with_headers`, or `bearer_token` |
| `--headers` | Custom headers as JSON object or file path (for proxy auth) |
| `--provider-metadata` | Provider-specific metadata as JSON object or file path |
### After creation
Capture the returned integration ID (e.g., `TGxtSW50ZWdyYXRpb246MTI6YUJjRA==`) — it is needed for evaluator creation and other downstream commands. If you missed it, retrieve it:
```bash
ax ai-integrations list --space-id SPACE_ID -o json
# or, if you know the ID:
ax ai-integrations get INT_ID
ax ai-integrations list --space SPACE -o json
# or by name/ID directly:
ax ai-integrations get NAME_OR_ID
```
---
@@ -214,19 +221,19 @@ ax ai-integrations get INT_ID
```bash
# Rename
ax ai-integrations update INT_ID --name "New Name"
ax ai-integrations update NAME_OR_ID --name "New Name"
# Rotate the API key
ax ai-integrations update INT_ID --api-key $OPENAI_API_KEY
ax ai-integrations update NAME_OR_ID --api-key $OPENAI_API_KEY
# Change the model list
ax ai-integrations update INT_ID --model-names "gpt-4o,gpt-4o-mini"
# Change the model list (replaces all existing model names)
ax ai-integrations update NAME_OR_ID --model-name gpt-4o --model-name gpt-4o-mini
# Update base URL (for Azure, custom, or NIM)
ax ai-integrations update INT_ID --base-url "https://new-endpoint.example.com/v1"
ax ai-integrations update NAME_OR_ID --base-url "https://new-endpoint.example.com/v1"
```
Any flag accepted by `create` can be passed to `update`.
Add `--space SPACE` when using a name instead of ID. Any flag accepted by `create` can be passed to `update`.
---
@@ -235,7 +242,8 @@ Any flag accepted by `create` can be passed to `update`.
**Warning:** Deletion is permanent. Evaluators that reference this integration will no longer be able to run.
```bash
ax ai-integrations delete INT_ID --force
ax ai-integrations delete NAME_OR_ID --force
ax ai-integrations delete NAME_OR_ID --space SPACE --force # required when using name instead of ID
```
Omit `--force` to get a confirmation prompt instead of deleting immediately.
@@ -249,8 +257,8 @@ Omit `--force` to get a confirmation prompt instead of deleting immediately.
| `ax: command not found` | See references/ax-setup.md |
| `401 Unauthorized` | API key may not have access to this space. Verify key and space ID at https://app.arize.com/admin > API Keys |
| `No profile found` | Run `ax profiles show --expand`; set `ARIZE_API_KEY` env var or write `~/.arize/config.toml` |
| `Integration not found` | Verify with `ax ai-integrations list --space-id SPACE_ID` |
| `has_api_key: false` after create | Credentials were not saved — re-run `update` with the correct `--api-key` or `--role-arn` |
| `Integration not found` | Verify with `ax ai-integrations list --space SPACE` |
| `has_api_key: false` after create | Credentials were not saved — re-run `update` with the correct `--api-key` or `--provider-metadata` |
| Evaluator runs fail with LLM errors | Check integration credentials with `ax ai-integrations get INT_ID`; rotate the API key if needed |
| `provider` mismatch | Cannot change provider after creation — delete and recreate with the correct provider |

16
skills/arize-ai-provider-integration/references/ax-profiles.md
View File

@@ -54,7 +54,7 @@ ax profiles create work --api-key $ARIZE_API_KEY --region us-east-1b
To use a named profile with any `ax` command, add `-p NAME`:
```bash
ax spans export PROJECT_ID -p work
ax spans export PROJECT -p work
```
## 4. Getting the API key
@@ -81,19 +81,19 @@ ax profiles show
Confirm the API key and region are correct, then retry the original command.
## Space ID
## Space
There is no profile flag for space ID. Save it as an environment variable:
There is no profile flag for space. Save it as an environment variable — accepts a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list -o json`.
**macOS/Linux** — add to `~/.zshrc` or `~/.bashrc`:
```bash
export ARIZE_SPACE_ID="U3BhY2U6..."
export ARIZE_SPACE="my-workspace" # name or base64 ID
```
Then `source ~/.zshrc` (or restart terminal).
**Windows (PowerShell):**
```powershell
[System.Environment]::SetEnvironmentVariable('ARIZE_SPACE_ID', 'U3BhY2U6...', 'User')
[System.Environment]::SetEnvironmentVariable('ARIZE_SPACE', 'my-workspace', 'User')
```
Restart terminal for it to take effect.
@@ -103,8 +103,8 @@ At the **end of the session**, if the user manually provided any credentials dur
**Skip this entirely if:**
- The API key was already loaded from an existing profile or `ARIZE_API_KEY` env var
- The space ID was already set via `ARIZE_SPACE_ID` env var
- The user only used base64 project IDs (no space ID was needed)
- The space was already set via `ARIZE_SPACE` env var
- The user only used base64 project IDs (no space was needed)
**How to offer:** Use **AskQuestion**: *"Would you like to save your Arize credentials so you don't have to enter them next time?"* with options `"Yes, save them"` / `"No thanks"`.
@@ -112,4 +112,4 @@ At the **end of the session**, if the user manually provided any credentials dur
1. **API key** — Run `ax profiles show` to check the current state. Then run `ax profiles create --api-key $ARIZE_API_KEY` or `ax profiles update --api-key $ARIZE_API_KEY` (the key must already be exported as an env var — never pass a raw key value).
2. **Space ID** — See the Space ID section above to persist it as an environment variable.
2. **Space** — See the Space section above to persist it as an environment variable.

4
skills/arize-ai-provider-integration/references/ax-setup.md
View File

@@ -4,7 +4,7 @@ Consult this only when an `ax` command fails. Do NOT run these checks proactivel
## Check version first
If `ax` is installed (not `command not found`), always run `ax --version` before investigating further. The version must be `0.8.0` or higher — many errors are caused by an outdated install. If the version is too old, see **Version too old** below.
If `ax` is installed (not `command not found`), always run `ax --version` before investigating further. The version must be `0.14.0` or higher — many errors are caused by an outdated install. If the version is too old, see **Version too old** below.
## `ax: command not found`
@@ -19,7 +19,7 @@ If `ax` is installed (not `command not found`), always run `ax --version` before
3. Install: `pip install arize-ax-cli`
4. Add to PATH: `$env:PATH = "$env:APPDATA\Python\Scripts;$env:PATH"`
## Version too old (below 0.8.0)
## Version too old (below 0.14.0)
Upgrade: `uv tool install --force --reinstall arize-ax-cli`, `pipx upgrade arize-ax-cli`, or `pip install --upgrade arize-ax-cli`