Skip to content

Agents

If your agent harness supports external endpoint providers, you can configure your agent to utilize the ALCF Inference Service endpoints as a backend.

Installation

Most agents have an installation script that you can run with one command to install in ~/.local/bin.

Agent Install Command
OpenCode1 curl -fsSL https://opencode.ai/install | bash
Pi1 curl -fsSL https://pi.dev/install.sh | sh
OpenAI Codex curl -fsSL https://chatgpt.com/codex/install.sh | sh
Claude Code curl -fsSL https://claude.ai/install.sh | bash
Aider curl -LsSf https://aider.chat/install.sh | sh

Most scripts will require you to reload your environment with source ~/.bashrc before the agent will be in your PATH.

Alternatively, you can also install via your system package manager (i.e. brew, apt, etc.).

Codex Version Requirements

Codex 0.95 and later require a model and endpoint with Responses API support. For Chat Completions-only endpoints, use 0.94 or lower. If you use Nix, you can use this command to pull that version into an ephemeral shell: nix-shell -p codex -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/cc4bd5f859cdf01153d999995c20c9a457045bd6.tar.gz.

Helper-based token refresh requires Codex 0.118.0 or newer. Reusable external profiles require Codex 0.134.0 or newer. With older versions, use alcf-ai agent run codex for a one-off launch or alcf-ai agent configure codex --no-profile to update the base configuration.

Automatic Configuration w/ alcf-ai

You can quick-configure most agents to use the ALCF Inference Service endpoints with alcf-ai (installable with pip install alcf-ai). This also handles authentication and pulling an API key from the service. See alcf-ai CLI and SDK for installation details and additional options.

Run Without Changing Configuration

alcf-ai agent run launches an agent with temporary environment and command-line overrides using your cached ALCF credentials. It does not create or modify the agent's configuration files:

alcf-ai agent run <agent>

Supported agents include:

  • claude
  • opencode
  • codex
  • aider

Specify the model and serving cluster with --default-model and --default-cluster. Arguments after -- are forwarded to the agent:

alcf-ai agent run claude \
    --default-model mistral-large-4 \
    --default-cluster tara \
    -- --print "Hello"

Model selection uses --default-model, then $ALCF_AI_MODEL, then mistral-large-4. Cluster selection is independent: --default-cluster defaults to tara, so set it to the selected model's serving cluster when overriding the default model. OpenCode receives an inline provider configuration, Codex receives one-off provider overrides, and Aider receives its model and credentials through environment variables.

Persistent Configuration

To write a reusable configuration instead, run:

alcf-ai agent configure <agent>

Supported agents include:

  • opencode
  • pi
  • codex
  • claude

Codex and Claude Code create a reusable profile named alcf by default, without changing their normal base settings:

alcf-ai agent configure codex
alcf-ai agent run codex --profile alcf

alcf-ai agent configure claude
alcf-ai agent run claude --profile alcf

After creating a profile, you can also launch it directly without alcf-ai:

codex --profile alcf
claude --settings ~/.claude/profiles/alcf.json

Codex profiles are stored at $CODEX_HOME/alcf.config.toml (usually ~/.codex/alcf.config.toml) and require Codex 0.134.0 or newer. Claude Code profiles are stored at ~/.claude/profiles/alcf.json and loaded with Claude's --settings option. Unspecified settings continue to inherit from the normal configuration.

Use --profile <name> with both configure and run to create and load another profile. Select the model and cluster with agent configure. agent run --profile cannot be combined with --default-model or --default-cluster. Use agent configure --no-profile to update $CODEX_HOME/config.toml or ~/.claude/settings.json directly instead. OpenCode and Pi always update their normal configuration files and do not support profiles.

Option Description
-m, --default-model Default model for the agent configuration (codex and claude). Takes precedence over $ALCF_AI_MODEL. Without either, it defaults to mistral-large-4.
-c, --default-cluster Cluster serving the default model (codex and claude). Default: tara.
--profile <name> Settings profile for codex and claude. Default: alcf.
--no-profile Update the base configuration instead of creating a Codex or Claude Code profile.
-y, --yes Approve all configuration file overwrites without prompting.

Configuration Backups

Before replacing an existing configuration file with different contents, agent configure asks for confirmation and creates a timestamped backup after approval. Creating a new file or writing identical content does not prompt. Use -y or --yes to approve all overwrites non-interactively.

Refreshing API Keys

alcf-ai agent configure <agent> is idempotent, so you can re-run it to reconfigure an agent at any time. When a token helper (alcf-tokens or alcf-ai) is on your PATH, the generated configuration refreshes access tokens automatically. Otherwise, alcf-ai embeds the current access token and warns you to re-run the command when it expires.

Configuration discovers every authorized cluster and model exposed by the service and uses the published model capabilities, including context limits and supported input types. Re-run configuration to update the generated entries when model availability or permissions change. OpenCode and Pi expose configured models in their model pickers. Choose the appropriate cluster/provider, such as Tara Direct API (alcf-inference-service-tara-api).

Agent API selection Reasoning controls
OpenCode Chat Completions Variants from the model's published effort levels.
Pi Responses when advertised; otherwise Chat Completions Thinking levels mapped to the model's published effort levels.
Codex Responses with current Codex versions Generated catalogs retain conservative settings: OpenAI-specific reasoning summaries and built-in tools are disabled, with no effort selector.
Claude Code Messages Client-dependent effort controls; requests must use the model's accepted values.

Mistral Large 4 on Tara is the CLI default. Codex preserves an existing model/provider selection in the target profile and generates a separate alcf-<cluster>-<model> profile for each configured model, replacing / in model IDs with - in the profile name. Select that profile with alcf-ai agent run codex --profile <name> to switch models, and regenerate its version-matched catalog after upgrading Codex. Claude Code's --default-model option sets its model and Opus/Sonnet/Haiku aliases. If the option is omitted, Claude Code uses $ALCF_AI_MODEL, if set, or Mistral Large 4. See API reasoning fields for protocol-specific controls.

Manual Configuration

OpenCode

After installing opencode, place the following in your ~/.config/opencode/opencode.jsonc.

{
    ...
    "provider": {
        "alcf-inference-service-tara-api": {
            "name": "ALCF Inference Service (Tara)",
            "npm": "@ai-sdk/openai-compatible",
            "options": {
                "baseURL": "https://inference-api.alcf.anl.gov/resource_server/tara/api/v1",
                "apiKey": "{env:ALCF_AI_TOKEN}"
            },
            "models": {
                "mistral-large-4": {
                    "name": "mistral-large-4",
                    "timeout": false,
                    "limit": {
                        "context": 1048576,
                        "output": 0
                    }
                }
            }
        }
    }
    ...
}

Before running opencode, a valid token needs to be stored in the ALCF_AI_TOKEN environment variable. You can either set a key manually (see API Access) or utilize alcf-tokens.

alcf-tokens login inference # follow interactive instructions to login
export ALCF_AI_TOKEN="$(alcf-tokens get-token inference)" # pull a token and store

opencode

Pi

Add the following provider to your ~/.pi/agent/models.json.

{
    ...
    "providers": {
        "alcf-tara": {
            "baseUrl": "https://inference-api.alcf.anl.gov/resource_server/tara/api/v1",
                "api": "openai-completions",
                "apiKey": "!alcf-tokens get-token inference",
                "compat": {
                    "supportsDeveloperRole": false,
                    "supportsReasoningEffort": false
                },
                "models": [
                    { "id": "inkling-bf16" },
                    {
                        "id": "mistral-large-4",
                        "api": "openai-responses",
                        "contextWindow": 1048576,
                        "input": ["text", "image"],
                        "reasoning": true,
                        "thinkingLevelMap": {
                            "minimal": null,
                            "low": "low",
                            "medium": "medium",
                            "high": "high",
                            "xhigh": null,
                            "max": null
                        },
                        "compat": { "supportsReasoningEffort": true }
                    }
                ]
        }
    }
    ...
}

Codex

Choose a model with Responses and tool-calling support, served by a Direct API cluster with Responses support, from Available Models. Direct API endpoints with native Responses support need no Harmony-format translation proxy. Do not assume that OpenAI-specific built-in tools or reasoning-summary controls are supported by every model. Then set these variables in your terminal:

export model="<model>"
export cluster="<cluster>"
export ALCF_AI_TOKEN="$(alcf-tokens get-token inference)"

In your ~/.codex/config.toml, merge the following template with your existing settings. Replace <model> and <cluster> with the values selected above; TOML does not substitute these placeholders. The cluster selects the URL, while the model ID is sent in requests, not appended to the URL.

model = "<model>"
model_provider = "alcf-inference-service-<cluster>-api"

[model_providers."alcf-inference-service-<cluster>-api"]
name = "ALCF Inference Service (<cluster>)"
base_url = "https://inference-api.alcf.anl.gov/resource_server/<cluster>/api/v1"
env_key = "ALCF_AI_TOKEN"
wire_api = "responses"

In addition, you need to add a model_catalog_json for Codex to understand the model's capabilities. This script uses the exported model and cluster variables and reads the context limit and input types from the selected model's published capabilities. Catalog filenames include the cluster, with / in model IDs replaced by -.

Model Config Generation Script
#!/usr/bin/env bash
set -euo pipefail

: "${model:?Set model to the served model ID}"
: "${cluster:?Set cluster to the serving cluster}"
: "${ALCF_AI_TOKEN:?Set ALCF_AI_TOKEN to a valid inference access token}"

CODEX_VERSION="$(
  codex --version |
    awk '{print $NF}' |
    sed 's/^v//'
)"

CATALOG_DIR="$HOME/.codex/catalogs"
UPSTREAM_CATALOG="$CATALOG_DIR/models-${CODEX_VERSION}.upstream.json"
UPSTREAM_PROMPT="$CATALOG_DIR/prompt-${CODEX_VERSION}.md"
MODEL_CATALOG="$CATALOG_DIR/${cluster}-${model//\//-}.json"

capabilities="$(
  curl -fsS --get \
    -H "Authorization: Bearer $ALCF_AI_TOKEN" \
    --data-urlencode "model_id=$model" \
    "https://inference-api.alcf.anl.gov/resource_server/${cluster}/models" |
    jq -e --arg model "$model" '
      .[] | select(.id == $model and .framework == "api")
      | .capabilities | select(.api_protocols | index("responses"))
    '
)"
CONTEXT_WINDOW="$(jq -er '.context_window_tokens' <<< "$capabilities")"
INPUT_MODALITIES="$(jq -ce '
  [.input_modalities[] | select(. == "text" or . == "image" or . == "audio")]
  | select(length > 0)
' <<< "$capabilities")"

mkdir -p "$CATALOG_DIR"

curl -fsSL \
  "https://raw.githubusercontent.com/openai/codex/rust-v${CODEX_VERSION}/codex-rs/models-manager/models.json" \
  -o "$UPSTREAM_CATALOG"

curl -fsSL \
  "https://raw.githubusercontent.com/openai/codex/rust-v${CODEX_VERSION}/codex-rs/models-manager/prompt.md" \
  -o "$UPSTREAM_PROMPT"

jq \
  --arg model "$model" \
  --arg cluster "$cluster" \
  --arg version "$CODEX_VERSION" \
  --argjson context_window "$CONTEXT_WINDOW" \
  --argjson input_modalities "$INPUT_MODALITIES" \
  --rawfile prompt "$UPSTREAM_PROMPT" '
    ([.models[] | select(.slug == "gpt-5.4")][0] // .models[0])
    | .slug = $model
    | .display_name = $model
    | .description = ($model + " served through ALCF " + $cluster)
    | .visibility = "list"
    | .supported_in_api = true
    | .priority = 0
    | .minimal_client_version = $version
    | .availability_nux = null
    | .upgrade = null
    | .context_window = $context_window
    | .max_context_window = $context_window
    | .auto_compact_token_limit = null
    | .effective_context_window_percent = 90
    | .comp_hash = null
    | .model_messages.instructions_template = $prompt
    | .model_messages.instructions_variables = null
    | .base_instructions = $prompt
    | .default_reasoning_level = null
    | .supported_reasoning_levels = []
    | .supports_reasoning_summary_parameter = false
    | .supports_reasoning_summaries = false
    | .default_reasoning_summary = "none"
    | .reasoning_summary_format = "none"
    | .support_verbosity = false
    | .default_verbosity = null
    | .shell_type = "shell_command"
    | .apply_patch_tool_type = "freeform"
    | .supports_search_tool = false
    | .web_search_tool_type = "text"
    | .experimental_supported_tools = []
    | .input_modalities = $input_modalities
    | .supports_image_detail_original = false
    | .prefer_websockets = false
    | .use_responses_lite = false
    | .tool_mode = null
    | .multi_agent_version = null
    | .auto_review_model_override = null
    | .model_specialty = null
    | .service_tiers = []
    | .additional_speed_tiers = []
    | .default_service_tier = null
    | .include_skills_usage_instructions = true
    | .include_plugin_usage_instructions = false
    | .include_apps_usage_instructions = false
    | (if has("supports_parallel_tool_calls") then .supports_parallel_tool_calls = false else . end)
    | (if has("node_repl_disabled") then .node_repl_disabled = true else . end)
    | (if has("node_repl_auto_review_required") then .node_repl_auto_review_required = false else . end)
    | {models: [.]}
  ' "$UPSTREAM_CATALOG" > "$MODEL_CATALOG"

jq -e --arg model "$model" '
  .models
  | length == 1
    and .[0].slug == $model
' "$MODEL_CATALOG" >/dev/null

printf 'Created %s for Codex %s\n' \
  "$MODEL_CATALOG" \
  "$CODEX_VERSION"

Version Compatibility

The catalog fields depend on your Codex version. If you upgrade Codex, re-run this script to regenerate the catalog with the correct schema for that version.

Finally, add a profile named alcf-<cluster>-<model>.config.toml beside config.toml, replacing / in the model ID with - in the filename. Replace <path> below with the catalog's absolute path printed by the script, and replace the model and cluster placeholders as above.

model_provider = "alcf-inference-service-<cluster>-api"
model = "<model>"
model_catalog_json = "<path>"

To run Codex, use the same model and cluster variables to select the profile. The manual configuration reads ALCF_AI_TOKEN; refresh that variable with alcf-tokens get-token inference when it expires. The automatic configuration above retains its token-helper refresh mechanism.

codex --disable apps --strict-config --profile "alcf-${cluster}-${model//\//-}" -c 'web_search="disabled"'

Claude Code

Add the following to ~/.claude/settings.json.

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://inference-api.alcf.anl.gov/resource_server/tara/api",
    "ANTHROPIC_AUTH_TOKEN": "<your-api-key-here>",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "mistral-large-4",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "mistral-large-4",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "mistral-large-4"
  }
}

Base URL

ANTHROPIC_BASE_URL intentionally omits the trailing /v1 used by the OpenAI-compatible endpoints. Claude Code automatically appends /v1/messages to the base URL.

Context Window with alcf-ai

Both alcf-ai agent run claude and alcf-ai agent configure claude set CLAUDE_CODE_MAX_CONTEXT_TOKENS from the selected model's catalog metadata. Use Claude Code 2.1.193 or newer for this override.

Anthropic Messages Support

You need to use a Direct API-backed endpoint that supports the Anthropic Messages API for Claude Code to function without shims or proxies (i.e. Minerva or Tara).

Other 3rd-Party Agents

Other OpenAI-compatible agents can also be configured to use the service, albeit with varying degrees of support. After familiarizing yourself with your agent's configuration format, see Available Clusters for integration choices. The model can be specified by its fully qualified name (see Available Models).

Shims/Proxies

Some AI agents need to have their inference requests translated and sanitized prior to calling the ALCF Inference Service endpoints for correct behavior. Several community-developed shims and resources are available to aid the integration of these agents with the service:

  1. alcf-proxy - Hongwei Jin
  2. DIY w/ llm-rosetta - Sam Foreman

Responses API Streaming Support

Streaming support is currently disabled for the OpenAI Responses API on non-Direct API endpoints. This means agents that rely on the Responses API as their default wire protocol will fail to send requests to clusters like Sophia. Either configure your agent to utilize the OpenAI Chat Completions API or switch clusters for compatibility.

Metis Tool Calling

There is a known issue related to SambaNova's sanitization of tool calls, leading to strange error responses like this when processing requests.

{\"error\":\"Model started a function call but did not complete it.\",\"error_code\":null,\"error_model_output\":\"\\u003c|channel|\\u003eanalysis\\u003c|message|\\u003eWe needto adjust imports and usage in main.rs.\\n\\nSearch for clear_lines usage. Already seen at line 327. Need to modify that block.\\n\\nOpen around lines 320-340.\\u003c|end|\\u003e\\u003c|start|\\u003eassistant\\u003c|channel|\\u003ecommentary to=functions.read\\u003c|message|\\u003e{\\\"filePath\\\":\\\"/Users/ewong/Documents/alcf/inference-service/vibecoding/rust_vibecoding/src/main.rs\\\",\\\"offset\\\":320,\\\"limit\\\":80}\",\"error_param\":null,\"error_type\":\"Invalid function calling output.\"}


  1. OpenCode and Pi explicitly support third-party/custom provider endpoints. ↩↩