CUGA LogoCUGA AGENT
Configure

Models

Configure different LLM providers for CUGA

CUGA supports multiple LLM providers with flexible configuration options. You can configure models through TOML files or override specific settings using environment variables.

Supported Platforms

  • OpenAI - GPT models via OpenAI API (also supports LiteLLM via base URL override)
  • IBM WatsonX - IBM's enterprise LLM platform
  • Azure OpenAI - Microsoft's Azure OpenAI service
  • Groq - Groq's LLM platform
  • OpenRouter - LLM API gateway provider
  • MiniMax - MiniMax-M3 (global or China endpoint)
  • Google Gemini - Google GenAI
  • Ollama - Local OpenAI-compatible models (for example gpt-oss)
  • RITS - IBM Research Internal Token Service (IBM internal)

Option 1: OpenAI 🌐

Setup Instructions:

  1. Create an account at platform.openai.com
  2. Generate an API key from your API keys page
  3. Add to your .env file: 
    # For OpenAI:
# AGENT_SETTING_CONFIG="settings.openai.toml" #Could be replaced by the settings in ./configurations/models
# OPENAI_API_KEY="XXXXXX"
# OPENAI_API_VERSION="2024-08-01-preview"

Default Values:

  • Model: gpt-4o
  • API Version: OpenAI's default API Version
  • Base URL: OpenAI's default endpoint

Environment Variable Override: You can override the model name using:

MODEL_NAME="gpt-4-turbo"

Option 2: IBM WatsonX 🔵

Setup Instructions:

  1. Access IBM WatsonX
  2. Create a project and get your credentials:
    • Project ID
    • API Key
    • Region/URL
  3. Add to your .env file: 
    # For WatsonX
# AGENT_SETTING_CONFIG="settings.watsonx.toml" #Could be replaced by the settings in ./configurations/models
# WATSONX_PROJECT_ID="XXXXXX"
# WATSONX_URL="https://us-south.ml.cloud.ibm.com"
# WATSONX_APIKEY="XXXXXX"

Default Values:

  • Model: meta-llama/llama-4-maverick-17b-128e-instruct-fp8

Environment Variable Override: You can override the model name using:

MODEL_NAME="meta-llama/llama-3.1-8b-instruct"

Option 3: Azure OpenAI

Setup Instructions:

  1. Add to your .env file: 
    # For Azure Openai
# AGENT_SETTING_CONFIG="settings.azure.toml" #Could be replaced by the settings in ./configurations/models
# AZURE_OPENAI_API_KEY="XXXXXX"
# AZURE_OPENAI_ENDPOINT="XXXXXX"

Environment Variable Override: You can override the model name using:

MODEL_NAME="gpt-4o"

Option 4: Groq

Setup Instructions:

  1. Add to your .env file: 
    # For Groq
# AGENT_SETTING_CONFIG="settings.groq.toml"
# GROQ_API_KEY="XXXX"

Environment Variable Override: You can override the model name using:

MODEL_NAME="openai/gpt-oss-120b"
  1. Install dependencies 
    uv sync --group groq

Option 5: OpenRouter

Setup Instructions:

  1. Create an account at openrouter.ai
  2. Generate an API key from your account settings
  3. Add to your .env file:
   # For OpenRouter
   # AGENT_SETTING_CONFIG="settings.openrouter.toml"
   # OPENROUTER_API_KEY="XXXX"
   # OPENROUTER_BASE_URL="https://openrouter.ai/api/v1"

Default Values:

  • Base URL: https://openrouter.ai/api/v1
  • Model: Configurable via MODEL_NAME environment variable

Environment Variable Override: You can override the model name using:

MODEL_NAME="openai/gpt-4o"

Note: OpenRouter provides access to multiple model providers. Refer to OpenRouter's model documentation for available models and their pricing.

Option 6: RITS (IBM Internal)

RITS (Research Internal Token Service) is an IBM-internal LLM gateway service.

Setup Instructions:

  1. Get your RITS API key from the internal service
  2. Add to your .env file: 
    # For RITS
AGENT_SETTING_CONFIG="settings.rits.toml"
RITS_API_KEY="XXXXXX"

Default Values:

  • Model: rits/openai/gpt-oss-120b
  • URL: http://localhost:4000

Environment Variable Override: You can override the model name using:

MODEL_NAME="rits/openai/gpt-4o"

Note: RITS is only available for IBM Research users with access to the internal service.

Option 7: MiniMax

MiniMax-M3 over an OpenAI-compatible API. Default endpoint is global English (https://api.minimax.io/v1).

Setup Instructions:

  1. Add to your .env file:

    AGENT_SETTING_CONFIG="settings.minimax.toml"
MINIMAX_API_KEY="XXXXXX"
# Optional: MINIMAX_BASE_URL="https://api.minimax.io/v1"

  2. For the China endpoint, either set MINIMAX_REGION=cn or MINIMAX_BASE_URL=https://api.minimaxi.com/v1.

Default Values:

  • Model: MiniMax-M3
  • URL: https://api.minimax.io/v1 (or China when MINIMAX_REGION is cn / china)

Environment Variable Override:

MODEL_NAME="MiniMax-M3"

Option 8: Google Gemini

Setup Instructions:

  1. Add to your .env file: 
    AGENT_SETTING_CONFIG="settings.google.toml"
GOOGLE_API_KEY="XXXXXX"

Environment Variable Override:

MODEL_NAME="gemini-2.0-flash"

Option 9: Ollama (local)

Ollama is configured as an OpenAI-compatible endpoint. A dummy API key is required by the client and ignored by Ollama.

Setup Instructions:

  1. Run Ollama locally and pull a model (for example gpt-oss:20b).
  2. Add to your .env file: 
    AGENT_SETTING_CONFIG="settings.ollama.toml"
OPENAI_API_KEY="ollama"
# Optional: OPENAI_BASE_URL="http://localhost:11434/v1"

The bundled profile uses a 300s timeout (local generation can exceed the default 61s) and turns vision off for text-only models such as gpt-oss.

Environment Variable Override:

MODEL_NAME="gpt-oss:20b"

LiteLLM Support

CUGA supports LiteLLM through the OpenAI configuration by overriding the base URL:

  1. Add to your .env file: 
    # For LiteLLM:
# AGENT_SETTING_CONFIG="settings.openai.toml" #Could be replaced by the settings in ./configurations/models
# OPENAI_API_KEY="XXXXXX"
# OPENAI_BASE_URL="XXXXXX"

Environment Variable Override: You can override the model name using:

MODEL_NAME="gpt-4-turbo"

Configuration Files

CUGA uses TOML configuration files located in src/cuga/configurations/models/:

  • settings.openai.toml - OpenAI configuration (also supports LiteLLM via base URL override)
  • settings.openai.reasoning-high.toml - same as OpenAI, with reasoning_effort = "high"
  • settings.openai.reasoning-high-32k.toml - high reasoning effort with larger token budgets
  • settings.watsonx.toml - WatsonX configuration
  • settings.azure.toml - Azure OpenAI configuration
  • settings.groq.toml - Groq configuration
  • settings.openrouter.toml - OpenRouter configuration
  • settings.minimax.toml - MiniMax configuration
  • settings.google.toml - Google Gemini configuration
  • settings.ollama.toml - local Ollama (OpenAI-compatible)
  • settings.rits.toml - RITS configuration (IBM internal)
  • settings.litellm.toml - LiteLLM gateway

Each file contains agent-specific model settings that can be overridden by environment variables.

Reasoning-effort profiles

For models served over an OpenAI-compatible endpoint that honor reasoning_effort:

AGENT_SETTING_CONFIG="settings.openai.reasoning-high.toml"

Use settings.openai.reasoning-high-32k.toml if high effort crowds out visible output (empty completions) — that profile enlarges max_tokens so reasoning and the answer can both fit. Reasoning tokens count against the same budget.

Settings File Content Structure

Each settings file contains configuration for different CUGA agents. You can configure each agent independently with different models and parameters:

OpenAI Configuration Example

[agent.task_decomposition.model]
platform = "openai"
temperature = 0.1
max_tokens = 1000

[agent.planner.model]
platform = "openai"
temperature = 0.1
max_tokens = 5000

[agent.chat.model]
platform = "openai"
temperature = 0.1
max_tokens = 5000

[agent.shortlister.model]
platform = "openai"
temperature = 0.1
max_tokens = 7000

WatsonX Configuration Example

[agent.shortlister.model]
platform = "watsonx"
model_name = "meta-llama/llama-4-maverick-17b-128e-instruct-fp8"
temperature = 0.1
max_tokens = 7000

[agent.planner.model]
platform = "watsonx"
model_name = "meta-llama/llama-4-maverick-17b-128e-instruct-fp8"
temperature = 0.1
max_tokens = 5000

[agent.chat.model]
platform = "watsonx"
model_name = "meta-llama/llama-4-maverick-17b-128e-instruct-fp8"
temperature = 0.1
max_tokens = 5000

OpenRouter Configuration Example

[agent.task_decomposition.model]
platform = "openrouter"
temperature = 0.1
max_tokens = 1000

[agent.planner.model]
platform = "openrouter"
temperature = 0.1
max_tokens = 5000

[agent.chat.model]
platform = "openrouter"
temperature = 0.1
max_tokens = 5000

[agent.shortlister.model]
platform = "openrouter"
temperature = 0.1
max_tokens = 7000

Switching Between Providers

Method 1: Environment Variable

# Switch to OpenAI
export AGENT_SETTING_CONFIG="settings.openai.toml"

# Switch to WatsonX
export AGENT_SETTING_CONFIG="settings.watsonx.toml"

# Switch to Azure
export AGENT_SETTING_CONFIG="settings.azure.toml"

# Switch to Groq
export AGENT_SETTING_CONFIG="settings.groq.toml"

# Switch to OpenRouter
export AGENT_SETTING_CONFIG="settings.openrouter.toml"

# Switch to MiniMax
export AGENT_SETTING_CONFIG="settings.minimax.toml"

# Switch to Gemini
export AGENT_SETTING_CONFIG="settings.google.toml"

# Switch to local Ollama
export AGENT_SETTING_CONFIG="settings.ollama.toml"

# Switch to RITS (IBM internal)
export AGENT_SETTING_CONFIG="settings.rits.toml"

Method 2: Edit .env File

# Edit .env file
AGENT_SETTING_CONFIG="settings.openai.toml"

📚 Next Steps

After configuring your model:

  1. Environment Setup: Configure other system settings
  2. Customization Overview: Fine-tune CUGA behavior
  3. Task modes: API, web, or hybrid

Model configured? Move to Environment Setup!

Configure

Models, environment, task modes, persistence, and external services.

Environment Configuration

Configure environment variables and system settings

On this page

Supported Platforms
Option 1: OpenAI 🌐
Option 2: IBM WatsonX 🔵
Option 3: Azure OpenAI
Option 4: Groq
Option 5: OpenRouter
Option 6: RITS (IBM Internal)
Option 7: MiniMax
Option 8: Google Gemini
Option 9: Ollama (local)
LiteLLM Support
Configuration Files
Reasoning-effort profiles
Settings File Content Structure
OpenAI Configuration Example
WatsonX Configuration Example
OpenRouter Configuration Example
Switching Between Providers
Method 1: Environment Variable
Method 2: Edit .env File
📚 Next Steps