Custom Provider

Integrate any LLM API using Rhai scripting

The Custom provider enables you to integrate any LLM API endpoint that isn’t already supported by writing a simple Rhai script. This is perfect for proprietary APIs, custom deployments, or non-standard endpoints.

How It Works

The Custom provider uses a Rhai script to translate between Circuit Breaker Labs’ internal message format and your custom API’s request/response schema.

1. CBL sends messages to your script

   Circuit Breaker Labs provides conversation history as an array of messages with role and content fields.

2. Your script builds the request

   The build_request() function transforms messages into your API’s expected format.

3. CBL posts to your endpoint

   The request body is sent as JSON to the URL you specify.

4. Your script parses the response

   The parse_response() function extracts the assistant’s message from the API response.

Configuration Options

--url · string · required

Endpoint URL to POST requests to.

Example: https://api.example.com/v1/chat/completions

--script · path · required

Path to the Rhai script file that defines request/response translation.

Example: --script ./providers/my_custom_api.rhai

Writing a Custom Script

Your Rhai script must implement two functions:

build_request(messages)

Transforms the conversation history into your API’s request format.

Parameters:

• messages (array): Array of message objects, each with:
  • role (string): Either "system", "user", or "assistant"
  • content (string): The message content

Returns: A map that will be serialized to JSON and sent to your endpoint.

parse_response(body)

Extracts the assistant’s response from your API’s response.

Parameters:

• body (dynamic): The full deserialized JSON response from your API

Returns: String containing the assistant’s message content.

Example Scripts

OpenAI-Compatible API

// For any OpenAI-compatible endpoint
fn build_request(messages) {
    #{
        "model": "gpt-4o",
        "messages": messages
    }
}

fn parse_response(body) {
    body["choices"][0]["message"]["content"].to_string()
}

Ollama API

// For Ollama's chat endpoint
fn build_request(messages) {
    #{
        "model": "llama3.2",
        "messages": messages,
        "stream": false
    }
}

fn parse_response(body) {
    body["message"]["content"].to_string()
}

Custom API with Authentication

// Example with API key in request body
fn build_request(messages) {
    #{
        "apiKey": "your-api-key",
        "model": "custom-model-v1",
        "prompt": messages,
        "maxTokens": 2000
    }
}

fn parse_response(body) {
    body["result"]["text"].to_string()
}

Anthropic Claude API

// For Anthropic's Claude API
fn build_request(messages) {
    // Separate system message from conversation
    let system_msg = "";
    let conversation = [];
    
    for msg in messages {
        if msg.role == "system" {
            system_msg = msg.content;
        } else {
            conversation.push(msg);
        }
    }
    
    #{
        "model": "claude-3-5-sonnet-20241022",
        "max_tokens": 4096,
        "system": system_msg,
        "messages": conversation
    }
}

fn parse_response(body) {
    body["content"][0]["text"].to_string()
}

Google Gemini API

// For Google's Gemini API
fn build_request(messages) {
    let contents = [];
    
    for msg in messages {
        if msg.role != "system" {  // Gemini doesn't use system role
            let role = if msg.role == "assistant" { "model" } else { "user" };
            contents.push(#{
                "role": role,
                "parts": [#{ "text": msg.content }]
            });
        }
    }
    
    #{
        "contents": contents
    }
}

fn parse_response(body) {
    body["candidates"][0]["content"]["parts"][0]["text"].to_string()
}

Complete Usage Example

Single-Turn

cbl single-turn \
    --threshold 0.5 \
    --variations 2 \
    --maximum-iteration-layers 2 \
    custom \
    --url https://api.example.com/v1/chat \
    --script ./my_provider.rhai

Rhai Language Basics

Rhai is a simple scripting language with JavaScript-like syntax:

Creating Objects (Maps)

let obj = #{
    "key": "value",
    "number": 42,
    "nested": #{
        "inner": "data"
    }
};

Accessing Fields

let value = obj["key"];           // Bracket notation
let nested = obj["nested"]["inner"];  // Nested access

Arrays

let arr = [1, 2, 3];
arr.push(4);
let first = arr[0];

Loops

for item in array {
    print(item);
}

String Conversion

let str = value.to_string();  // Convert to string

For complete Rhai documentation, visit rhai.rs/book.

Example Scripts Repository

Circuit Breaker Labs provides example scripts in the repository:

ls examples/providers/

Available examples:

• openai_completions.rhai - OpenAI chat completions API
• ollama_chat.rhai - Ollama chat API
• openai_responses.rhai - OpenAI with response formatting
• ollama_completions.rhai - Ollama completions endpoint

Debugging Your Script

Rhai scripts can use print() and debug() functions for logging:

fn build_request(messages) {
    print("Building request with " + messages.len() + " messages");
    debug(messages);
    
    let request = #{
        "model": "custom-model",
        "messages": messages
    };
    
    debug(request);
    return request;
}

These will output to the CBL CLI logs during evaluation runs.

Authentication

For APIs requiring authentication, you have several options:

1. Include in Request Body

fn build_request(messages) {
    #{
        "api_key": "your-key-here",
        "messages": messages
    }
}

2. Use HTTP Headers

Set authentication headers using environment variables or CBL configuration. The Custom provider automatically includes headers from the CBL context.

3. URL Parameters

// Include key in URL when creating custom provider
// --url https://api.example.com/chat?key=YOUR_KEY

Security: Avoid hardcoding API keys in scripts. Use environment variables or secure configuration management instead.

Error Handling

If your script encounters an error, CBL will report it with the error location:

Error: Script error in build_request: Property not found: 'message'
  at my_provider.rhai:12

Common issues:

• Accessing non-existent fields: Check the API response structure
• Type mismatches: Ensure proper type conversions with .to_string()
• Missing return values: Both functions must return a value

Tips

Start with Examples: Copy an existing example script from examples/providers/ and modify it for your API.

Test Your API First: Use curl or Postman to understand your API’s request/response format before writing the script.

Use Debug Logging: Add print() statements to see what data your script is working with.

Response Validation: Ensure parse_response() always returns a non-empty string, or evaluations will fail.

Related Resources

Rhai Language Book

Complete Rhai scripting language documentation

Example Scripts

Ready-to-use provider scripts for common APIs