Skip to content

LLM Models

LLM Models are configuration profiles that connect NetOrca Pack to large language model providers such as OpenAI, Anthropic Claude, Google Gemini, Enterprise LLM Gateways, etc. They supply the intelligence behind NetOrca's AI features. Organizations can also connect to private LLMs through internal gateways or enterprise AI platforms to maintain data privacy and comply with organizational policies.

Setting Up LLM Models

Each LLM Model in NetOrca represents a configured connection to a Large Language Model with specific parameters. Different LLM providers require different configuration parameters, authentication methods, and connection settings. The following providers are currently supported. See the examples below for the required configuration per provider:

Provider Examples:

POST /v1/ai/llm_models/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "name": "MyAgent",
    "provider": "openai",
    "model_name": "gpt-4o",
    "prompt": "You are a devops engineer assistant. Follow the service owner's instructions.",
    "timeout": 60,
    "input_price_per_1m_tokens": 2.50,
    "output_price_per_1m_tokens": 10.00,
    "extra_data": {
        "api_key": "sk-...",
        "base_url": "https://api.openai.com/v1"
    }
}

POST /v1/ai/llm_models/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "name": "MyAgent",
    "provider": "anthropic",
    "model_name": "claude-opus-4-5-20251101",
    "prompt": "You are a devops engineer assistant. Follow the service owner's instructions.",
    "timeout": 60,
    "input_price_per_1m_tokens": 3.00,
    "output_price_per_1m_tokens": 15.00,
    "extra_data": {
        "api_key": "sk-ant-...",
        "base_url": "https://api.anthropic.com"
    }
}

POST /v1/ai/llm_models/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "name": "MyAgent",
    "provider": "gemini",
    "model_name": "gemini-3-pro-preview",
    "prompt": "You are a devops engineer assistant. Follow the service owner's instructions.",
    "timeout": 60,
    "input_price_per_1m_tokens": 1.25,
    "output_price_per_1m_tokens": 5.00,
    "extra_data": {
        "api_key": "AIza...",
        "base_url": "https://generativelanguage.googleapis.com/v1beta"
    }
}

POST /v1/ai/llm_models/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "name": "MyAwesome-GenAI",
    "provider": "genai",
    "model_name": "gpt3o-mini",
    "prompt": "You are a devops engineer assistant. Follow the service owner's instructions.",
    "timeout": 30,
    "extra_data": {
        "token_url": "https://auth.example.com/token",
        "request_url": "https://gateway.example.com/generate",
        "credentials": "Bearer AwesomeKey",
        "verify_ssl": false,
        "request_payload": {}
    }
}

Prompt (System Prompt)

The system prompt defines the AI model's role, expertise, and behavioral guidelines. It provides the foundational context that shapes how the model interprets and responds to all requests.

Recommendation for writing effective prompts: Clearly define the role the AI should act as, specify the technical domain and knowledge areas, and set boundaries or quality requirements to ensure reliable responses. An Example:

You are an AWS DevOps engineer who knows all AWS configurations and endpoints.
You will be given a service owner prompt that acts as your further instructions. 
Your inputs are consumer declarations, and based on them, you must generate accurate, 
ready-to-deploy AWS configuration files.

Costs

NetOrca can track and calculate the cost of AI operations through Cost Settings in LLM Models, helping you monitor spending and optimize usage. These costs, along with other LLM-related metrics and analytics, are available in the AI Integration Dashboard.

Note that token counts are reported by each LLM provider, so the counting methodology may vary between providers.

How Cost Calculation Works

All major LLM providers return the number of input and output tokens consumed with each API response. This has become a standard practice. NetOrca uses these token counts to calculate costs by multiplying them against the rates you define per model.

You can configure the following pricing fields for each LLM Model based on your provider's rates:

  • Input Price per 1M Tokens — Cost for tokens sent to the model (prompts, context)
  • Output Price per 1M Tokens — Cost for tokens generated by the model in responses

For example, if a request consumes 2,000 input tokens and 500 output tokens, and the configured rates are $2.50 / $10.00 per 1M tokens respectively, the cost would be: (2,000 / 1,000,000 × $2.50) + (500 / 1,000,000 × $10.00) = $0.01

Example pricing (USD):

  • GPT-4o: $1.50 input / $3.00 output per 1M tokens
  • Claude 3.5 Sonnet: $2.00 input / $4.00 output per 1M tokens

Rate Limits

LLM providers and gateways usually limit how many requests and tokens can be used in a given period, and exceeding these limits leads to failed requests or extra costs. To stay within them, administrators can set rate limits on each LLM Model. The limits apply to all AI Processors using that LLM Model.

Rate limits are set in the extra_data of the LLM Model and are supported for all providers:

"extra_data": {
    "api_key": "sk-...",
    "token_limit_1m": 100000,
    "requests_limit_1m": 50,
    "token_limit_1d": 5000000,
    "requests_limit_1d": 2000,
    "fail_on_rate_limit": false
}
Field Description
token_limit_1m Maximum number of tokens per minute.
requests_limit_1m Maximum number of requests per minute.
token_limit_1d Maximum number of tokens per day.
requests_limit_1d Maximum number of requests per day.
fail_on_rate_limit Fail requests instead of delaying them when a limit is reached. Defaults to false.

Each limit is optional; if it is not set, there is no limit for it. The windows follow the clock: the minute limits reset at the start of every minute, and the day limits reset at midnight.

When a limit is reached, new requests to the LLM Model are handled based on fail_on_rate_limit:

  • false (default) — The request is kept in the SCHEDULED state and sent automatically once the LLM Model is within its limits again. If the LLM Model stays over its limit for longer than 6 times its timeout, the request is marked as FAILED.
  • true — The request fails immediately with status code 429 and a message naming the limit that was reached, e.g. Rate limit exceeded: requests per minute limit reached (50/50).

Test Connection

While configuring an LLM Model and before assigning it to an AI Processor, you can test the connection to verify that the credentials, endpoint, and model settings are correct before it is used in production workflows.

The test works by sending a short basic message to the LLM and expecting an actual generated response back — not just an HTTP status code. This ensures that the model is reachable.

The connection can be tested in two ways. In both cases, nothing is saved.

Before creating the LLM Model — send the full LLM Model configuration, the same body used to create it:

POST /v1/ai/llm_models/test_connection/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "name": "MyAwesome-Gemini",
    "provider": "gemini",
    "model_name": "gemini-1.5-pro",
    "prompt": "You are a devops engineer assistant. Follow the service owner's instructions.",
    "timeout": 60,
    "extra_data": {
        "api_key": "AIza...",
        "base_url": null
    }
}

For an existing LLM Model — send the request to the LLM Model's ID. The body is optional: if empty, the saved configuration is tested; if it contains some fields, they replace the saved values for this test only, which is useful to check a change (e.g. a new API key) before applying it:

POST /v1/ai/llm_models/test_connection/<llm_model_id>/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "extra_data": {
        "api_key": "AIza...new-key",
        "base_url": null
    }
}