Skip to content

LLM Models

LLM Models serve as connectors that integrate NetOrca Pack with various Large Language Models, such as OpenAI, Google Gemini, Grok, etc., providing the intelligence needed for NetOrca AI features. Additionally, organizations can connect to their private LLMs through internal gateways or enterprise AI platforms, ensuring data privacy and compliance with organizational policies.

Setting Up LLM Models

Each LLM Model in NetOrca represents a configured connection to a Large Language Model with specific parameters. Different LLM providers require different configuration parameters, authentication methods, and connection settings to establish successful connections. So far, the following LLM providers are supported. In order to configure the LLM Model, you need to provide the following information:

Provider Examples:

POST /v1/ai/llm_models/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "name": "MyAwesome-OpenAI-GPT4",
    "provider": "openai",
    "model_name": "gpt-4o",
    "prompt": "Do whatever service owner asks",
    "timeout": 60,
    "input_price_per_1m_tokens": 2.50,
    "output_price_per_1m_tokens": 10.00,
    "extra_data": {
        "api_key": "sk-...",
        "base_url": null,
    },
    "metadata": {
        "tags": ["reasoning", "code"],
        "capabilities": ["function_calling", "vision"],
    },
}

POST /v1/ai/llm_models/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "name": "MyAwesome-OpenAI-Assistant",
    "provider": "openai_assistant",
    "model_name": "gpt-4o",
    "prompt": "Do whatever service owner asks",
    "timeout": 120,
    "input_price_per_1m_tokens": 2.50,
    "output_price_per_1m_tokens": 10.00,
    "extra_data": {
        "OPENAI_API_KEY": "sk-...",
        "OPENAI_ASSISTANT_ID": "asst_...",
    },
}

POST /v1/ai/llm_models/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "name": "MyAwesome-Claude",
    "provider": "anthropic",
    "model_name": "claude-3-5-sonnet-20241022",
    "prompt": "Do whatever service owner asks",
    "timeout": 60,
    "input_price_per_1m_tokens": 3.00,
    "output_price_per_1m_tokens": 15.00,
    "extra_data": {
        "api_key": "sk-ant-...",
        "base_url": null,
    },
}

POST /v1/ai/llm_models/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "name": "MyAwesome-Gemini",
    "provider": "gemini",
    "model_name": "gemini-1.5-pro",
    "prompt": "Do whatever service owner asks",
    "timeout": 60,
    "input_price_per_1m_tokens": 1.25,
    "output_price_per_1m_tokens": 5.00,
    "extra_data": {
        "api_key": "AIza...",
        "base_url": null,
    },
}

POST /v1/ai/llm_models/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "name": "MyAwesome-GenAI",
    "provider": "genai",
    "model_name": "custom-model",
    "prompt": "Do whatever service owner asks",
    "timeout": 30,
    "extra_data": {
        "token_url": "https://auth.example.com/token",
        "request_url": "https://gateway.example.com/generate",
        "credentials": "Bearer AwesomeKey",
        "verify_ssl": false,
    },
}

POST /v1/ai/llm_models/ HTTP/1.1
Content-Type: application/json
Authorization: Token <YOUR_TOKEN>
{
    "name": "MyAwesomeLLM",
    "prompt": "Do whatever service owner asks",
    "extra_data": {
        "model_type": "Custom HTTP",
        "base_url": "https://api.example.com",
        "authorization": "<auth header, e.g. 'Bearer AwesomeKey'>",
        "timeout": 30,
        "verify": true,
    },
}

Prompt (System Prompt)

The base system prompt that defines the AI model's role, expertise, and instructions. This is the foundational instruction set that shapes how the AI will interpret and respond to requests. The prompt acts as a wrapper that provides context and guidelines for all interactions with this model.

Recommendation for writing effective prompts: Clearly define the role the AI should act as, specify the technical domain and knowledge areas, and set boundaries or quality requirements to ensure reliable responses. An Example:

You are an AWS DevOps engineer who knows all AWS configurations and endpoints.
You will be given a service owner prompt that acts as your further instructions. 
Your inputs are consumer declarations, and based on them, you must generate accurate, 
ready-to-deploy AWS configuration files.

Costs

You can configure NetOrca via the Cost Settings in LLM Models, to track and calculate the cost of your AI operations to help you monitor spending and optimize usage. You will be able to view these costs, as well as other important LLM related data in the AI Integration Dashboard. The token count calculation comes from the LLM provider, and so it differs between each provider.

Setting Up Cost Tracking

When configuring an LLM model, set these pricing fields based on your provider's rates:

  • Input Price per 1M Tokens - Cost for tokens you send to the model (prompts, context)
  • Output Price per 1M Tokens - Cost for tokens the model generates in responses

Example pricing (USD):

  • GPT-4o: $2.50 input / $10.00 output per 1M tokens
  • Claude 3.5 Sonnet: $3.00 input / $15.00 output per 1M tokens