External inference services Innovation Release

Context

An external inference service connects Hybrid Manager (HM) to a remote model provider hosted outside the cluster — such as OpenAI, Google Gemini, Anthropic, or NVIDIA NIM. You configure the provider's URL, model name, and API key once; HM stores the credentials in Kubernetes secrets and handles authentication transparently for every downstream request.

This page covers:

• Viewing the Inference Services list
• Registering a new external inference service
• Getting the details of a registered service
• Updating a registered service
• Deregistering a service

Inference Services list

Open the Inference Services list page from the Estate → Inference Services menu in your HM project.

The list displays each service's name, model, and current status. When the status isn't Ready, a short Status Message explains the cause.

Status

Status is refreshed every 5 minutes by a background health check. After you create or edit a service, the connectivity probe runs inline so the result is reflected immediately.

Register an external inference service

Navigate to Estate → Quick Actions → Register External Inference Service in your HM project.

Prerequisites

Before registering, confirm you have:

• The provider's base URL (scheme and host, without a trailing /v1).
• The model name exactly as the provider expects it (case-sensitive).
• A valid API key for the provider.
• Network reachability from the HM cluster to the upstream hostname. Your HM administrator may need to allow egress to the provider's domain.

Quick reference by provider

Use this table to fill in the form for the most common providers. The columns map directly to the fields described below.

For providers not listed here, see the form field descriptions below.

Form fields

External Service Name (required)

A unique identifier for this service within HM. Must follow DNS-style naming rules:

• Lowercase letters and digits only.
• Hyphens (-) are allowed within segments but not at the start or end.
• Dots (.) are allowed as segment separators.
• No uppercase letters, underscores, or spaces.
• Maximum 63 characters.

Example: openai-gpt-4o-mini, azure.gpt-4o.prod.

Tags (optional)

Reuse existing HM tags to group and filter services. Tags have no effect on request routing or authentication.

Model name (required)

The exact identifier the upstream provider expects, as documented by the provider. This value is case-sensitive. Slash-separated names (such as meta/llama-3.1-8b-instruct or openai/gpt-4o-mini) are supported. See the Quick reference by provider table above for examples.

API Key (required for most providers)

The API key only — don't include the Authorization: Bearer … prefix. HM adds the correct auth header automatically based on the API Protocol Version you select.

Model Base URL (required)

The scheme and host (plus any required path prefix) for the provider's API. See the Quick reference by provider table above for the correct value for common providers.

Don't include /v1 — consumer applications append /v1 (or /v1beta for Gemini) themselves. Including /v1 here causes duplicated paths such as /v1/v1/chat/completions, which returns a 404. A trailing slash is tolerated and stripped automatically.

Functions (optional, multi-select)

Capability tags that consumer applications filter on when discovering available models. Use the predefined values below for HM's built-in consumers; for your own applications, any string is valid.

Leave this field empty if you're exposing the service exclusively to custom applications that perform their own model selection.

API Protocol Version (required)

Controls both the request body format and the outbound authentication header. Choose the option that matches the provider's native API.

Allow Insecure Connection (optional, default off)

Disables TLS certificate verification on outbound calls to the upstream. Enable this only if the upstream uses a self-signed certificate or a certificate signed by a CA not trusted by the HM cluster.

Use this toggle when registering another HM's external inference endpoint (for example, HM2 federating through HM1). HM1's ingress typically presents a self-signed certificate, which causes TLS errors for both the registration probe and runtime forwarding. HM-to-HM federation also requires both ends to use OPENAI_V1 as the API Protocol Version.

After clicking Register

HM validates the endpoint before creating any infrastructure. For OpenAI (OPENAI_V1), Anthropic (ANTHROPIC_V1), and Google Gemini (GEMINI_V1_BETA), HM performs a live connectivity probe that checks both reachability and credential validity. If the endpoint is unreachable or the API key is rejected, registration fails immediately with an error — no resources are created.

After registration, the status is refreshed every 5 minutes by a background health check. If the upstream becomes unreachable or starts rejecting credentials, the service flips to Failed (with an explanatory Status Message) at the next refresh tick.

Use the service

Once the service is ready, it's available to:

• HM chatbot — The chatbot picks up services tagged with chatbot-gen-content automatically.
• Pipeline Designer — Registered external models appear in the model picker alongside HM-hosted models. For details, see External inference services in Pipeline Designer.
• Gen AI Builder — Models are available as inference targets in Gen AI Builder pipelines once registered.

Retrieve inference service details

Click a service name in the Inference Services list to open its detail view, which shows the service's configuration, current status, and available actions.

Details

Troubleshooting from the Status Message

When a service is Failed, the Status Message tells you what to fix. Common patterns:

Update inference service parameters

Some parameters can be updated after registration; others require deregistering and re-registering the service.

What can and cannot be changed

How to edit a service parameter

Open the service detail page and select Quick Actions → Edit Service, or click the pencil icon on the Inference Services list.

De-register an external inference service

How to deregister

To delete a service, either open the service detail page and select Quick Actions → Deregister External Inference Service, or click the trash icon on the Inference Services list.

HM blocks deletion if the service is currently referenced by one or more pipelines. Remove or update those pipelines first, then retry.

When deletion succeeds, HM:

1. Removes all Kubernetes resources backing the service (including the API key secret).
2. Removes the service record from the database.
3. Clears all tags associated with the service.