Model Serving in Hybrid Manager

Model Serving in Hybrid Manager provides a scalable, Kubernetes-native way to serve AI models as production-grade inference services.

It is implemented using KServe and runs on GPU-enabled nodes in your Hybrid Manager project’s Kubernetes cluster. Model Serving enables Gen AI applications, Knowledge Bases, and custom pipelines to use high-performance models under your control.

How Model Serving fits in the Hybrid Manager architecture

Model Serving is a core capability of Hybrid Manager’s AI Factory workload:

• Models are deployed as KServe InferenceServices within the project’s Kubernetes cluster.
• Model Serving is powered by GPU-enabled infrastructure that you provision and manage.
• Model images come from the Asset Library (formerly Model Library), backed by Hybrid Manager’s image governance.
• Model endpoints (HTTP/gRPC) are available to:
• Gen AI Builder Assistants.
• AIDB Knowledge Bases.
• External applications and APIs.

Model Serving in Hybrid Manager ensures that all model serving is governed, auditable, and runs securely within your infrastructure — enabling Sovereign AI patterns.

How it works in Hybrid Manager

• KServe is installed and managed by Hybrid Manager within your project’s Kubernetes cluster.
• You must provision GPU node groups or node pools to support high-performance model serving.
• GPU nodes must be correctly labeled and configured to support KServe workloads.
• Models are deployed from the Asset Library via ClusterServingRuntime and InferenceService definitions.
• Your applications and AI Factory workloads can invoke model endpoints via REST or gRPC.

Key Hybrid Manager considerations

• GPU infrastructure is required for most advanced models, such as LLMs, embeddings, and vision models.
• Hybrid Manager enables full observability of model serving, including Prometheus metrics and Kubernetes-native monitoring.
• Model serving endpoints are secured and managed within your Hybrid Manager project scope.
• Governance for model images and deployment comes from Hybrid Manager’s integrated Asset Library and image controls.

Typical use cases

• Power Gen AI Builder Assistants with LLM or embedding models.
• Enable AIDB Knowledge Bases with GPU-accelerated embedding pipelines.
• Serve image models (OCR, vision) as part of multi-modal retrieval systems.
• Expose enterprise-grade model APIs to downstream applications.

Links to learn more

• Deploying NVIDIA NIM models in Hybrid Manager
• Verify deployed models
• Model Serving Concepts Hub
• Model Serving FAQ
• AI Factory Concepts
• How to set up GPU resources