AI workloads, data platforms, and infrastructure notes, written from the engineering edge between benchmarks and production.

RSS feed
/ /

NVIDIA Kubernetes Engine: Managed Kubernetes Comes to the AI Factory

At KubeCon NA, NVIDIA open-sourced NVKE and NVKR to run GPU Kubernetes at AI-factory scale—tenant clusters, signed node images, partner buzz, and the data layer question.

I

Itzik — VP Mission Alignment, VAST Data

·

·

16 min read


NVIDIA Kubernetes Engine NVKE logo for managed GPU Kubernetes at the AI factory

For the last few years, the hard part of running a GPU cloud with NVIDIA Kubernetes Engine (NVKE) has not been getting Kubernetes to see a GPU. The GPU Operator, device plugins and now DRA handled that. The hard part has been everything around the cluster: standing up dozens or hundreds of isolated clusters on bare metal, keeping drivers, kernels, firmware and Kubernetes versions in a combination that actually works, upgrading without breaking tenants, and proving to a customer that the node they just got is the node you think it is.

At KubeCon + CloudNativeCon North America in Salt Lake City this week, NVIDIA went after exactly that problem. On November 10 it published a technical post introducing two open source projects, NVIDIA Kubernetes Engine (NVKE) and NVIDIA Kubernetes Recipes (NVKR). Both are in public preview, both are Apache 2.0, and both live under the NVIDIA DSX AI Factory organization on GitHub. The code itself quietly appeared on GitHub on November 6, a few days ahead of the show, and picked up a fair bit of discussion before the official announcement.

Below I cover what NVKE is, how it is put together, where it slots into NVIDIA’s fast-growing Kubernetes portfolio, what else NVIDIA shipped the same week, how partners and the press are reacting, and why I think the most interesting unanswered question is about data rather than compute.

The problem: GPU capacity, sold as Kubernetes

AI factories and NVIDIA Cloud Partners increasingly want to sell GPU capacity the way hyperscalers do. The customer calls an API, gets a kubeconfig back and runs their workloads, while the provider stays responsible for everything underneath. Delivering that well means clusters have to be created programmatically, kept apart from one another, upgraded and healed without drama, and, the part people underestimate, re-validated every time a new GPU generation, driver, NIC firmware or Kubernetes release enters the picture.

Anyone who has tried to stitch this together from Cluster API providers, a homegrown image pipeline and a pile of Ansible knows that the last item, qualification, is where most of the pain lives. NVKE’s design reflects that: half of the story is a build system.

Two projects, one pipeline

NVIDIA splits the problem into a builder and a manager:

•NVKR (the builder) is the factory floor. Starting from sources whose versions are locked, it turns out everything a cluster needs: OS images for the nodes, the NVKE binaries, container images and Helm charts. Each build keeps a record of exactly what went into it, and recipes capture which software versions have been proven to work together on which GPU generation. Builds that come out of NVIDIA’s hosted pipeline are signed and carry provenance; anything you build locally is not signed.
•NVKE (the manager) is the operator. It takes those artifacts and uses them to stand up and look after Kubernetes clusters, driven through a REST API, with its control loops living on a separate management cluster.

You do not have to buy into both. NVKE does not need NVKR at runtime, and NVKR is happy to produce artifacts for somebody else’s cluster manager. That separation turns out to matter a lot for the partner ecosystem, which I will come back to.

NVKR builds signed artifacts; NVKE requests machines via CAPNICo and NICo for tenant Kubernetes clusters

Figure 1. NVKR builds and signs artifacts, the NVKE management plane requests machines through CAPNICo and NICo, and each tenant gets an isolated workload cluster.

Under the hood

The project goes out of its way to say what NVKE is not. It is not another Kubernetes distro, and it is not just one more Cluster API variant. The closest description is a cluster factory that bootstraps itself and then manufactures clusters for other people. Here is how that works in practice:

•Seed and management cluster. Everything starts from a throwaway kind cluster that nvkectl spins up on a single bootstrap machine. Its only job is to give birth to the real management cluster, after which it is torn down.
•Workload clusters. Tenant clusters are cut out of the management cluster one at a time, and every one of them gets a private control plane, a private etcd and its own pools of nodes. No two tenants ever talk to the same API server, which is about as hard as Kubernetes isolation gets.
•Bare metal through NICo. The physical servers are handed out by NVIDIA Infra Controller (NICo), NVIDIA’s bare-metal lifecycle service. NVKE reaches it through CAPNICo, which plugs NICo into Cluster API as an infrastructure provider.
•Attested node identity. A server does not get credentials to join a cluster on trust. It first has to prove who it is, using its TPM and a SPIFFE/SPIRE identity. Encrypting etcd is a per-cluster option on top of that.
•Open standards. Under the covers it is plain upstream Kubernetes assembled with kubeadm and Cluster API, so your existing kubectl and clusterctl habits still apply next to nvkectl. The CNI is OVN-Kubernetes, and observability comes as a Helm chart built on OpenTelemetry that ships all three signal types out over OTLP.
•Certified. NVIDIA has put NVKE through CNCF conformance on Kubernetes 1.36, and it holds both the Certified Kubernetes Installer and Certified Kubernetes AI Platform marks.

Day 2 is where NVKE earns its keep. After a cluster is up, it handles growing and shrinking node pools, moving to newer Kubernetes versions, swapping out bad nodes and keeping certificates from expiring. It is also the cluster manager NVIDIA has validated as the foundation underneath DSX OS.

From source to tenant cluster

NVIDIA’s walkthrough is a good way to understand the operating model. It boils down to five steps:

NVKE operator workflow from building a node image to delivering a tenant kubeconfig

Figure 2. The NVKE operator workflow, from building a node image to handing a tenant a kubeconfig.

1.Build the node image. Everything begins in the NVKR repo. Its quickstart produces a QCOW2 disk image that already contains the NVKE node agent, OVN-Kubernetes, the SPIRE agent and Kubernetes 1.36 packages whose checksums are locked down.
2.Deploy the management plane. You describe the site in a spec file and hand it to nvkectl bootstrap site. NICo hands over three machines, they become the management cluster, the seed cluster disappears, and nvkectl config set-site switches the CLI over to talking to the new NVKE API.
3.Publish a release set. This is the menu you offer tenants. Each named release combines three things: how a machine boots (delivered as an iPXE script), how the control plane is configured and which Kubernetes version it runs, and how worker nodes are set up. The tenant just chooses a version they trust; boot images and cloud-init stay your problem, not theirs. Different node pools can also boot different images, which is handy when one cluster mixes Arm and x86 hosts or two GPU generations.
4.Create a customer cluster. NVIDIA’s example asks nvkectl create cluster for Kubernetes 1.36.3, a three-node control plane and four GPU workers. The cluster reports Provisioning while the first control-plane node initializes with kubeadm and the networking and NodeLocal DNSCache add-ons go in. When that work finishes, it reports Provisioned.
5.Hand it off. nvkectl get kubeconfig gives you the credentials to pass to the tenant over a secure channel. From then on the tenant uses the cluster, and you keep running it through the API.

In command form, the operator’s side of the flow looks like this (flags and spec files omitted; see the NVKE docs for the full syntax):

# 1. bootstrap the management plane from a site spec

nvkectl bootstrap site …

nvkectl config set-site …

# 2. create a tenant cluster and wait for Provisioned

nvkectl create cluster …

# 3. hand the tenant their credentials

nvkectl get kubeconfig …

Try it on a laptop. You can kick the tires without a single GPU or an NVIDIA login. A modest Linux box or VM on x86 or Arm will do, as long as it can spare six cores, 12 GiB of RAM and 100 GiB of disk, and has Docker plus a virtual TPM. On a Mac, run it inside Colima.

Where NVKE sits in NVIDIA’s Kubernetes stack

NVKE does not arrive alone. Over the past year NVIDIA has been assembling a layered set of open source pieces, and NVKE fills the cluster-lifecycle layer that was missing between bare metal and the in-cluster runtime:

Layered GPU Kubernetes stack: NVKE manages clusters; AICR components manage in-cluster AI workloads

Figure 3. A layered view of the GPU Kubernetes stack. NVKE manages whole clusters; AICR and its components manage what runs inside them.

•AI Cluster Runtime (AICR) generates version-locked recipes for the accelerated runtime inside a cluster. Its component catalog covers the GPU Operator, Network Operator, the DRA driver for GPUs, Node Feature Discovery, NodeWright and NVSentinel. NVKE installs AICR as the optional accelerated runtime on day 2.
•NodeWright is a Kubernetes-native package manager for node-level changes such as kernel parameters, sysctls and host tuning, defined as custom resources. It ran in production inside NVIDIA under the name Skyhook before being renamed and open sourced.
•NVSentinel watches GPU health, publishes health events and can cordon, drain or reboot faulty nodes. In AICR’s default posture the remediation pieces are off, so detection works out of the box and automated action is opt-in.

This layering is also the cleanest way to clear up a confusion I have already seen. One analysis this week framed the question as NVKE versus the GPU Operator and correctly called the comparison a category error: NVKE provisions and operates whole clusters on bare metal, while the GPU Operator manages GPU software inside a cluster that already exists. Some early write-ups went further off course and described NVKE as a set of device plugins for fractional GPU sharing. It is not. NVKE does nothing to change how pods are scheduled onto GPUs; the in-cluster scheduling story, and its limits, stays exactly where it was.

The ecosystem is already lining up

Because NVKR and NVKE are decoupled and open, partners can adopt one, the other, or both. According to NVIDIA:

•Cloud partners: Firmus and SK Telecom are building both projects into the AI factories they run.
•Platform vendors: Mirantis, Rafay, Spectro Cloud, OpenNebula Systems and vCluster are each pulling pieces of the work into their own managed Kubernetes products.
•Networking: Netris is wiring its network automation up to NVKE.

The interesting tension here is that several of those vendors sell their own cluster lifecycle management. My read is that NVKR is the more immediately useful piece for them: a signed, provenance-tracked, NVIDIA-validated artifact pipeline is something every GPU cloud platform needs, regardless of whose management plane sits on top.

The wider news: NVKE was one of several moves

NVKE did not land on its own. It was one item in a broader set of KubeCon announcements, and that set is itself the latest step in a year where NVIDIA has steadily pushed GPU infrastructure software into the open.

Timeline of NVIDIA open source Kubernetes releases in 2026 leading to KubeCon North America NVKE launch

Figure 4. NVIDIA’s open source Kubernetes releases over 2026, culminating in the KubeCon North America announcements.

Same week, same stage

•NVIDIA Snapshot. The third new DSX AI Factory component targets cold starts. Instead of loading and warming a model every time an inference replica scales out, Snapshot brings back a replica that was checkpointed after initialization. The partner numbers are striking: design partner Crusoe reported a GLM-5.3 deployment going from about eight minutes to ready down to 31 seconds, and Saturn Cloud measured vLLM scale-from-zero startups falling from a range of 2 to 8.5 minutes to between 6 and 34 seconds.
•OpenShell goes to the CNCF. NVIDIA is handing OpenShell, its policy runtime that keeps autonomous agents secure and under governance, to the foundation, so it will no longer be a single-vendor project.
•Agent Substrate. NVIDIA is now contributing to this Google-originated CNCF Sandbox project for running long-lived agents on Kubernetes, with a focus on making it play well with Dynamo and the KAI Scheduler, and on tying it to OpenShell.

A year of building the stack in the open

Seen across 2026, the pattern is hard to miss. In March, NVSentinel and AI Cluster Runtime were introduced around GTC, and at KubeCon Europe in Amsterdam NVIDIA gave its DRA driver for GPUs to the CNCF, saw the KAI Scheduler accepted into the Sandbox, introduced Grove, and joined the CNCF as a platinum member. Over the year it also open sourced NodeWright and the NVIDIA Cluster Readiness Engine (NVCRE), which runs real NCCL and training jobs to prove a cluster is healthy before customers arrive. NVIDIA now groups much of this under DSX OS, organized roughly as configure (AICR), certify (NVCRE) and operate (NVSentinel). NVKE adds the missing piece underneath: something that actually creates and runs the clusters those tools act on.

Details the press picked up

German outlet heise added several practical details from the documentation. NVKE hands over servers without GPU or fabric software installed, which leaves that work to the runtime layer. New clusters get OVN-Kubernetes, with Calico kept only for existing installations, and highly available control planes rely on a virtual IP that NVKE sets up with kube-vip. heise also points out an operational catch: NICo itself needs a Kubernetes cluster of at least three nodes, so a real deployment stacks two management layers. Finally, heise mentions a second infrastructure provider, LaunchLayer, which can be rolled out with Argo CD as part of the DSX tooling. NVIDIA’s own preview docs still list NICo as the supported provider, so treat that one as something to verify against the repo.

How partners are framing it

The partner messaging is telling. Spectro Cloud describes NVKE as the Kubernetes management layer inside DSX OS and says it is building a phased NVKE integration into PaletteAI. Rafay has been publishing about building a consumable AI cloud on DSX OS components and recently completed NVIDIA’s GPU Operator partner validation for its managed Kubernetes service. vCluster, meanwhile, launched its own bare-metal control plane, vMetal, at GTC this year. Several of these companies sell exactly the kind of cluster lifecycle management NVKE now provides for free, so expect a mix of cooperation and competition as the details of each integration emerge.

The skeptics

Not everyone is rushing in. Independent analysis this week leaned cautious, recommending that teams keep production on the generally available GPU Operator stack and treat NVKE as a pilot. Commenters on Hacker News, as summarized by explainx, suggested the project mostly benefits operators chasing NVIDIA Cloud Partner status or wanting NVIDIA support across the entire stack, rather than typical enterprise platform teams. Coverage from TechRepublic and Techstrong also stressed that both projects remain previews.

Preview reality check

NVIDIA is refreshingly candid about the gaps. Before anyone plans a production rollout, these are the limits worth knowing:

Area

What the preview says

Production readiness

NVIDIA says not to run production on it yet, and warns that the API surface and settings can shift from release to release.

API security

Out of the box the API has no authentication and runs over plain HTTP. Lock it down yourself before anyone else can reach it.

Management cluster

It cannot yet grow, patch or heal itself, and there is no published way to upgrade it or back it up and restore it.

Infrastructure provider

NVIDIA’s docs list NICo, reached through CAPNICo, as the supported provider. heise reports a second option (LaunchLayer) that is worth verifying.

Workload lifecycle

Node replacement and Kubernetes upgrades work, but with preview limits and manual steps.

Disk encryption

The full-disk-encryption images are a demo. Do not count on them to protect data at rest.

Driver updates

The GPU driver is part of the immutable node image. Moving to a new driver means a new image and fresh nodes, not an in-place update.

The practical advice from early independent analysis matches mine: keep production GPU clusters on what you run today, and evaluate NVKE in a bounded, disposable pilot. Test the isolated-cluster tenancy model, check whether the GPU Operator behaves inside an NVKE workload cluster (this is not yet confirmed as a supported configuration), measure how the manual lifecycle steps scale, and think hard about how you would recover if you lost the management plane.

My take: clusters become cattle, so where does the data live?

Read the NVKE design closely and a clear philosophy emerges. Node images are immutable and pinned. A driver update is not a rolling in-place patch; it is a new image and replaced nodes. Every tenant gets its own control plane and etcd. Clusters are created and deleted through an API. That is the right model for operating compute at AI-factory scale, and it pushes the industry further toward treating GPU clusters as disposable.

What it does not answer, and to be fair does not try to answer, is where the state goes. NVKE’s preview documentation does not define a storage layer at all; the only storage-related feature is per-cluster etcd encryption. Yet the things that actually matter to an AI customer, their datasets, checkpoints, model weights, KV cache, vector indexes and logs, have to outlive every one of those disposable clusters. They also have to be shared across many GPU nodes at high throughput while staying strictly isolated between tenants.

NVIDIA’s own rules for its cloud partners show how central this is. Its published NCP requirements expect a financially backed control-plane availability target of 99.95% or better for managed Kubernetes, and they ask partners to have high-performance storage in place, with its API integrated, weeks before the GPUs and clusters are delivered. In other words, the storage platform is expected to be ready before the first NVKE cluster ever exists. Snapshot points the same way: restoring a checkpointed inference replica in seconds only works if that checkpoint can be read back very fast from somewhere.

That creates a few requirements that I would put on any AI factory’s pilot checklist alongside the ones above:

•Data outside the blast radius. If a node, a node pool or an entire workload cluster can be rebuilt from an image at any time, persistent data must live on a platform that is independent of that lifecycle.
•Tenant isolation that mirrors the cluster model. A hard-isolated control plane per tenant should be matched by hard isolation at the data layer: separate tenants, namespaces, credentials and QoS, not just separate directories.
•Standard interfaces from day one. Each new workload cluster needs a CSI driver (and increasingly COSI for object buckets) that can be installed as part of the release set or the day-2 runtime, so tenants get file, object and block storage the moment their cluster flips to Provisioned.
•Performance that keeps GPUs fed. A fleet that NVKE can spin up in minutes is only valuable if checkpoints and data loading do not leave those GPUs idle.

This is the space where a shared, multi-tenant data platform like VAST naturally sits: one namespace serving NFS, S3 and NVMe/TCP, exposed to Kubernetes through CSI, with tenant-level isolation and QoS, outliving any individual cluster. To be clear, NVIDIA has not announced any storage integrations for NVKE, and the preview does not define one. But as NVKE matures toward GA, I expect the question of how persistent data attaches to an NVKE-managed tenant cluster to become one of the first things AI factory operators ask.

Bottom line

NVKE and NVKR are NVIDIA’s answer to the least glamorous and most expensive part of running a GPU cloud: building, qualifying and operating many isolated Kubernetes clusters on bare metal. The design is opinionated in the right places (pinned builds, attested nodes, hard tenant isolation, upstream Kubernetes), honest about its preview gaps, and already attracting both cloud partners and platform vendors. It is not ready for production, and it does not change anything about GPU scheduling inside a cluster. But it is a clear signal of where the AI factory operating model is heading, and it is worth spinning up the local quickstart this week to see it for yourself.

Related on Lots of Data: AI on Kubernetes and Remote Storage · VAST CSI 2.7 for Kubernetes · From VM Sprawl to Agent Sprawl