AI workloads, data platforms, and infrastructure notes, written from the engineering edge between benchmarks and production.

RSS feed
/ / /

VAST AI OS – 5.5 Replicating Kubernetes workloads with CSI 2.6.6

Replicate Kubernetes workloads with VAST Data 5.5 and CSI 2.6.6: protected paths, Global Snapshot Clones, and practical DR for OpenShift and cloud-native apps.

I

Itzik — VP Mission Alignment, VAST Data

·

·

14 min read


Replicating Kubernetes workloads with VAST Data 5.5 & CSI 2.6.6

Replicate Kubernetes workloads with VAST Data 5.5 and CSI 2.6.6: protected paths, Global Snapshot Clones, and practical DR for OpenShift and cloud-native apps.

A Practical Kubernetes Disaster-Recovery

.

Kubernetes makes it easy to schedule applications, but it does not make a stateful application recoverable across storage sites by itself. A PersistentVolumeClaim (PVC) tells Kubernetes where an application should mount storage; it does not define how the data reaches a second VAST cluster, how a remote copy becomes writable, or how a workload switches to that copy during an outage.

Starting with VAST CSI Driver 2.6.6 (General Availability, August 2026), VAST Block CSI supports asynchronous replication of Block volumes between VAST clusters, including failover to a secondary peer. Data from one source cluster can be replicated to multiple destination clusters.

.

VAST CSI Replication Operator closes that gap for VAST Block deployments. The extensions controller ships with the VAST Block CSI Helm chart and turns a Kubernetes declaration into the VAST protection policy, protected path, replication stream, CSI-addons resources, destination PV/PVC pairs, and—when enabled—pod remapping needed for a controlled role change.

This guide explains the model and then walks through a two-site VAST Block deployment. It aligns with the published Replicating Block Volumes with VAST CSI Replication Operator documentation and the VAST CSI Driver 2.6.6 release notes, reviewed on August 5, 2026.

Scope: This is asynchronous, active/standby replication: one site is primary at a time. Your recovery point objective (RPO) is bounded by the protection-policy cadence, not by Kubernetes. Test the procedure on non-production data before using it for a production runbook.

What Kubernetes replication needs to solve

A useful disaster-recovery workflow has to coordinate four things:

Data protection: snapshots and replication move Block data between VAST clusters.
Kubernetes identity: the destination needs static PV/PVC objects that represent the replicated data.
Application placement: after a role change, pods need to mount the destination claims rather than the former source claims.
Operations: a team needs a clear way to observe state, initiate a planned or unplanned failover, and safely return to the original site.

.

Without an operator, each layer is a separate integration task. An administrator must pre-create array-side objects, create or update CSI-addons resources, provision matching destination claims, and coordinate workload changes. The VAST CSI Replication Operator owns those integration points from a VastStorageClassReplication (VSCR) or VastVolumeReplication (VVR) resource.

Using the information in those CRDs, the operator can:

create a NATIVE_REPLICATION ProtectionPolicy on the source VAST cluster for each pair of clusters listed in the CRD;
create a ProtectedPath on the source peer and associate a replication stream for each destination peer;
create the VolumeReplicationClass and VolumeGroupReplicationClass definitions required by the driver;
start replication according to the schedule in the CRD;
on failover: create or delete PVC-PV pairs on each former destination, create replica objects (volumes), and—when pvcRemap: true—restart pods that used source PVCs and redirect them to the matching claims on the new primary.

.

Choose the right replication scope: VSCR, VVR, or subsystem

The published Block replication guide supports three related patterns:

One-volume replication with a VVR: one named PVC, independent schedule or topology.
Multi-volume replication with a VSCR: every PVC created with a particular StorageClass (optionally restricted to a volume group / subpath).
Subsystem replication: replicate volumes of a particular Block subsystem; the destination view is created automatically from the StorageClass subsystem specification.

.

Decision

VSCR

VVR

Scope

All PVCs on a StorageClass (optionally a volume group)

One named PVC (volumeName)

Destination PVC timing

Waits at least one replication cycle before creating destination PVCs

Creates the destination PVC without that additional wait

Typical fit

Application group or common tier

One-off workload or independent SLA

.

This guide uses a VSCR because it is the clearest pattern for a Block application with several claims.

The VAST Block model

Figure 2. The controller connects Kubernetes declarations to VAST replication objects.

The Block StorageClasses are the bridge between Kubernetes and the two VAST clusters. Each site has its own:

StorageClass;
secret that contains that site’s VMS credentials;
Block subsystem;
VAST ReplicationPeer relationship with the other site.

.

For two sites, create one replication peer relationship. For three sites, create one peer relationship for every pair in the mesh. The VSCR topology is undirected: two sites need one entry; three sites need three; in general, n sites need n(n-1)/2 entries. Native Replication uses a dedicated VAST replication VIP pool; it is distinct from the protocol VIP pool that serves workload I/O.

For Block, the StorageClass subsystem parameter identifies the Block subsystem. The native-replication data plane is path-based on the BLOCK view: every volume under the protected view path is replicated as part of that protection domain, rather than being independently replicated by the array. The admission webhook injects these labels when a matching PVC is created:

vastdata.com/storageClass on every PVC;
vastdata.com/subsystem on Block PVCs.

.

Those labels allow the volume-group replication selectors to locate the intended claims without developers hand-authoring replication labels.

Requirements

VAST cluster

VAST clusters running release 5.5.0 or later, with network connectivity through their replication ports.
On the source cluster, preconfigure:
an Element Store view that exposes the subsystem with the source volumes;
a replication virtual IP pool for asynchronous Block replication;
an asynchronous replication peer for each destination cluster.
On each destination cluster, preconfigure:
a replication virtual IP pool;
for single-volume or multi-volume replication: a view with the same subsystem name as on the source;
for subsystem-level replication: do not manually create the destination view—the operator creates it from the StorageClass subsystem specification.

.

These preconfiguration steps are performed with the VAST Web UI, CLI, or REST API, as described in the VAST Cluster Administrator’s Guide.

Kubernetes

Kubernetes 1.26 or later.
A Kubernetes secret containing VMS credentials for each source and destination cluster.
Secrets that use VMS API tokens are not supported for this workflow.
Do not put real credentials in Helm values, source control, screenshots, or narrated recordings.

.

Application and operational assumptions

Decide the ProtectionPolicy schedule and retention. Remember that lowercase m means minutes, while uppercase M means months (30 days).
Validate the plan with application owners. A replicated volume group is crash-consistent; databases that need application consistency should quiesce or use their own recovery process.
Confirm the DR scope. The operator does not migrate pods between Kubernetes clusters, and COSI/S3 bucket replication is not part of this Block CSI workflow.
Note the 2.6.6 limitation VCSI-488: creating a PVC from a snapshot (or from another PVC) on a source or destination replication peer is not allowed.

.

Upgrade to VAST CSI Driver 2.6.6

Block CSI replication requires CSI Driver 2.6.6 or later. If you are already on a Helm-based deployment from 2.2.1 or later:

Refresh Helm repository data (helm repo update).
Point the chart values at the 2.6.6 image tag.
Run helm upgrade against the existing release.
Confirm the new chart with helm ls.

.

Older upgrade paths (2.2.0, 2.1.x, 2.0.x, 1.0.x) follow the published Steps to Upgrade VAST CSI Driver guide: some require removing the old chart, redeploying 2.6.x, and optionally migrating PVs or creating View-per-Volume views for existing PVCs. Chart packages and assets are published under the helm-vastcsi-2.6.6 (and related Block / operator) tags on the vast-csi GitHub releases page.

Step 1: Configure Block StorageClasses and enable the Replication Operator

Add the extensions controller to the Block CSI Helm chart configuration, then deploy or upgrade the chart.

storageClassDefaults:
  vipPool: vippool-1
storageClasses:
  vastdata-block:
    secretName: vast-mgmt-a
subsystem: source
    volumeGroup: “/k8s”
vastdata-block2:
    secretName: vast-mgmt-b
subsystem: destination
    volumeGroup: “/k8s-repl”
extensions:
enabled: true
webhook:
    disablePvcLabelsWebhook: false
    filterConfig:
      storageClassNameRegex: “^vastdata-.*”
replication:
    pvcNameFormat: “{pvc_name}-repl-{endpoint}”
    pvNameFormat: “{pv_name}-repl-{endpoint}”
  applyExistingPvcs: true
image:
  vastExtensionController:
repository: vastdataorg/csi
tag: v2.6.6-extensions

applyExistingPvcs: true is important if PVCs existed before the webhook was enabled. On each primary-StorageClass change, the controller can backfill the storage-class label (and the subsystem label for Block StorageClasses) for existing PVCs whose backing VAST object is known in the relevant VolumeMapping.

Note on the extension controller image: the chart defaults vastExtensionController.tag to {{ .Values.image.csiVastPlugin.tag }}-extensions, so it normally tracks the driver tag automatically with an -extensions suffix appended. If you pin it explicitly, as shown above, keep that suffix — a bare v2.6.6 tag is the main CSI plugin image, not the extensions controller image, and will not run the replication controller.

Destination PVC and PV names default to {pvc_name}-repl-{endpoint} and {pv_name}-repl-{endpoint} (for example, myapp-data-repl-16-0-0-2). You can change the format with placeholders such as {pvc_name}, {pvc_name_suf:<N>}, {endpoint}, {sc_name}, and the corresponding PV / prefix variants documented in the administrator’s guide.

To restrict a VSCR to a volume group rather than every claim on the StorageClass, set volume_group (or the chart-equivalent volumeGroup) on the source StorageClass.

Apply the chart and make sure the extensions controller is running:

helm upgrade –install csi charts/vastblock/ -f values.yaml
kubectl get pods -n vast-csi

Step 2: Declare the protection topology with a VSCR

The VSCR expresses the intended state. It names the primary StorageClass, the two-site topology, a snapshot/retention policy, and the synchronization behavior.

apiVersion: vastdata.com/v1alpha1
kind: VastStorageClassReplication
metadata:
name: app-replication
namespace: default
spec:
  primaryStorageClass: vastdata-block
  protectionTopology:
– source: vastdata-block
destination: vastdata-block2
# peerName: cluster-a-cluster-b-peer # optional when one peer is discoverable
  protectionPolicyTemplate:
params:
– every: 15m
        keepLocal: 2d
        keepRemote: 7d
  failoverType: gracefulFailover
  syncIntervalSeconds: 900
  syncPVCPV: true
  syncVastObjects: true
  pvcRemap: true
  destVolReclaimPolicy: Retain

For one source and two destinations (A -> B and A -> C), include the full undirected mesh so failover can resynchronize every pair—for example A-B, A-C, and B-C.

Key fields:

primaryStorageClass is the site that is writable now.
protectionTopology names the StorageClass pairs. Set peerName when peer auto-discovery is ambiguous; otherwise the controller expects one shared peer between the two StorageClasses. The value matches the Peer Name property in VMS.
protectionPolicyTemplate defines the ProtectionPolicy. every is the snapshot cadence; keepLocal and keepRemote define retention. You may use multiple schedule frames and an optional startAt (YYYY-MM-DD HH:MM:SS or HH:MM). Units: s/S seconds, m minutes (lowercase only), h/H hours, d/D days, w/W weeks, M months (uppercase only; 30 days), y/Y years (365 days).
syncIntervalSeconds is the maximum permitted gap between synchronization points — when omitted or zero, it is derived from the every value of the first protectionPolicyTemplate entry. There is no fixed system default; 900 above follows from every: 15m, not from a hardcoded value.
syncPVCPV: true creates destination static PV/PVC pairs so workloads can mount the replica after failover (default true).
syncVastObjects: true keeps destination Block volumes synchronized.
pvcRemap: true restarts pods and directs them to the matching mirror claim after a role change. The default is false; enable it only after testing the restart behavior.
destVolReclaimPolicy is Retain (default) or Delete. It controls whether destination volumes are kept or deleted when the VSCR or VVR is removed.

.

For a single named volume, use a VastVolumeReplication with the same fields plus volumeName.

Apply and watch the resource:

kubectl apply -f app-replication.yaml
kubectl get vscr -w
kubectl get volumegroupreplicationclass
kubectl get volumegroupreplication
kubectl get events –sort-by=‘.lastTimestamp‘ | tail -20

A healthy VSCR converges to syncStatus: Completed. After the source PVCs are created, replication proceeds on the schedule defined in the CRD.

Step 3: Verify both Kubernetes and VAST state

The status fields are a quick health check:

Field

What it means

syncStatus

Completed, InProgress, Unreachable, Error, Invalid, Deleting, or Failed (a permanent infrastructure error that requires user intervention)

currentPrimaryStorageClass

The StorageClasscurrently treated as primary

ppathName

The VAST ProtectedPath created by the controller

ppathDirMapping

Maps every StorageClass in the constellation to its predicted ProtectedPath source directory

lastFailoverType

The most recent failover type (graceful or ungraceful) that was applied

.

On the VAST primary, verify:

a ProtectionPolicy for the peer relationship;
a ProtectedPath with a Source role;
a replication stream for the destination.

.

On Kubernetes, verify that the primary volume-group replication object has replicationState: primary and the destination has replicationState: secondary.

The operator also ships a vcsi CLI (GitHub release v1.0) for day-2 operations:

vcsi list
vcsi status —vscr app-replication

Step 4: Create application PVCs

PVCs remain ordinary Kubernetes resources. A developer selects the primary StorageClass; the webhook and controller handle the replication plumbing.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myapp-data
namespace: default
spec:
  accessModes:
ReadWriteOnce
resources:
requests:
storage: 10Gi
  storageClassName: vastdata-block

Apply the claim, then inspect labels and destination objects:

kubectl apply -f myapp-data.yaml
kubectl get pvc –show-labels
kubectl get pv

With a VSCR, destination mirror creation waits until the VAST backend object is available after at least one replication cycle. Do not infer that replication failed simply because the destination PVC is not present the moment the source claim is created.

Step 5: Deploy and validate the workload

Mount the primary PVC as usual. For a first test, use a disposable workload that writes an identifiable record:

apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
namespace: default
spec:
replicas: 1
selector:
    matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
– name: writer
image: busybox
command:
sh
– -c
– while true; do date >> /data/log.txt; sleep 5; done
          volumeMounts:
– name: data
              mountPath: /data
volumes:
– name: data
          persistentVolumeClaim:
            claimName: myapp-data

Check the initial primary before proceeding:

vcsi list
vcsi status —vscr app-replication

Step 6: Perform a failover

Figure 3. A failover changes the primary and can remap workloads to mirror claims.

Choose graceful failover for planned maintenance: the controller completes a final replication step before it switches roles. Choose ungraceful failover for an outage in which the old primary cannot be assumed reachable. Details of each manner are covered in the VAST Cluster Administrator’s Guide.

Use the CLI for an explicit operational command:

vcsi failover —vscr app-replication \
–primary vastdata-block2 \
–manner graceful

Or update the VSCR declaratively (edit primaryStorageClass, then apply):

kubectl patch vscr app-replication –type=merge \
p ‘{“spec“:{“primaryStorageClass”:”vastdata-block2″,”failoverType”:”gracefulFailover”}}

Watch the transition:

kubectl get vscr app-replication -w
kubectl get volumegroupreplication
kubectl get pods -n default -w

Expected result:

The old primary becomes secondary.
The destination becomes primary.
currentPrimaryStorageClass becomes vastdata-block2.
syncStatus advances through InProgress and returns to Completed.
With pvcRemap: true, affected pods are restarted and mount the corresponding destination mirror claim.

.

Failback is another role change after the original site is healthy and replication is re-established in the new direction. Treat it as a tested operation, not as an automatic reversal.

Step 7: Trigger an immediate resync when needed

You can request on-demand synchronization in either of two equivalent ways.

Declarative, one-shot flag on the CRD:

kubectl patch vscr app-replication –type=merge \
p ‘{“spec“:{resync“:true}}’

The controller resets spec.resync to false after triggering the operation.

Or use the CLI:

vcsi sync —vscr app-replication

To remove a replication configuration when you are finished with a lab or cleanup:

vcsi delete —vscr app-replication

destVolReclaimPolicy then determines whether destination volumes are retained or deleted.

Operational guardrails

Do not put raw VMS usernames or passwords in demo YAML, terminal recordings, blog screenshots, or repositories. Use secret references and rotate any credentials that were exposed. API-token secrets are not supported for this replication workflow.
Keep the topology, StorageClasses, Block subsystems, replication VIP pools, and VAST peers consistent. A StorageClass name is part of the control-plane contract.
Make the RPO visible to application owners. A 15m schedule is not a zero-data-loss configuration.
Test both graceful and ungraceful failover, and test pvcRemap before production. A pod restart is part of the remap behavior.
Do not create PVCs from snapshots (or from other PVCs) on a replication peer while that site participates in the relationship (VCSI-488).
Verify VAST-side and Kubernetes-side state together. vcsi status —vscr is the fast operational view; VSCR status, volume-group replication objects, events, ProtectedPaths, and replication streams provide diagnosis when convergence stalls.

.

A concise runbook

Confirm VAST 5.5.0+ peers, Element Store views / subsystems, and replication VIP pools.
Upgrade or install VAST Block CSI 2.6.6 with extensions.enabled: true.
Create one StorageClass and one VMS-credential secret per VAST site.
Apply a VSCR (or VVR) and wait for syncStatus: Completed.
Create test PVCs and verify injected labels plus destination mirror objects after a replication cycle.
Mount a test workload, write known data, and verify the protected path and streams on VAST.
Run a graceful role change, then an ungraceful drill in a non-production environment.
Confirm the new primary, the pod restart/remap, and data recovery.
Capture the successful commands, timings, owners, and rollback criteria in the production DR runbook.

.

Further reading

Replicating Block Volumes with VAST CSI Replication Operator

VAST CSI Driver 2.6.6 Release Notes

Steps to Upgrade VAST CSI Driver

vast-csi GitHub releases (helm-vastcsi-2.6.6 and vcsi CLI)

Discover more from Lots of Data

Subscribe now to keep reading and get access to the full archive.

Continue reading