Every CSI release has a theme, whether the release notes admit it or not. For 2.7, the theme is day-two operations. Getting a volume into a pod has been a solved problem for a while. The harder questions come later: what happens to that data if a site goes down? How do you give fifty application teams their own S3 buckets without handing each of them a storage admin? How do you encrypt NFS traffic per tenant when you don’t control the worker node images?

VAST CSI Driver 2.7.0 tackles the last two head-on. It turns the COSI object driver into something a platform team can actually build a self-service offering on, and makes NFS encryption and bucket access much easier to set up. For the first question, the replication introduced in 2.6.6 is now fully documented and supported. This post walks through what’s new in plain language, with the gotchas worth knowing before you upgrade.

Figure 1: The three areas that 2.7 improves

Figure 1: The three areas that 2.7 improves

1. Object storage: COSI grows up

COSI (Container Object Storage Interface) is to S3 buckets what CSI is to volumes. An app asks for a bucket with a BucketClaim and gets back credentials, with no ticket to the storage team. Earlier releases covered the basics. 2.7 adds most of what’s needed to run COSI in production and share it between teams.

Figure 2: From platform policy to a running app, in five objects

Figure 2: From platform policy to a running app, in five objects

Quotas: buckets with a size limit

A BucketClass can now carry a max_size (Helm: maxSize), and every bucket created from it gets a hard quota on VAST. An individual team can override it on their own claim with the cosi.vastdata.com/maxSize annotation, and the claim value wins.

kind: BucketClaim

apiVersion: objectstorage.k8s.io/v1alpha1

metadata:

name: team-a-logs

annotations:

cosi.vastdata.com/maxSize: “5Gi” # overrides the class default

spec:

bucketClassName: vastdata-bucket

protocols: [s3]

Two things to know. First, the quota sits on the bucket’s path, so it counts data written through any protocol (S3, NFS or SMB), not only S3. Second, the quota is set once, when the bucket is created. Changing max_size on a BucketClass later doesn’t resize existing buckets, and changing or removing the quota on an existing bucket isn’t supported.

Lifecycle rules: buckets that clean up after themselves

Add lifecycleRules to a BucketClass and VAST creates S3 lifecycle rules when the bucket is created, and removes them when the bucket is deleted. A typical use is to expire logs/ after 30 days and tmp/ after 7, and to clean up abandoned multipart uploads. Nobody has to remember to do it. As with quotas, the rules are applied at creation time and can’t be modified afterwards through the driver.

Versioned buckets

Set s3_versioning: “true” on a BucketClass and every bucket created from it keeps previous versions of overwritten or deleted objects. It works well with lifecycle rules that expire old versions. Note that versioning can’t be disabled or suspended once a bucket is created with it.

Writable bucket clones (in the repo)

Need a copy of the production training dataset for a dev experiment? Annotate a new claim with cosi.vastdata.com/sourceBucket: <source> and you get a writable clone based on a VAST snapshot. The clone is available almost immediately, however large the source is. For production use, add cosi.vastdata.com/blockingClones: “true” so the claim only completes when the clone is fully ready. This capability is in the v2.7 GitHub changelog and examples but isn’t listed in the official COSI 2.7.0 release notes yet, so check with VAST before relying on it in production.

Credentials that apps can actually consume

•Flat credentials. Add the annotation cosi.vastdata.com/flatten-credentials: “true” to a BucketAccess. The driver then creates a sibling *-flat Secret and ConfigMap with ordinary AWS_* and BUCKET_* environment variables, the same layout Rook uses. Apps can load them with envFrom, with no JSON parsing and no init container.

•Bring your own keys. If your keys come from Vault or another secrets manager, point the BucketAccessClass at a Kubernetes Secret (credentialsSecretName / credentialsSecretNamespace) and the driver installs those keys on VAST. This requires VAST 5.4 or later and works for local VAST users only. To rotate, update the Secret and recreate the BucketAccess.

•Existing owners. Buckets can be owned by an existing VMS local, Active Directory, LDAP or NIS user, such as a corporate S3 service account, via bucket_owner and bucket_owner_context. Deleting the bucket doesn’t delete that user. Keep in mind that VMS allows only two access keys per user per tenant.

Real multi-tenancy

Each BucketClass can now use its own VMS credentials (vastdata.com/secret-name and vastdata.com/secret-namespace). That means one COSI driver can serve several VAST tenants, each using a tenant-admin account (this requires VAST 5.3.3 or later). Pair this with vipPoolFQDN so each tenant’s buckets get a DNS-based endpoint (for example s3-team-a.example.com) instead of a raw VIP pool name. Single-tenant installs can keep using one global secretName.

Naming rule to remember

S3 bucket names max out at 63 characters, and COSI appends a 36-character claim ID. Keep BucketClass names to 27 characters or fewer. In 2.7 the driver rejects names that are too long instead of quietly truncating them, so the error appears at creation time rather than as a strange name later.

2. File and block: security and polish

Per-tenant NFS mTLS, no host changes

NFS over mutual TLS (VAST 5.5+) has one practical obstacle: the TLS handshake daemon, tlshd, normally has to be configured on every worker node. In a shared or managed Kubernetes environment you often can’t touch the nodes, and different tenants need different certificate authorities.

In 2.7, the driver’s csi–nfs-services sidecar can run tlshd itself. You give it a ConfigMap with tlshd.conf and a Secret with the tenant’s CA and client certificate (node.nfsServices.tlshd in Helm). The sidecar then handles the handshake using those files, even if the host also has its own tlshd. The certificates stay in a dedicated vastcsi keyring.

Figure 3: TLS configuration lives in Kubernetes objects, not on the node image

Figure 3: TLS configuration lives in Kubernetes objects, not on the node image

To use it, add xprtsec=mtls to the storage class mountOptions. With NFSv3, also add mountproto=tcp. With NFSv4, leave mountproto out.

Mount an existing S3 bucket as a folder

VAST is multi-protocol: the same data can be read over S3 and NFS. 2.7 brings that into pods. An inline CSI volume with volumeAttributes.bucket_name mounts an existing S3 bucket over NFS. A pipeline that writes objects through S3 can now feed a training job that just reads files, with no copying involved.

Smaller fixes that matter

•Leaner NFS mounts. The driver no longer adds mountproto=tcp to every NFS mount by default. The exception is NFSv3 with TLS, as described above.

•OpenShift-safe snapshots. OpenShift tends to generate long project and snapshot names, which could exceed the 128-character limit in VMS. The new truncateSnapshotName setting (default 128) shortens them safely.

•RHEL 10 block multipath. RHEL 10 no longer exposes one of the sysfs files the driver checked for NVMe multipath. The driver now recognizes multipath correctly there, and still refuses to proceed when it’s clearly disabled.

•More reliable bucket credentials. The COSI sidecar moves to objectstorage-sidecar v0.2.2, which fixes an intermittent case where a bucket was provisioned without S3 access keys (VCSI-520). A related fix makes the BucketAccess secret resync correctly when a BucketClaim is recreated (VCSI-587).

•Block read latency. The Block CSI driver adjusts internal timeouts that could, in some cases, increase read latency (VCSI-565).

•Deleting volumes with snapshots. A PVC can now be deleted even if snapshots were taken from it. Previously the driver blocked the deletion.

•Install first, add storage classes later. The CSI and Block CSI Helm charts now install even if the values file defines no storage classes.

3. Replication: from new feature to fully documented

Asynchronous volume replication isn’t new in 2.7. It arrived in 2.6.6, together with VAST 5.5, and I covered it in detail, including a full failover walkthrough, in my earlier post, Replicating Kubernetes Workloads with VAST Data 5.5 & CSI 2.6.6 (link in the resources below). If you haven’t set it up yet, start there.

So what changes in 2.7? The concepts, custom resources and requirements stay the same. The difference is in how finished the feature is:

•Official documentation. The 2.7 Administrator’s Guide now has a full section, “Replicate Volumes with VAST CSI Replication Operator”, covering prerequisites, both custom resources, multi-destination topologies and failover. The 2.7 release notes list replication as a headline feature for this reason: it’s now fully documented and supported.

•More ready-made examples. The v2.7 repo adds separate NFS and block examples for the operator-based setup, plus Helm-based volume and storage-class replication examples.

•Renamed sidecar commands. The replication sidecar now uses the replication and server subcommands. The Helm charts already include this, but custom manifests need updating (see “Before you upgrade”).

•A documented limitation. On a cluster that is a source or destination of replication, you can’t create a PVC from a snapshot or from another PVC. Keep volumes that depend on clones out of replicated storage classes.

Figure 4: Replication topology, unchanged from 2.6.6 and now fully documented

Figure 4: Replication topology, unchanged from 2.6.6 and now fully documented

A quick recap for anyone new to it: you declare what to protect with a VastVolumeReplication (specific volumes) or a VastStorageClassReplication (everything in a storage class). The VAST CSI Replication Operator then creates the protection policies, protected paths and replication streams on VAST. You fail over with vcsi failover or by changing primaryStorageClass. Requirements are still VAST 5.5.0+, Kubernetes 1.26+, and a VMS Secret with a username and password (API tokens aren’t supported). If one source replicates to several destinations, define every cluster pair (for example B to C as well) so any surviving cluster can take over as primary.

4. Before you upgrade

Most of 2.7 is opt-in, but check these before upgrading:

•Breaking change for custom manifests. Two extensions-controller entrypoints were renamed: pvc-label-webhook is now webhook, and the replication sidecar now uses the replication and server subcommands. The Helm charts already include this. If you maintain your own manifests with the old arguments, update them.

•Air-gapped sites need to mirror registry.k8s.io/sig-storage/objectstorage-sidecar:v0.2.2. That sidecar currently runs as root, so it won’t pass the Kubernetes Restricted Pod Security Standard yet. A non-root option is planned.

•VMS secret hygiene. Only use the tenant: field for tenant-admin username/password credentials. Leave it out for cluster-admin accounts and for any API token. An API token combined with tenant: can make VMS return HTTP 400.

•Coming from before 2.6.3? First delete and recreate your Helm-created storage classes, so that secretName and secretNamespace are applied correctly. From 2.6.3 or later, a normal helm upgrade is enough.

•Running pods keep the old version until they’re recreated. New pods use 2.7 immediately.

Compatibility at a glance

Area	Requirement
Kubernetes	1.25 to 1.36 per the admin guide; the v2.7 branch is CI-tested on 1.25, 1.29, 1.31 and 1.33
OpenShift	Supported through the VAST CSI Operator
VAST Cluster (base)	4.1 or later
Async volume replication (since 2.6.6)	VAST 5.5.0+, Kubernetes 1.26+, username/password VMS secret
NFS mTLS	VAST 5.5.0+
COSI multi-tenant (per-BucketClass secrets)	VAST 5.3.3+
COSI bring-your-own keys	VAST 5.4+, local VAST users only
Writable clones	VAST 4.6.0+ (Global Snapshot Clones)

Wrapping up

If you run stateful workloads on Kubernetes with VAST, 2.7 is mostly about making the second year as smooth as the first day. COSI now covers what a self-service object platform needs: quotas, lifecycle rules, clones, tenant isolation and credentials apps can use directly. On the NFS side, mTLS and bucket mounts become Kubernetes objects instead of node-level projects. And replication, introduced in 2.6.6, now has the documentation and examples to go with it.

The best place to start is the examples/ folder in the v2.7 branch of the repo. It has ready-to-apply YAML for almost everything described here, from cosi-bucket-clone.yaml to tlshd-mtls-truststore.yaml.