From Fibre Channel VAAI to NVMe/TCP: why Full Copy still matters, and what Instant Data Cloning for Block changes for today’s clones.

We recently released VAST AI OS 5.5 which i wrote about here VAST AI OS 5.5 Part 2 & VAST AI OS 5.5 Part 1

now it’s time for a deep-dive around one of the features we’ve improved: VMware XCOPY

VMware XCOPY is not a new idea. It is the Full Copy primitive from VAAI — the same offload that Fibre Channel arrays have used for more than a decade to clone VMs, deploy templates, and Storage vMotion without the ESXi host reading every block. What is new is the transport. On NVMe/TCP, there is no SCSI CDB. vSphere 8.0 U3 and later maps the same intent to NVMe Cross Namespace Copy. VAST AI OS 5.5 implements the array side as Instant Data Cloning for Block: the target references the same flash extents instead of rewriting them.

On a small two-CNode lab cluster we observed cloning up to seven to eight times faster with that offload enabled versus the host software Data Mover. Exact wall-clock times depend on guest size, uniqueness of data, and host load. The ratio is the planning number: on modest hardware, array-side clone is in a different class from a host copy.

A short history of VMware XCOPY

Before VAAI, a clone was a host job. ESXi’s VMkernel software Data Mover read the source VMDK and wrote it back across the SAN. Large guests took minutes to hours. The copy burned CPU, DMA buffers, and HBA queue slots that production VMs needed. Storage vMotion on the same array looked the same from the host: a full read/write of the virtual disk.

VMware shipped vStorage APIs for Array Integration (VAAI) with vSphere 4.1 in 2010, first for block protocols on Fibre Channel, iSCSI, and FCoE. Three SCSI T10 primitives defined the block contract:

• Hardware Accelerated Copy (XCOPY). SCSI Extended Copy, opcode **0x83**. Clone VM, deploy from template, and same-array Storage vMotion ask the array to copy extents. ESXi stops being the copy engine.
• Hardware Accelerated Locking (ATS). SCSI Compare and Write, opcode **0x89**. VMFS metadata updates, snapshots, and power-on no longer take SCSI reservations that stall the whole LUN.
• Block zeroing (WRITE SAME). SCSI WRITE SAME, opcode **0x93**. Eager-zero thick disks and thin extends without the host streaming zeroes.

vSphere 5.0 added NAS offloads and thin-provisioning primitives (UNMAP, out-of-space behavior). The industry still talks about “VAAI” as if it were one feature. Operators care about XCOPY, because that is the clone, the template, and the Storage vMotion.

On Fibre Channel the contract is simple. The HBA sends a SCSI XCOPY CDB. A VAAI-capable array copies internally on the backend. The host sees a completed task, not a multi-gigabyte data path. That model held for fifteen years of VMFS on FC and iSCSI. NVMe/TCP broke the assumption that the wire still speaks SCSI.

How Fibre Channel VAAI translates to NVMe/TCP

NVMe over TCP is not SCSI with a different cable. There is no 0x83 CDB for ESXi to issue. The intent is unchanged — “copy these logical blocks from namespace A to namespace B without pulling them through the host” — but the command set is NVMe.

vSphere 8.0 Update 3 is the line in the sand. For NVMe-oF, including NVMe/TCP, Hardware Accelerated Copy maps to NVMe Copy / Cross Namespace Copy (CNC). ESXi still exposes the same administrator toggle, `DataMover.HardwareAcceleratedMove`. The datastore still reports Hardware Acceleration: Supported. Underneath, the adapter issues an NVMe copy between namespaces instead of a SCSI Extended Copy.

That mapping is why a VAST block volume on NVMe/TCP can participate in the same VMware workflows FC arrays have used for years: clone, deploy-from-template, same-array Storage vMotion. ATS and WRITE SAME have NVMe equivalents as well; they still matter for VMFS locking and zeroing. Extended XCOPY (ex clone, typically used to copy between arrays) is a different SCSI feature. VAST documents that it is not supported — same-array Full Copy is the contract.

VAST AI OS 5.5 is the array-side half of that translation. Instant Data Cloning for Block does not read a source range and rewrite it. The target references the same flash. The NVMe copy completes as a metadata operation. Host CPU, NVMe queues, and VIP bandwidth stay available for running VMs. VMS Analytics in 5.5 makes the offload visible: Block Volume IOPS by Size — XCOPY, Offloaded IOPS, Offloaded I/O latency, and Data Flow on the NVMe/TCP VIPs.

Today’s use cases

The primitives have not changed. The workloads have. Clones are no longer a once-a-quarter gold-image exercise. They are how platform teams ship environments to software engineers, how VDI farms refresh, and how DR drills stay honest.

• Golden images and fleet clones. Patch a gold VM once, clone it across the cluster. XCOPY keeps the rollout off the host data path so login storms and batch clones do not fight production I/O.
• Templates and VDI / lab farms. Deploy-from-template is the same Full Copy primitive. Desktop pools, training labs, and ephemeral desktops refresh at array speed instead of streaming every block through ESXi.
• Storage vMotion on the same array. Moving a VM between two VMFS datastores that both live on VAST should not be a host copy. Offload keeps maintenance, Storage DRS, and datastore rebalance online without saturating NVMe/TCP paths.
• Test/dev refresh and DR drills. Cloning a production-sized database VM into an isolated sandbox is a terrible host-copy candidate: unique data, large VMDKs, repeated runs. Array-side clone makes the drill cheap enough to run often. Referenced extents stay space-efficient until the clone diverges.
• CI/CD and inner-loop provisioning. Per-PR job VMs, parallel test matrices, and a self-service “clean box” for application engineers. This is where clone time stops being a benchmark and becomes a product the platform team can offer. Detail below.

CI/CD: rapid provisioning for software engineers

Containers did not retire VMs in CI. A large share of company engineering still needs a real guest OS: Windows MSBuild agents, licensed compilers, vendor appliances, CUDA or FPGA toolchains, fat integration stacks (app + database + message bus), and UI farms that only fail on the corporate image. Those jobs are still cloned from a gold VM on vSphere. The bottleneck is not Git. It is how long that gold image takes to become a throwaway machine.

When the clone is a multi-minute host copy, pipelines serialize, engineers reuse dirty shared agents, and “fresh env per PR” stays a slide. When the same clone is an array metadata operation — in our two-CNode observation, up to 7–8× faster — provisioning becomes a pipeline step. Compile and test dominate wall-clock time again.

The pattern: gold in, ephemeral VMs out

Keep one (or a few) gold images on the VAST VMFS datastore: patched OS, SDKs, certs, test fixtures, seed data. The CI orchestrator — Jenkins, GitLab CI, GitHub Actions runners that call vSphere, Tekton, or Backstage — requests N clones. ESXi issues hardware-accelerated copy. VAST 5.5 turns that into a metadata operation; the new VMDK references the same flash extents until the job writes. The job runs, artifacts are collected, the VM is destroyed.

Monday morning: a dozen engineers need a clean environment

Twelve software engineers (or twelve parallel PR jobs) hit provision at the start of a sprint day, from the same gold image on the same two-CNode cluster.

• XCOPY off: Each clone is a host Data Mover job. Serialized standup is a large fraction of the morning. Even in parallel, ESXi CPU, NVMe queues, and fabric are busy copying instead of running tests. Teams cope with a pool of long-lived agents that drift.
• XCOPY on: Each clone is a metadata operation on the array. A dozen environments stand up in the same class of time as a single offloaded clone, not an hour of host copy. Fresh image on every PR becomes cheaper than arguing about a shared lab. That is the 7–8× observation applied to a standup burst.

Rapid operations the platform team can actually offer

Once clone time drops by that order of magnitude, these become catalog items:

• Per-PR ephemeral environments. Every pull request gets its own VM (or a small set: unit, integration, UI).
• Parallel test matrix. Windows build, Linux integration, Playwright, and a scanner as four clones of the same gold image, started together.
• Merge-queue / gate VMs. Gated check on main: clone, run the contract suite, destroy.
• Nightly reset. Rebuild the integration farm from gold overnight. Known-good fixtures, not a snowflake cluster.
• Onboarding workstation. New-hire day-one VM from the same catalog the pipeline uses.
• Data-heavy integration. Clone a production-sized DB VM for a schema migration rehearsal on the PR, not only on the release weekend.

Inner loop: “give me a clean box”

CI is the batch path. Engineers also need an interactive path. Put a catalog in Backstage, ServiceNow, or a Jenkins self-service job: “Java 21 + Postgres fixture,” “Windows MSBuild agent,” “CUDA toolchain.” The portal calls the same vSphere clone the pipeline uses. XCOPY on VAST means SSH or RDP in about a minute — not a ticket that waits on a host-side copy.

When a clean VM is expensive, people hoard them. When it is cheap, they destroy and reclone the way they already destroy containers. Platform teams stop rationing environments and start versioning gold images.

What VAST AI OS 5.5 adds

Instant Data Cloning for Block converts block-level copy from data movement into a metadata operation. Large copies — VM clones, Storage vMotion, SMB server-side copy — complete without the host read/rewrite path and without taxing write buffer the way a full copy would.

Primitive	VMware operations	VAST 5.5
XCOPY / NVMe Copy	Clone VM, deploy from template, same-array Storage vMotion	Supported (not Extended XCOPY)
ATS	VMFS locking, snapshots, VM power-on	Supported
WRITE SAME	Thin provision, eager-zero, extend	Supported

Requirements that matched the lab: VAST AI OS 5.5 block view and volume, ESXi host registered by NQN, NVMe/TCP adapter with Hardware Acceleration Supported, vSphere 8.x (8.0 U3+ for NVMe Cross Namespace Copy), and `DataMover.HardwareAcceleratedMove` left at the default of `1`.

Host Data Mover versus array clone

The A/B test is the same one FC operators have run for years. Disable `DataMover.HardwareAcceleratedMove` and clone on the same datastore: ESXi uses the software Data Mover. Enable it again and clone the same VM: ESXi issues hardware-accelerated copy, which on this fabric is NVMe Cross Namespace Copy, handled on VAST as Instant Data Cloning for Block.

On this two-CNode cluster, the offloaded clone was up to seven to eight times faster than the host copy of the same unique disk. That is an observation, not a datasheet guarantee. A larger cluster, a colder cache, or a thinner guest will move the ratio. What did not move: with XCOPY on, the host was no longer the copy engine, and VMS showed the offload on the volume.

How to wire it without fighting the array

• Keep source and target on VAST. Gold images live on the same VAST VMFS / NVMe subsystem as the job VMs. XCOPY does not follow a copy that leaves the cluster.
• Do not disable XCOPY in CI. Leave DataMover.HardwareAcceleratedMove = 1. A pipeline that disables it “to be safe” puts you back on the host Data Mover.
• Treat VMs as ephemeral. Clone, run, delete (or revert). Do not recycle job VMs across unrelated PRs.
• Patch the image, not the fleet. Patch gold once, then let XCOPY fan it out.
• Prove it in VMS. Watch Block Volume IOPS by Size — XCOPY and Offloaded IOPS during a standup burst. If host CPU climbs and offloaded IOPS stay flat, clones are not offloading.

What to take into production

• Keep Hardware Acceleration on. Leave DataMover.HardwareAcceleratedMove = 1 except when measuring a baseline.
• Stay on one array / subsystem. XCOPY does not help a copy that leaves the VAST cluster.
• Use vSphere 8.0 U3+ for NVMe/TCP offload. Pair VAST AI OS 5.5 block with vSphere 8.0 U3+ NVMe Cross Namespace Copy. That is the FC-to-NVMe translation.
• Watch the new XCOPY graphs. If Offloaded IOPS stay flat while host CPU climbs, the copy is not offloading.
• Treat unique data honestly in a PoC. Empty thin disks clone quickly even without XCOPY. Unique data is what makes a PoC honest.

Monitoring XCOPY

In VAST AI OS 5.5, we also added to monitor the traffic XCOPY generates, directly from the UI

XCOPY is useful because it returns ESXi to running VMs — the same reason it shipped on Fibre Channel in vSphere 4.1. On NVMe/TCP the SCSI CDB is gone; Cross Namespace Copy carries the intent. VAST AI OS 5.5 makes that offload a metadata operation on block and lets you see it in VMS. On a two-CNode lab cluster, that was up to 7–8× faster cloning than the host Data Mover. That is the integration: not a faster host copy, but no host copy for the work VAAI was designed to give back to the array.

Related on Lots of Data: VAST AI OS 5.5 Part 1 · VAST AI OS 5.5 Part 2 · VAST NVMe/TCP for VMware