Inside the VAST Event Broker: Real-Time Analytics and AI at Scale (Part 2 of 2)

Part 2 of 2: How one Kafka-compatible platform stores streams as queryable tables, scales without cluster sprawl, and delivers up to ten times the performance of Kafka.

Part 1 made the case that real-time event streaming is the backbone of modern AI, and that legacy platforms built for the batch era struggle under the weight of operational complexity, scalability limits, delayed analytics, replication cost, and silos. The common thread was data movement: the same events copied and shuttled between separate systems before anyone could use them. This post is about the engine built to remove that movement. The VAST Event Broker unifies streaming, analytics, and AI on a single platform, so event data is usable the instant it lands.

Table of contents

• Executive summary
• What the VAST Event Broker is
• Built for Kafka compatibility
• Why VAST built its own broker
• The DASE foundation
• Topics become queryable tables
• Instant analytics in practice
• Core capabilities, in plain terms
• How the data flows
• The metrics that matter
• Performance and scalability
• What drives the performance edge
• Cost and operational value
• What this means for you
• A scenario walkthrough
• Key takeaways
• The VAST VAST Event Broker series hub — full series indexEvent Broker series

Fast data is only half the equation. What you can do with it the moment it arrives is what actually drives the business. Most organizations have invested heavily in streaming infrastructure and still find themselves waiting: waiting for ETL pipelines to finish, waiting for data to move between systems, waiting for the analytics layer to catch up to what the broker already received. The VAST Event Broker was built to close that gap entirely.

Executive summary

The VAST Event Broker is a Kafka-compatible event streaming platform built directly into the VAST AI Operating System (AIOS). Because it implements the Kafka protocol, it works with the client libraries, APIs, and tools your teams already use, so it can act as a drop-in replacement without rewriting applications. What makes it different is what happens to the data after it arrives: Kafka topics are automatically stored as columns in VAST DataBase (VDB) tables, which means streaming data becomes instantly queryable with SQL, with no separate ETL step and no second system to copy it into.

Underneath sits DASE, short for disaggregated, shared-everything, an architecture that separates compute from storage and connects every compute node to all of the storage over a fast network. That removes the partition rebalancing and cluster sprawl that constrain legacy brokers, and it lets performance scale linearly as you add compute nodes. The measured result is up to ten times the performance of Kafka, with sub-four-millisecond latency, and the operational result is radical simplification: store data once instead of three times, manage one platform instead of several, and deploy in minutes instead of months.

What the VAST Event Broker is

At its simplest, the VAST Event Broker brings event streaming, real-time analytics, and AI together into one platform rather than three. It integrates a real-time event architecture directly into the VAST AI Operating System, so the same system that receives your events can also analyze them and trigger AI pipelines against them. You no longer need to manage separate systems or move data between tools, because ingestion, storage, querying, and AI all happen in one place.

The VAST Event Broker unifies streaming, analytics, and AI on the VAST AI Operating System, so three systems become one platform.

That unification is the heart of the design. In a traditional stack, streaming lives in one product, analytics in another, and AI in a third, each with its own copy of the data and its own operational overhead. By bringing them onto a single platform, VAST removes the integration tax: fewer moving parts, fewer failure points, and far less time spent moving data from where it landed to where it can finally be used.

Built for Kafka compatibility

A new streaming platform is only useful if it fits the world your teams already work in, and Kafka is that world. Kafka has become the de facto standard for streaming data, with a rich ecosystem and broad adoption, so the VAST Event Broker implements the Kafka protocol directly. In practice that means producers and consumers keep using familiar Kafka client libraries and APIs, and existing data pipelines keep working. The broker can act as a drop-in replacement without forcing changes to your applications.

This matters because it removes the usual reason teams tolerate a platform that is hurting them: the fear of a painful migration. Compatibility means you can adopt the VAST Event Broker behind the same interfaces your applications already speak, and gain the benefits underneath, without a rewrite. The familiar Kafka surface stays the same, while the engine beneath it changes completely.

Why VAST built its own broker

It is fair to ask why VAST built an event broker at all rather than reusing an existing one. The honest answer is that existing event-based solutions, while mature, imposed trade-offs at the data layer that limited performance and flexibility. To support high-throughput AI and analytics workloads, VAST needed a broker that could fully take advantage of its storage architecture, and that meant building one that was optimized for both speed and resilience and that integrated cleanly with the rest of the platform.

There was a deeper architectural reason too. Most Kafka-compatible solutions are built on shared-nothing designs, where data is split across nodes that each own their slice. To keep that data durable and available, those designs lean on complex replication and consensus mechanisms, which introduce operational overhead, generate heavy network traffic between nodes, and create bottlenecks as data volumes grow. VAST set out to avoid that whole category of problem by starting from a different foundation.

The DASE foundation

That foundation is DASE, which stands for disaggregated, shared-everything. The idea is straightforward to state and powerful in practice: separate compute from storage, but connect every compute node to all of the storage over a high-speed, low-latency network, with no east-west traffic between nodes. There are no shards and no single owner of any piece of data, because every compute node can reach all of the flash storage directly.

DASE separates compute from storage and lets every compute node reach all storage over a fast fabric, with no east-west traffic.

A few consequences fall straight out of that design, and they map directly onto the legacy problems from Part 1:

• Independent scaling. Need more throughput? Add compute nodes, called CNodes. Need more capacity? Add storage. You stop over-buying one just to get more of the other.
• No rebalancing. Because storage is one shared pool rather than a set of owned shards, growing the system does not require re-partitioning or shuffling existing data across nodes.
• Built-in resilience. With no node owning data, durability and high availability do not depend on triplication or complex consensus. Losing a node is an availability event, not a data-loss event.
• Steady performance under mixed load. Heavy reads and bursts of writes hit a shared all-flash pool over a fast network instead of fighting over one node’s local disk.

DASE is what makes everything else possible. Once no single server owns the data and every node can reach all of it at flash speed, the walls between streaming, analytics, and AI stop being necessary, and the replication overhead that drives legacy cost and complexity largely disappears.

Topics become queryable tables

Here is the capability that most clearly separates the VAST Event Broker from a traditional broker. With VAST, Kafka topics are automatically stored as columns in VAST DataBase (VDB) tables. The events you stream in are not parked in an opaque log that later has to be extracted and reshaped. They land as structured, columnar table data, which means they are immediately available to run SQL queries against, the moment they arrive.

Kafka topics arrive through the standard Kafka API and are stored as columns in VAST DataBase tables, instantly queryable with SQL and no ETL.

Think about what that removes. In a legacy stack, making streaming data analyzable means building and running an ETL pipeline to pull events out of the broker, transform them, and load them into a separate analytical store, and only then can anyone query them. With the VAST Event Broker, that step is gone. Streaming data is stored once, as queryable tables, with the benefits of the VAST DataBase such as compression, deduplication, and encryption applied automatically. The same events are available for real-time querying and for historical analysis without a second copy.

This is what bridges the gap between real-time messaging and structured data storage. Live event data and historical data sit in the same place and the same form, so you can detect an anomaly as it happens, run a query that spans the last second and the last year together, and feed both to a model, all without waiting for a connector or a load job to finish.

Instant analytics in practice

To see why storing topics as tables matters, picture a concrete moment. A payment stream is flowing into a topic. In a legacy stack, spotting a fraud pattern means waiting for that stream to be extracted, transformed, and loaded into an analytical system, by which point the suspicious session is long over. With the VAST Event Broker, the same events are already columnar tables in the VAST DataBase, so an analyst or an automated job can run a SQL query against them as they land and flag the pattern while the transaction is still live.

The same property powers everyday analytics. A dashboard can show this minute’s activity next to the last year of history in a single query, because both live in the same tables in the same form. A data scientist can train or refresh a model on data that includes the last hour rather than last night. An operations team can watch a real-time view that is genuinely real time, not a snapshot from the last successful load. None of this needs a connector, a staging area, or a second copy. The data is queryable because of how it was stored, not because a pipeline went and prepared it.

Core capabilities, in plain terms

The unified design produces four capabilities that are worth stating directly, because each one answers a specific pain from Part 1:

• Streaming, analytics, and AI in one. A real-time event architecture is built into the AI Operating System, so you can run real-time event analytics and trigger or execute AI pipelines without moving data between separate systems.
• Instant analytics. Because topics are stored as queryable tables, you can run SQL on live event data the moment it arrives, with no ETL jobs and no connectors, across both real-time and historical data.
• Built-in reliability and simplicity. Fault tolerance and automatic data optimization are part of the platform, which removes the manual tuning and maintenance jobs that consume engineering time elsewhere.
• No pipeline complexity. Ingestion, storage, and querying happen in one unified platform, which means fewer moving parts, lower operational overhead, and far less time spent getting data ready to use.

How the data flows

It helps to follow the data through the platform from end to end, because the flow is simpler than the legacy version it replaces. There are four stages, and each one carries forward the DASE benefits described above.

Data flow with the VAST Event Broker: ingestion through the Kafka API, columnar storage in the VAST DataBase, instant processing, and DASE resilience.

• 1. Data ingestion. Producers send messages using standard Kafka APIs, so existing pipelines and tools work unchanged.
• 2. Storage. Messages are written as columns in VAST DataBase tables, benefiting automatically from compression, deduplication, and encryption.
• 3. Data processing. As soon as messages are stored, they are accessible to analytics engines and processing tools, which enables real-time insight without a load step.
• 4. Resilience. DASE ensures durability and high availability without traditional replication or complex consensus operations.

The whole path runs from ingestion to analytics inside one platform, which is what delivers speed, resilience, and operational simplicity at the same time. Nothing has to be exported, reshaped, or copied to move from one stage to the next, because every stage is reading and writing the same governed data on the same shared flash.

The metrics that matter

Before looking at numbers, it helps to agree on what to measure, because four metrics decide how a streaming platform performs and they mean slightly different things to different readers:

• Throughput is the volume of data a system can handle per unit of time, measured in megabytes or messages per second. High throughput is what lets a platform keep up with massive event volumes without falling behind.
• Latency is the time it takes for a message to travel from producer to consumer, usually measured in milliseconds. Low latency is what makes real-time use cases such as fraud detection or live monitoring actually real time.
• Scalability is the ability to maintain performance as data, users, and workload complexity grow. A scalable platform grows with you without sacrificing speed or reliability.
• Resilience is the ability to keep data durable and available through failures and heavy load, which is essential for anything mission critical.

Keeping these four in mind is what lets you read a benchmark honestly rather than being dazzled by a single figure. A platform that wins on throughput but collapses under mixed load, or that posts a low latency only at small scale, has not actually solved the problem. The VAST Event Broker is built to hold all four at once, which is what the following numbers reflect.

Performance and scalability

A platform claim is only as good as the numbers behind it, so it is worth being concrete. Legacy brokers like Kafka struggle to keep up as data volumes and workloads grow, which forces teams to add more brokers, manage complex partitioning, and constantly tune clusters just to hold performance steady. The VAST Event Broker takes a different path. Because DASE separates compute from storage, you scale each independently, and adding CNodes increases performance without bottlenecks or tuning. That is what true linear scalability means: add compute, get proportional throughput.

Linear scalability: adding CNodes increases throughput in proportion, with up to ten times Kafka performance and sub-four-millisecond latency.

The measured performance follows from that architecture. A single CNode can process on the order of one to two million events per second, more than ten times the rate of traditional Kafka deployments on the same hardware, with sub-four-millisecond latency and automatic optimization. Across head-to-head workload tests against Redpanda and Apache Kafka, the advantage holds consistently rather than only in a favorable corner case:

Workload / metric	VAST Event Broker	Redpanda	Apache Kafka
Mixed read/write throughput (per broker)	1.1 GB/s	666 MB/s	275 MB/s
Write-only burst (per node)	~17 million msgs/sec	n/a	n/a
Write-only sustained (per node)	4.75 million msgs/sec	n/a	n/a
Throughput per CNode	1 to 2 million events/sec	n/a	n/a
Latency	Sub-4 ms	n/a	n/a
Relative performance	Up to 10x Kafka	baseline+	baseline

In the mixed read/write scenario that most closely matches real production, the VAST Event Broker delivers 1.1 gigabytes per second per broker, compared with 666 megabytes per second for Redpanda and 275 megabytes per second for Apache Kafka. That is nearly double Redpanda and roughly four times Kafka. For write-heavy workloads such as clickstreams or IoT telemetry, it can absorb burst traffic of around seventeen million messages per second per node and sustain ingestion at 4.75 million messages per second, which means it can handle large spikes without data loss or lag.

What drives the performance edge

The performance gap is architectural, not accidental, and it comes down to a handful of design choices that work together:

• Direct-to-flash storage. DASE writes data directly to high-performance flash, which minimizes bottlenecks and maximizes throughput under heavy load.
• A tabular back end. Instead of relying on traditional log replication and sharding, VAST uses a tabular back end in the VAST DataBase that eliminates much of the replication overhead, which streamlines both storage and retrieval.
• Efficient ingestion and analytics together. Ingested data is immediately available for analytics and downstream processing, which reduces latency and accelerates time to insight.
• Reduced traffic and complexity. By minimizing east-west network traffic and consensus operations, VAST avoids the scaling and performance penalties built into shared-nothing designs.

Put together, these explain why the advantage is consistent across workloads. The platform was built from the ground up to handle high-throughput AI workloads, rather than retrofitted to support them, so the throughput is a property of the architecture rather than a result of careful tuning.

Cost and operational value

Performance is only part of the story. For most teams, the bigger day-to-day difference is operational, and it follows from a single principle: radical simplification. By unifying streaming, analytics, and AI on one platform, the VAST Event Broker eliminates the separate Kafka clusters, redundant tools, and complex pipelines that make legacy environments expensive to run. That reduces both infrastructure sprawl and the amount of specialized effort needed to operate real-time data at scale.

The operational payoff: store data once, manage one platform, deploy in minutes, run zero ETL pipelines, and let the platform optimize itself.

Five advantages stand out:

• No data triplication. Unlike Kafka, which keeps multiple copies of data for high availability, VAST stores event data once in a unified platform, which removes the cost and complexity of triplication while still providing reliability.
• Zero pipeline complexity. Ingestion, storage, and querying happen in one system, so there are no ETL jobs to build and no connector sprawl to maintain, which shortens the time from data arriving to insight being available.
• Unified management. Streaming, analytics, and AI are managed through one platform, with no separate Kafka clusters, no ZooKeeper, and no local broker storage to operate, which simplifies daily work and reduces the risk of outages from fragmented systems.
• Rapid deployment. The platform can be deployed in minutes rather than months, so teams can go from zero to production quickly and respond faster to new needs.
• Automatic optimization. Built-in reliability, fault tolerance, and automatic data optimization reduce the need for manual tuning and ongoing maintenance, which lowers operational cost and frees the team for higher-value work.

What this means for you

As in Part 1, the value lands differently depending on where you sit, so here is the same platform framed for four readers:

• If you run storage: you store each stream once instead of three times, capacity becomes one shared flash pool you grow without re-sharding, and resilience comes from the architecture rather than from replicated copies.
• If you run databases or the data platform: streaming data arrives as queryable tables, so the ETL pipeline that fed your analytics system disappears, and there is one governed copy of the data instead of several that drift apart.
• If you build AI or machine learning: live and historical event data sit together and are instantly queryable, so you can trigger AI pipelines and retrieve fresh context without exporting events into a separate feature store or vector database.
• If you own the business outcome: one platform replaces several, deployment takes minutes, and real-time analytics become the default, which means lower total cost of ownership (TCO) and faster reaction to what is happening now.

A scenario walkthrough

Imagine a fast-growing company that wants real-time analytics across customer activity and operations, plus an AI assistant grounded in current data. In the traditional approach this is at least four systems: a Kafka cluster for streaming, an analytical store for reporting, an ETL layer to move data between them, and a separate store for the AI, all glued together with pipelines and four sets of access rules. Events are copied repeatedly, the dashboard runs hours behind, and every new use case means another cluster or another copy.

On the VAST Event Broker, the same events are produced through the standard Kafka API and immediately stored as columns in VAST DataBase tables. The real-time dashboard runs SQL against that data as it lands. The AI pipeline retrieves fresh, governed context from the same tables, with no separate corpus to copy and secure. One platform, one copy of the data, two very different workloads served from the same foundation, and none of the waiting that the traditional design treats as unavoidable. The win is measured in what is no longer there: no ETL jobs, no triplicated storage, no second security model, no stale dashboard.

Key takeaways

• The VAST Event Broker is Kafka-compatible, so it works with existing Kafka client libraries, APIs, and tools and can act as a drop-in replacement without application rewrites.
• It stores Kafka topics as columns in VAST DataBase tables, so streaming data is instantly queryable with SQL, with no ETL and no second system.
• DASE separates compute from storage and shares all storage with all compute, which removes rebalancing and cluster sprawl and enables true linear scalability as you add CNodes.
• Benchmarks show up to ten times Kafka performance, including 1.1 GB/s per broker in mixed read/write versus 666 MB/s for Redpanda and 275 MB/s for Kafka, with sub-four-millisecond latency.
• The operational payoff is radical simplification: store data once, manage one platform, deploy in minutes, run zero ETL pipelines, and let the platform optimize itself.

Read the first post in this series: Why Event Streaming Is the Backbone of Real-Time AI, and Where Legacy Kafka Falls Short. For a deeper look at the storage engine underneath, see Inside the VAST DataBase Engine.

Related reading: see how the same platform turns streamed data into analytics and AI in From Lakehouse to AI on the VAST DataBase.

VAST Data · The VAST Event Broker · Real-Time Analytics and AI series

The VAST

VAST Event Broker series hub — full series index

Event Broker series

This article is part of a two-part series on real-time event streaming for AI. Continue reading:

• Part 1: Why Event Streaming Is the Backbone of Real-Time AI
• Part 2: Inside the VAST Event Broker: Real-Time Analytics and AI at Scale (you are here)

Related reading: see the platform underneath in Inside the VAST DataBase Engine, and how streamed data powers analytics and AI in From Lakehouse to AI on the VAST DataBase.