Why Event Streaming Is the Backbone of Real-Time AI, and Where Legacy Kafka Falls Short (Part 1 of 2)

Part 1 of 2: How the move from batch to real time changed the rules, and why the platforms most teams rely on were built for a different era.

Table of contents

• Executive summary
• What event streaming actually is
• What real-time streaming makes possible
• The shift from batch to real time
• Event-driven architecture, briefly
• Why legacy platforms struggle
• The cost of a short memory
• Why this is a structural problem, not a tuning problem
• A day in the life of one event
• Signs the streaming tax is hurting you
• What this means for you
• Where this series is going
• Key takeaways
• The VAST VAST Event Broker series hub — full series indexEvent Broker series

The way data moves through an organization has changed, and most of the infrastructure underneath it has not caught up. Not long ago the standard approach to data was straightforward: collect it, store it, and analyze it when you had time. That worked well enough when decisions could wait until the next morning or the next quarter. The rise of AI has changed the terms of competition, because modern AI systems do not analyze data after the fact. They react to it the moment it arrives.

This first post is about that shift and the strain it puts on older infrastructure. Before we look at how the VAST Event Broker solves the problem (the subject of Part 2), it helps to be honest about why real-time event streaming has become the foundation of every serious AI and analytics effort, and why the platforms most organizations rely on to carry that streaming data, especially Apache Kafka, were designed for a world that no longer exists.

Executive summary

If you read nothing else, read this. AI systems do not wait. They trigger inference, retrieval, automation, and training the moment data lands, which makes event streaming the backbone of modern AI rather than a nice-to-have for a few specialist teams. Event streaming is simply the continuous flow of data generated by your systems, applications, and devices, captured and processed as it happens instead of in a nightly batch.

The trouble is that legacy streaming platforms like Kafka were built for batch-era thinking. As data volumes grow and AI raises the bar for freshness, those platforms create five recurring problems: operational complexity, scalability bottlenecks, delayed analytics, high costs from data replication, and siloed architectures. None of these are anyone’s fault. They are the built-in consequences of an older design. This post explains the shift to real time and walks through each of those five problems. Part 2 explains the engine that removes them.

What event streaming actually is

Let us define the term plainly, because it gets used loosely. Event streaming is the continuous flow of data generated across an organization by systems, applications, and devices, captured and made available the instant each event occurs. An event can be a click, a purchase, a sensor reading, a log line, a database change, or a message from another application. Instead of waiting for a scheduled job to gather yesterday’s events into a pile, a streaming platform carries each event forward as it happens.

That continuous flow is exactly what modern AI pipelines need. A real-time AI pipeline depends on four things happening without pause:

• Continuous ingest: new events arrive and are captured the moment they are created, at any volume.
• Continuous inference: models score and react to those events as they land, not hours later.
• Continuous enrichment: each event is joined with context and history so it means something.
• Continuous retrieval: fresh data is immediately available to be searched and fed back into applications and models.

A continuous event stream feeding a real-time AI pipeline: ingest, inference, enrichment, and retrieval, all without pause.

This is why streaming sits underneath so many of the applications people now expect to work instantly. Retrieval-augmented generation (RAG) pipelines that ground an assistant in current data, recommendation systems that react to what you just did, telemetry and anomaly detection that flag a problem while it is happening, video intelligence, and agentic systems that take actions on their own all rest on a steady supply of fresh events. When the stream slows down or the data is stale, every one of those applications quietly gets worse.

What real-time streaming makes possible

It is worth slowing down on those use cases, because they are not edge cases. They are the workloads that increasingly define a competitive business, and each one depends on the stream being both fast and immediately usable:

• RAG and AI assistants. A model is only as good as the context you can give it. Retrieval-augmented generation grounds an assistant in your own current data, which means the freshness of the stream directly shapes the quality and trustworthiness of every answer.
• Recommendations. Suggestions that react to what a customer is doing right now convert far better than ones based on what they did last week. The value lives in the seconds while the person is still engaged.
• Telemetry and anomaly detection. Catching a failure, a fraud pattern, or a security event while it is unfolding is worth far more than discovering it in tomorrow’s report, when the damage is already done.
• Video intelligence. Analyzing frames and signals as they stream in enables real-time monitoring and response rather than after-the-fact review.
• Agentic systems. Software agents that take actions on their own need a continuous, reliable feed of events to reason over, because an agent acting on stale data acts on a world that no longer exists.

What every one of these shares is a refusal to wait. They do not want a copy of the data tomorrow. They want the event now, in a form they can use without a detour through a pipeline. That is the bar real-time streaming has to clear, and it is exactly the bar that batch-era infrastructure was never built to meet.

The shift from batch to real time

To see why this matters, it helps to contrast the two ways of working. In the batch model, data is collected over a period, stored, and then analyzed later in a scheduled run. The classic example is the nightly job that loads the day’s sales into a reporting system so the numbers are ready in the morning. Batch is simple and predictable, and for decades it was good enough because the decisions it supported could wait.

Batch collects, stores, and analyzes later. Real time reacts the moment an event arrives, which is what AI now demands.

In the real-time model, the system reacts the moment data arrives. Inference runs, retrieval fires, automation kicks off, and training updates, all triggered by events as they happen. The difference is not just speed for its own sake. It is the gap between catching a fraudulent transaction while the session is live and discovering it the next day, between recommending the right product while the customer is still browsing and emailing them after they have left, between a model that reasons over what is happening now and one that reasons over last week.

AI is what pushed this from a preference to a requirement. A model that reacts to events as they occur needs an infrastructure that can deliver those events continuously, reliably, and at scale. That puts enormous pressure on the layer that carries the stream, and for most organizations that layer was built around assumptions from the batch era. The result is a widening gap between what AI demands and what legacy streaming platforms can deliver.

Event-driven architecture, briefly

A couple of terms come up constantly in this space, and they are worth defining plainly so the rest makes sense regardless of your background. An event-driven architecture is a way of designing systems so they capture and react to events as those events happen, rather than polling for changes or waiting for a scheduled batch. An event bus, or message broker, is the pipe that carries those events between the systems that produce them and the systems that consume them. Kafka is the best-known example of that pipe.

In a healthy event-driven setup, a producer publishes an event once, and any number of consumers can react to it independently: one updates a dashboard, another feeds a model, another triggers an automation. That decoupling is powerful, and it is why event streaming became the connective tissue of modern data platforms. The catch is that the bus in the middle becomes critical infrastructure. If it is hard to scale, expensive to replicate, or slow to make data queryable, every consumer downstream inherits those limits. The quality of the whole real-time estate is capped by the platform carrying the events.

Why legacy platforms struggle

Apache Kafka became the de facto standard for streaming data, and for good reason. It has a rich ecosystem, broad adoption, and a familiar set of tools. But Kafka and platforms like it were designed for a different era, one organized around batch thinking rather than real-time reaction. As workloads grow and AI raises the stakes, the operational complexity, cost, and bottlenecks they introduce start to slow teams down rather than speed them up.

It is worth being specific about where the strain shows up, because the symptoms are easy to mistake for bad luck or under-staffing when they are really structural. Five problems come up again and again.

1. Operational complexity

Legacy platforms often require separate infrastructure for streaming and for analytics, which leads to cluster sprawl and a growing management burden. Every additional cluster is another set of moving parts, another set of failure points, and another thing that has to be patched, tuned, and monitored. Teams spend significant time tuning, maintaining, and troubleshooting, and that time comes straight out of the budget for building new things. The more the platform grows, the more of the team’s week it consumes.

Legacy streaming at scale: many broker nodes, tangled partitions and replication, constant tuning, and a separate analytics cluster off to the side.

2. Scalability bottlenecks

Scaling a traditional event broker means managing the tight relationship between brokers, partitions, replication, and storage. As workloads grow, teams have to rebalance data, tune partitioning strategies, and add infrastructure just to keep throughput and latency predictable. Because compute and storage are bound together, you often end up adding whole servers, and capacity you did not need, simply to get more processing power. The outcome is a set of bottlenecks that limit performance at exactly the moment growth demands more of it.

3. Delayed analytics

This is the problem that hits analytics and AI teams hardest. Slow, inefficient extract, transform, and load (ETL) pipelines, combined with row-based data structures, make it hard to correlate events and generate insight in real time. On top of that, storage constraints and log-retention policies force organizations to keep only small windows of event data, so the history needed for trend analysis and model training is often gone. The net effect is delayed analytics and missed opportunities for immediate action, even though the events themselves arrived on time.

The gap between an event arriving and an insight being available: a slow ETL conveyor, row-based tables, and a dashboard that updates late.

4. High costs from data replication

To provide high availability and fault tolerance, legacy systems typically keep multiple copies of the data, often three. This data triplication adds cost and complexity that grow directly with data volume. You are paying for three times the storage to protect one set of events, and the expense compounds as the stream gets busier. What looks like a reasonable reliability setting on day one becomes a major and ever-rising line item as the business scales.

Data triplication: one event stored three times for fault tolerance, with cost and wasted capacity that climb as volume grows.

5. Siloed architectures

Finally, organizations end up using separate tools for streaming, for analytics, for AI, and for storage, which creates siloed data architectures. Each silo keeps its own copy of the data, enforces its own access rules, and speaks its own dialect. That fragmentation increases infrastructure complexity and makes it genuinely hard to derive unified insight, because answering a single question means stitching together data that has been deliberately scattered. Silos slow down innovation and make it harder to respond quickly when the business needs to change direction.

Separate, disconnected silos for streaming, analytics, AI, and the warehouse, each with its own copy of the data and its own rules.

▶  The five problems share one root cause. Legacy streaming platforms treat the broker, the analytics system, and the AI store as separate worlds, so the same events have to be copied, replicated, and moved before they can be used. Every copy adds cost, delay, and another place for the data to fall out of sync.

The cost of a short memory

One consequence of the legacy design deserves a closer look, because it quietly limits what AI can do. To control storage cost and because keeping data in the broker is expensive, organizations set log-retention policies that hold only a small window of recent event data, often days or weeks. Older events are dropped. In a batch-reporting world that was tolerable, since most reports looked at the recent past anyway.

For AI and advanced analytics it is a serious handicap. Models learn from history, and the more representative that history is, the better they generalize. Trend analysis, seasonality, and rare-event detection all need a long memory. When the platform throws away everything older than a few weeks, the data needed to train a good model or spot a slow-moving pattern is simply gone, and no pipeline can recover what was never kept. Teams are then forced into yet another copy: exporting events into a separate, cheaper store just to retain them, which reintroduces all the movement and drift this series is about.

The deeper point is that retention should not be a painful trade-off between cost and capability. A platform where keeping event data is affordable and where that data stays directly queryable removes the choice entirely. You keep the history because you can, and it remains usable rather than frozen in an archive. Part 2 shows how storing streams as compressed, columnar tables makes exactly that practical.

Why this is a structural problem, not a tuning problem

The natural response to these symptoms is to try harder: add brokers, tune partitions, optimize the ETL jobs, buy more storage. Teams pour real skill into this, and it helps at the margins. But it does not fix the underlying issue, because the issue is not configuration. It is the shape of the architecture. A platform that keeps streaming, analytics, and AI in separate systems will always need to copy data between them, and no amount of tuning removes a copy that the design requires.

This is why the problem gets worse precisely as you succeed. More data, more use cases, and more users all multiply the number of copies and the work of keeping them in sync. The very growth that should be a sign of a healthy data culture turns into a rising operational burden. When the cost of asking a new question or onboarding a new stream scales up with success rather than down, that is the signature of a structural limit, not a temporary one.

The way out is not a faster version of the same stack. It is an architecture that does not require the copies in the first place, so that streaming data is usable for analytics and AI the instant it arrives. That is the shift Part 2 describes, and the rest of this post is simply the case for why it is worth making.

A day in the life of one event

To make the tax concrete, follow a single event through a typical stacked architecture. A customer clicks buy on a website. That event lands in a Kafka topic, where it is replicated three times for safety. To analyze it, a pipeline later extracts it, reshapes it, and loads it into a separate analytical system, which is where a dashboard finally sees it the next morning. To use it for AI, yet another job copies it again into a feature store or a vector database. One simple event has now been stored four or five times across three or four systems, each with its own cost and its own access rules.

Nothing in that chain is a mistake. Each system is doing the job it was built for. The failure is in the shape of the architecture: the event has to keep moving and keep being copied before anyone can act on it, and every hop adds latency, expense, and risk. By the time the data is queryable and the model can use it, the moment when acting on it would have mattered most has often passed. This is the pattern that real-time AI cannot tolerate, and it is the pattern Part 2 is about breaking.

Signs the streaming tax is hurting you

You do not need a formal architecture review to know whether this problem is yours. A handful of everyday symptoms give it away, and most organizations recognize several at once:

• Your streaming platform needs near-constant attention: rebalancing partitions, tuning clusters, and firefighting just to hold throughput and latency steady.
• The freshest data your analytics or AI can actually use is hours old, because it still has to pass through an ETL pipeline before it is queryable.
• You retain only a short window of event history, so questions that span months, and the model training that depends on them, are simply out of reach.
• Your storage bill keeps climbing because every stream is replicated several times for fault tolerance.
• Adding a new use case means standing up another cluster or another copy of the data, rather than pointing a new consumer at data you already have.

None of these are exotic failures. They are the ordinary background noise of a batch-era platform being asked to do a real-time job. The reason to name them is to make an invisible cost visible. If three or more sound like your week, the issue is not your team and it is not your tools. It is the shape of the architecture they are working inside.

What this means for you

The same problem looks different depending on where you sit, so here is the short version for four readers who will all recognize it:

• If you run storage: you are the one provisioning capacity for triplicated streams and for every separate cluster, and keeping copies of the same events in several systems. The tax shows up as replicated data and stranded capacity.
• If you run databases or the data platform: you are the one maintaining the ETL jobs that move events into the analytics system, and explaining why the dashboard is hours behind the stream.
• If you build AI or machine learning: you are the one waiting on data that is already stale, then copying it again into a feature store or vector database before you can train or retrieve against it.
• If you own the business outcome: you are the one paying for a stack of separate tools, waiting on insight that arrives late, and watching real-time opportunities close before the data is ready.

Different seats, same root cause. All four are paying, in their own currency, for event data that has to move and be copied before it can be used.

Where this series is going

This post named the shift and the problem: AI runs on real-time event streaming, and legacy platforms built for the batch era create operational complexity, scalability bottlenecks, delayed analytics, replication cost, and silos. Part 2 turns to the solution. It explains what the VAST Event Broker is, how it stays compatible with Kafka so it can drop into your existing pipelines, and how its architecture stores streaming data as queryable tables so analytics and AI can use it the instant it arrives, with up to ten times the performance of Kafka and none of the cluster sprawl.

▶  If your streaming platform is eating your team’s time, your analytics are always a step behind the stream, or your AI projects keep stalling behind another data copy, the real need is not a bigger cluster. It is to stop moving the data.

Key takeaways

• AI systems react to data the moment it arrives, which makes real-time event streaming the backbone of modern AI and analytics, not an optional extra.
• Event streaming is the continuous flow of events from your systems, applications, and devices, feeding continuous ingest, inference, enrichment, and retrieval.
• Legacy platforms like Kafka were built for batch-era thinking and struggle as volumes grow and AI raises the bar for freshness.
• Five problems recur: operational complexity, scalability bottlenecks, delayed analytics, high costs from data replication, and siloed architectures.
• They share one root cause: the same events must be copied and moved between separate systems before they can be used, which Part 2 sets out to fix.

Continue reading: Inside the VAST Event Broker: Real-Time Analytics and AI at Scale, the companion post that shows how VAST removes these problems at the architecture level.

Related reading: the VAST Event Broker stores streams as queryable tables inside the VAST DataBase. For the storage engine underneath, see Inside the VAST DataBase Engine.

VAST Data · The VAST Event Broker · Real-Time Analytics and AI series

The VAST

VAST Event Broker series hub — full series index

Event Broker series

This article is part of a two-part series on real-time event streaming for AI. Continue reading:

• Part 1: Why Event Streaming Is the Backbone of Real-Time AI (you are here)
• Part 2: Inside the VAST Event Broker: Real-Time Analytics and AI at Scale

Related reading: see the platform underneath in Inside the VAST DataBase Engine, and how streamed data powers analytics and AI in From Lakehouse to AI on the VAST DataBase.