Third‑Generation Stream Processing: Turning IoT Data into Real‑Time Action

Kelly Herrell of Hazelcast

In the pre‑digital era, IT teams leveraged a range of data‑management tools—data warehouses, analytical platforms, and various database systems—to capture and preserve information for future analysis.

Today, data is being generated and streamed by IoT devices at an unprecedented rate. The “Things” in IoT—sensors, mobile apps, connected vehicles—create an explosive volume of information. Add the network effect, where value scales with the number of connected users, and it’s clear why IDC projects the IoT market to reach $745 billion (€665 billion) next year and surpass $1 trillion (€0.89 trillion) by 2022.

In this megatrend, the traditional emphasis on historical data is giving way to the temporal value of streaming data. In the streaming paradigm, value is directly tied to immediacy for two reasons:

• Difference: Like water molecules passing through a hose, each snapshot of data flowing through the network is unique.
• Perishability: The opportunity to act on insights often evaporates shortly after data is generated.

These concepts mean that sudden changes detected in streams demand instant action—whether it’s a real‑time facial‑recognition alert or a drilling‑rig sensor detecting abnormal vibrations that could lead to catastrophic failure if not addressed immediately.

In today’s fast‑moving world, IoT and streaming data accelerate the pace of change, reshaping how we process information. Stream processing itself is evolving rapidly.

Two generations, same problems

The first generation of stream processing relied heavily on batch processing within complex Hadoop‑based architectures. Data would be collected, stored, and only after a delay would it be streamed into the processing engine. The combination of complexity and latency made this approach largely inadequate.

The second generation introduced micro‑batches, shrinking batch sizes to reduce processing time. However, the underlying complexity remained, and the delay in setting up each micro‑batch still meant that changes were identified only after they had become historical.

Third‑generation stream processing

The challenge for IT organizations is clear: how can we ingest and process streaming data in real time, turning raw data into actionable insight “now” without the overhead of batch‑oriented systems? The answer lies in software that is lightweight, not batch‑centric, and small enough to run close to—or even embedded in—the data source.

First‑ and second‑generation engines require multiple components, resulting in a footprint too large for most edge and IoT environments. A lightweight engine can be deployed right at the source, eliminating the need to send the entire stream across the network for processing. This proximity dramatically reduces latency and addresses the perishability challenge.

Third‑generation stream processing can handle live, raw data at scale, delivering insights instantly and enabling actions that capitalize on the fleeting value of real‑time information.

Streaming in practice

A drilling rig is a quintessential symbol of the energy sector, yet its operating costs are high and any downtime can severely impact the bottom line. Preventive insights can dramatically reduce losses.

SigmaStream specializes in high‑frequency data streams generated during drilling. Their rigs are outfitted with dozens of sensors that capture minute vibrations. The volume can reach 60 to 70 high‑frequency channels entering the stream‑processing system.

By processing this data in real time, SigmaStream enables operators to act instantly—adjusting drilling parameters on the fly—to prevent failures and downtime. Coupled with a third‑generation streaming engine and the right analytics tools, operators can monitor almost imperceptible vibrations. The result: customers have saved millions of dollars and reduced on‑site time by up to 20 %.

In the digital age, latency is the new downtime. Stream processing is the logical next step for organizations that need to ingest information faster, enable quicker actions, and meet the growing demand for real‑time data. By mainstreaming stream processing, enterprises can thrive in a world dominated by ultra‑high‑performance applications and deliver time‑sensitive information that meets rising expectations.

The author is Kelly Herrell, CEO of Hazelcast