Real-Time Data Processing: Stream Processing Architecture with Kafka and Flink [2026]

Fraud detection systems must make decisions within milliseconds of a transaction occurring — not after a nightly batch run. Personalization engines need to update recommendations while the user is still browsing. IoT anomaly detection requires alerting before a machine fails, not after. These scenarios share a common requirement: real-time data processing, where data is acted upon as it arrives rather than accumulated for later batch analysis.

This guide covers the full stream processing stack: the architectural case for real-time over batch, core concepts (events, topics, partitions, windowing), the leading frameworks (Kafka, Kafka Streams, Apache Flink, Spark Structured Streaming), and the operational realities of running streaming pipelines in production. By the end, you will be able to evaluate whether a use case genuinely needs streaming and design an architecture to meet its requirements.

Real-Time Data Processing vs Batch: The Fundamental Tradeoff

The choice between batch and streaming is not about technical preference — it is about latency tolerance and what decisions your system makes with data.

Batch Processing Model

Batch systems accumulate data over a time window (hourly, daily, or weekly), then process all accumulated records in a single job. MapReduce, traditional ETL pipelines, and Spark in batch mode follow this model. Batch processing is simpler to implement, easier to retry on failure, and highly efficient for large-volume processing — a single Spark job can aggregate terabytes of data with predictable resource usage.

The limitation is latency. If your batch job runs at 2:00 AM, your dashboard reflects yesterday's reality. For operational decisions — fraud prevention, real-time pricing, live customer support — stale data is not useful.

Streaming Processing Model

Streaming systems process events as they are produced. Each event is handled individually or in micro-batches of milliseconds to seconds. Latency drops from hours to milliseconds, enabling decisions at the moment of action.

The cost: streaming systems are more complex to implement correctly. State management, handling out-of-order events, ensuring exactly-once processing, and fault recovery all require explicit architectural decisions that batch systems handle implicitly through simple reruns.

When Streaming Is Justified

Real-time data processing adds value only when low latency changes outcomes:

• Fraud detection: A transaction flagged after it completes has already transferred funds. Flagging at transaction time blocks the attempt.
• Real-time personalization: A recommendation updated while the user is still on the page drives conversion. A recommendation updated 6 hours later reaches a user who has long since left.
• IoT anomaly detection: A temperature threshold exceeded on a manufacturing machine requires immediate shutdown. A batch job finds it the next morning after damage occurred.
• Live dashboards: Operational teams monitoring system health need metrics in the last 60 seconds, not the last 24 hours.
• Algorithmic trading: Market conditions change in milliseconds. Decisions made on price data that is even one minute stale are based on incorrect assumptions.

For financial reporting, HR analytics, customer segmentation, and most compliance reporting, batch is appropriate and cheaper to operate.

Core Stream Processing Concepts

Events, Topics, and Partitions

The unit of streaming is an event — an immutable record of something that happened. Events carry a key, a value, a timestamp, and optional metadata headers. In Apache Kafka, events are organized into topics — named, ordered, durable logs. Each topic is divided into partitions that can live on different brokers, enabling parallel processing.

Consumers read from partitions independently. The number of partitions determines maximum parallelism: a topic with 24 partitions can be consumed by at most 24 consumer instances simultaneously. Increasing partition count scales throughput — but only if downstream processing can parallelize accordingly.

Kafka retains messages for a configurable period (days to weeks, or indefinitely with compaction enabled). This retention property is critical: it allows consumers to reprocess historical events from any point, replay pipelines after fixing bugs, and recover from consumer failures without data loss.

Producers and Consumers

Producers write events to topics. Consumer groups read from topics, with each group maintaining its own offset (position) in each partition. Multiple consumer groups can independently consume the same topic without interfering — a single transaction stream can feed a fraud detection consumer, an analytics consumer, and a customer notification consumer simultaneously.

Consumer offset management is the primary reliability mechanism. Committing an offset marks all messages through that point as processed. If a consumer fails before committing, it re-reads uncommitted messages on restart — ensuring at-least-once delivery. Combining offset management with idempotent processing logic achieves exactly-once behavior at the application level.

Event Time and Processing Time

Every event has two timestamps: when it occurred (event time) and when the system processes it (processing time). Network delays, consumer lag, and backpressure cause these to diverge. A sensor reading generated at 14:05:30 may arrive at the processing cluster at 14:05:45, then be processed at 14:05:47.

Windowing based on event time produces correct results — a 5-minute aggregation window captures events that occurred in that 5-minute period, regardless of when they arrived. Windowing based on processing time produces results that depend on network and system conditions, making them difficult to reproduce.

Event-time processing requires a watermark mechanism: the system tracks how far behind the slowest event source is and uses this lag estimate to determine when a time window is "complete enough" to compute results.

Windowing Strategies in Real-Time Data Processing

Windows convert an infinite stream into finite chunks for aggregation.

Tumbling Windows

Non-overlapping, fixed-size time windows. Each event belongs to exactly one window. A 5-minute tumbling window produces one result every 5 minutes. Use tumbling windows for periodic reporting: transactions per minute, errors per hour, signups per day.

Sliding Windows

Overlapping windows that advance by a smaller interval than their size. A 10-minute window with 1-minute slide produces one result every minute, each covering the last 10 minutes. Use sliding windows for moving averages and trend detection. Each event belongs to multiple windows, which increases compute cost proportionally to the overlap ratio.

Session Windows

Dynamically sized windows that close after a configurable gap of inactivity. If a user makes no events for 30 minutes, the session window closes. Session windows model user behavior naturally — a session is not a fixed time period, it is a burst of activity. Use session windows for user journey analysis, engagement metrics, and clickstream analytics.

Global Windows

A single window covering all events of a given key. Useful for running totals and all-time aggregations. Requires a trigger mechanism to emit intermediate results without closing the window.

Kafka Streams for Stateful Stream Processing

Kafka Streams is a Java/Scala library that runs stream processing logic inside your application — no separate cluster required. It reads from Kafka topics, processes events, and writes results back to Kafka topics.

KStream and KTable Abstractions

KStream represents an event log: each record is a new, independent event. KTable represents a changelog: each record updates the current value for a key, and only the latest value per key is retained. The stream-table duality maps directly to how data actually works — order events are a stream; customer profile state is a table.

Joining a KStream of purchase events to a KTable of customer profiles enriches each event with the customer's current loyalty tier, geographic region, and risk score — without external database calls.

State Stores

Stateful processing in Kafka Streams uses RocksDB-backed local state stores. Aggregations (sum, count, average over a window), join lookup tables, and deduplication state all live in these stores. State is changelog-backed to Kafka, enabling recovery after node failures by replaying the changelog to rebuild local state.

Interactive Queries expose state store data via a queryable API, letting other services read aggregated state without a separate database.

Exactly-Once Semantics

Kafka Streams supports exactly-once processing via Kafka's transactional API. Setting processing.guarantee=exactly_once_v2 wraps the read-process-write cycle in a distributed transaction. Each event is processed exactly once, even if the consumer restarts mid-batch. This guarantee comes with throughput overhead — typically 10–30% lower than at-least-once for the same workload.

Choose exactly-once for financial calculations and user-visible counters. Use at-least-once with idempotent sinks for metrics pipelines and analytics where occasional duplicate events are acceptable.

Apache Flink for Complex Stream Processing

Apache Flink is a distributed stream processing framework designed for high-throughput, low-latency processing with sophisticated state management. Unlike Kafka Streams (a library embedded in your application), Flink runs as a standalone cluster and manages parallelism across many machines.

DataStream API

Flink's DataStream API provides fine-grained control over event processing. Operators — map, flatMap, filter, keyBy, window, reduce, aggregate — chain together to form a processing pipeline. The process() function exposes event time, state, and timers directly, enabling custom processing logic that more declarative APIs cannot express.

State Management

Flink supports multiple state types per operator: ValueState (single value per key), ListState (list per key), MapState (map per key), and aggregating states. State backend choice affects scale: HashMapStateBackend holds all state in JVM heap (fastest, limited by memory); EmbeddedRocksDBStateBackend stores state on local disk (slower, supports terabytes of state per node).

Checkpointing and Fault Tolerance

Flink's checkpointing mechanism periodically snapshots the full state of all operators to durable storage (typically S3 or HDFS). On failure, Flink restarts all operators from the last successful checkpoint and re-reads unprocessed events from Kafka. The time to recover is proportional to checkpoint frequency and state size.

Savepoints are user-triggered checkpoints that enable controlled restarts: upgrading application code, scaling parallelism, or migrating to new hardware — all without data loss or reprocessing.

Complex Event Processing (CEP)

Flink's CEP library detects patterns across event sequences. Define pattern rules: "event A, then event B within 5 minutes, then event C" — and Flink emits matches when the full pattern occurs in the event stream.

Use cases:

• Fraud detection: "Same card used at two different international locations within 30 minutes"
• Equipment monitoring: "Temperature above threshold, then pressure spike, then vibration anomaly"
• User behavior: "Item added to cart, checkout started, but payment not completed within 10 minutes"

CEP eliminates the need for complex stateful aggregation logic in application code — the pattern engine handles sequencing, timeouts, and partial matches.

Spark Structured Streaming

For teams already running Spark for batch processing, Spark Structured Streaming provides stream processing using the same DataFrame and SQL APIs. Micro-batch mode processes events in 100ms–1 second windows. Continuous mode achieves sub-millisecond latency for select operations but has limited operator support.

Spark Structured Streaming integrates naturally with Spark ML pipelines, Delta Lake, and existing Spark infrastructure. If your team already has Spark skills and infrastructure, the incremental cost of adding streaming is low. For net-new streaming infrastructure without existing Spark investment, Flink's event time handling and state management are more mature.

Operational Patterns for Streaming Pipelines

Consumer Lag Monitoring

Consumer lag — the gap between the latest message produced and the latest message consumed — is the primary operational signal for streaming pipelines. Rising lag means consumers cannot keep up with producers. Alert when lag exceeds a threshold that would cause unacceptable end-to-end latency.

Monitor lag per topic-partition, not just per topic. Lag concentrated in one partition points to a hot partition or a partition-specific processing bottleneck.

Backpressure Handling

Backpressure occurs when downstream operators cannot process events as fast as upstream operators produce them. In Flink, backpressure propagates upstream automatically, slowing producers to prevent queue overflow. In Kafka Streams, backpressure is implicit through consumer lag.

Design pipelines with explicit capacity planning: measure throughput at each stage, identify bottlenecks, and add parallelism before deploying to production. A pipeline that handles 10,000 events per second in testing may saturate at 3,000 events per second under production conditions due to state serialization, I/O, or network overhead.

Dead Letter Queues

Events that fail processing — deserializtion errors, schema mismatches, unexpected null values — should route to a dead letter queue (DLQ) rather than blocking the pipeline or being silently dropped. Monitor DLQ growth rate; a sudden spike indicates a source-side schema change or data quality issue.

Kubernetes Deployment

Flink runs natively on Kubernetes via the Flink Kubernetes Operator, managing job submission, scaling, and checkpoint lifecycle through standard Kubernetes resources. Kafka Streams applications deploy as regular Kubernetes Deployments — stateful sets with persistent volume claims for RocksDB state stores, ensuring state survives pod restarts.

Auto-scaling based on consumer lag metrics (available via KEDA's Kafka trigger) adjusts consumer replicas as load fluctuates, reducing cost during off-peak periods and maintaining latency SLAs during spikes.

Architecture Decision Guide

Requirement	Recommended Approach
Simple stream-to-stream transforms	Kafka Streams
Complex event patterns (CEP)	Apache Flink
Existing Spark investment	Spark Structured Streaming
Exactly-once with minimal ops overhead	Kafka Streams + exactly_once_v2
Sub-millisecond latency	Flink with async I/O
Very large state (100GB+)	Flink + RocksDB backend
ML model serving in stream	Flink async I/O or ONNX integration

Conclusion

Real-time data processing is not universally better than batch — it is appropriate when the latency difference changes outcomes. Before investing in streaming infrastructure, confirm that the decision being made is time-sensitive at the millisecond-to-second scale. If it is, Kafka provides the durable, partitioned event backbone that scales from thousands to millions of events per second. Kafka Streams handles the majority of stateful streaming use cases with minimal operational overhead. Apache Flink serves complex event processing, very large state, and multi-stream join scenarios that Kafka Streams cannot address.

Start with a simple consumer group reading from a Kafka topic before adopting a full stream processing framework. Many real-time data processing requirements can be met with a consumer group, a few operators, and a target database — without the operational weight of a Flink cluster.