Real-Time Data Synchronization: CDC, Conflict Resolution, and Sync Patterns [2026]

A price comparison platform showing a price that changed six hours ago loses user trust immediately. A property listing site displaying sold inventory as available damages both reputation and conversion. An inventory management system out of step with a POS system by even a few minutes causes order failures. Data freshness is not a technical nicety — it is a direct business metric. Real-time data synchronization is the engineering practice that keeps distributed data stores aligned.

This guide focuses specifically on synchronization patterns: keeping data consistent across multiple systems, databases, and services. The coverage includes Change Data Capture implementations, polling and webhook strategies, bidirectional sync with conflict resolution, event sourcing for complete data history, and the operational systems needed to monitor sync health at scale. By the end, you will have concrete patterns for each synchronization scenario your architecture might encounter.

Real-Time Data Synchronization: The Core Problem

Modern applications decompose data across multiple stores. A microservices architecture might have an orders service writing to PostgreSQL, a fulfillment service reading from MongoDB, a customer service reading from a Redis cache, and an analytics warehouse reading from BigQuery — all needing to reflect the same underlying state.

Synchronization connects these systems. Done poorly, you get stale data, duplicate records, or conflicting states. Done well, every system reads the correct current state within an acceptable lag window.

The fundamental challenge: distributed systems cannot have instantaneous synchronization. The CAP theorem guarantees that under network partition, you choose consistency or availability. In practice, this means accepting eventual consistency — all systems converge to the same state, but not simultaneously.

Change Data Capture Patterns

Change Data Capture (CDC) intercepts changes at the source database and propagates them to downstream systems. CDC is fundamentally more efficient than full-table polling: only changed records are processed, regardless of total table size.

Log-Based CDC

Log-based CDC reads the database transaction log directly to capture changes. PostgreSQL writes all changes to its Write-Ahead Log (WAL). MySQL records changes in the binary log (binlog). MongoDB maintains an operations log (oplog). These logs record every INSERT, UPDATE, and DELETE in the exact sequence they occurred.

Debezium is the dominant open-source log-based CDC platform. Running as a Kafka Connect connector, Debezium reads the WAL and publishes change events to Kafka topics — one topic per source table. Each event includes the before and after states of the row, the operation type, and the transaction timestamp.

Advantages over other CDC methods:

• Near-zero load on the source database (reads sequential log data, not executing queries against tables)
• Captures all change types including deletes — polling-based approaches miss deleted rows without soft-delete columns
• Sub-second propagation latency in most deployments
• Schema change detection — Debezium tracks DDL changes alongside DML changes

Operational requirements: the source database must have the appropriate log level enabled (PostgreSQL requires wal_level=logical; MySQL requires binlog_format=ROW). Log retention must exceed the maximum time a Debezium connector might be offline to prevent gaps.

Query-Based Incremental CDC

For databases or APIs without accessible transaction logs, incremental queries using timestamp-based high-watermarks provide change detection. The pattern:

1. Store the timestamp of the last successfully processed record
2. On each cycle, query for records where updated_at > last_processed_timestamp
3. Process the result set, update the high-watermark on success

This approach works for any queryable system — relational databases, REST APIs, external data sources. Limitations include inability to detect hard deletes (rows removed without an is_deleted flag), potential for missed records at the boundary when clock skew exists between systems, and query load on the source system proportional to the polling frequency.

Use timestamp-based CDC when: the source is an external API with no CDC protocol, the source database cannot enable logical replication, or the volume of changes is low enough that polling is acceptable.

Trigger-Based CDC

Database triggers fire on INSERT, UPDATE, and DELETE operations and write change records to a dedicated audit table. This approach captures every change immediately without log access. The cost: every write operation on the source table executes additional trigger logic, adding latency to write transactions and increasing source database load.

For write-heavy tables, trigger-based CDC is typically too expensive operationally. Reserve it for compliance scenarios where a complete audit trail at the source is required, or for low-write-frequency reference tables where the operational overhead is acceptable.

Polling and Webhook Synchronization

Not all synchronization scenarios involve database-level capture. Many integrations work with APIs that do not expose CDC protocols.

Polling Strategy Design

Polling queries a source system periodically to detect changes. The design question is frequency: too frequent wastes resources; too infrequent misses the freshness target.

Adaptive polling adjusts frequency based on observed change rate. A catalog page that changes daily does not need 5-minute polling. A price feed that updates every 30 seconds does. Track each source's historical change rate and assign polling intervals accordingly:

• High-change sources (price feeds, inventory levels): 1–5 minute intervals
• Medium-change sources (product attributes, customer profiles): 15–60 minute intervals
• Low-change sources (reference data, category structures): Daily or on-demand

Exponential backoff reduces polling frequency when no changes are detected, increasing it when changes resume. This concentrates API calls during active periods and reduces waste during quiet periods.

Jitter adds randomized delay to polling schedules. Without jitter, many consumers polling the same source simultaneously create synchronized load spikes — the "thundering herd" problem. A ±20% random delay distributes load naturally.

Webhook Integration

Webhooks invert the polling model: the source system sends a notification when data changes. The downstream service receives a push rather than polling for pulls.

Building a reliable webhook receiver requires:

Authentication: Verify webhook origin with HMAC signatures. The source signs the payload with a shared secret; the receiver verifies the signature before processing. This prevents unauthorized or spoofed webhook injection.

Idempotency: Webhook providers deliver at-least-once. The same event may arrive multiple times due to network retries or provider-side failures. The receiver must process duplicate events safely — either by using event IDs as deduplication keys or by making the processing operations idempotent.

Async acknowledgment: Accept the webhook request immediately (return 200), then process asynchronously. If the receiver returns slowly or errors, the provider retries — creating duplicate deliveries. Immediate acknowledgment with queue-based processing is the correct pattern.

Dead letter handling: Events that fail all processing attempts go to a dead letter queue for manual review rather than being silently dropped.

Webhooks alone are insufficient for reliable synchronization. Webhook delivery can fail, and many APIs impose delivery retry limits. Combine webhooks for low-latency updates with periodic polling for consistency validation.

Bidirectional Sync and Conflict Resolution

When multiple systems can independently modify the same data, conflicts occur. Two users edit the same record simultaneously; the same product is updated in two regional databases before the sync completes. Conflict resolution strategy determines which value wins.

Last Write Wins (LWW)

The simplest strategy: the most recently timestamped value wins. LWW is easy to implement and adequate for non-critical data. The weakness: clock drift between systems can cause the "wrong" write to win. A write at 14:05:00 on a system with a 10-second clock offset appears older than a write at 14:04:58 on another system, even though it was causally later.

Mitigate clock issues with NTP synchronization and, for critical data, use logical clocks (Lamport timestamps, vector clocks) that track causal ordering independent of wall clock time.

Source Priority Ranking

Assign each data source a trust rank for each field. When conflicts occur, the higher-ranked source wins:

• Price: official vendor API takes precedence over scraped data
• Inventory: warehouse system takes precedence over sales system estimate
• Customer address: CRM takes precedence over shipping system

This model works well for asymmetric systems where one source is authoritative. Implement as a merge policy evaluated per field, not per record — a single record's fields may be authoritative from different sources.

Field-Level Merge

In some scenarios, different sources are authoritative for different fields of the same record. A customer record might have the most accurate email in the CRM, the most accurate mailing address in the fulfillment system, and the most accurate phone number from a verification service. Field-level merge combines the authoritative value from each source:

def merge_customer(crm_record, fulfillment_record, verification_record):
    return {
        "email": crm_record["email"],          # CRM is authoritative
        "address": fulfillment_record["address"],  # Fulfillment is authoritative
        "phone": verification_record["phone"],    # Verification service is authoritative
    }

Maintain an explicit field-level source policy document that teams reference when adding new fields. Without this, field authority becomes implicit knowledge that creates bugs during system changes.

Operational Transforms

For collaborative editing scenarios (document editors, shared configuration), operational transforms reconcile concurrent edits algebraically. Each change is represented as an operation (insert at position X, delete characters Y–Z) rather than a final value. Conflicts are resolved by transforming operations relative to each other, preserving intent from both edits.

OT is complex to implement correctly. For most data synchronization use cases, simpler conflict strategies are more appropriate — save OT for genuine collaborative editing requirements.

Data Versioning and Event Sourcing

Tracking how data arrived at its current state enables debugging, compliance, and recovery from incorrect transformations.

Bi-Temporal Data Modeling

Bi-temporal models store two time dimensions for every record:

• Valid time: when the fact was true in the real world (the transaction occurred at 14:05)
• Transaction time: when the fact was recorded in the system (entered at 14:12 due to processing delay)

This enables "as-of" queries: "What was the customer's address as of March 15th, as it was known at that time?" — even if the address was corrected retroactively in the system. Price history, contract terms, and regulatory reporting all benefit from bi-temporal modeling.

PostgreSQL 16+ supports temporal tables natively. Snowflake and BigQuery support bi-temporal patterns through dedicated columns and query patterns.

Event Sourcing

Event sourcing stores every state change as an immutable event rather than overwriting the current state. The current record state is derived by replaying events in sequence.

Benefits for data synchronization:

• Complete audit trail of every change, who made it, and when
• Replay capability: reprocess events with corrected logic without re-extracting from sources
• Temporal queries: reconstruct the state at any past point by replaying events up to that time
• Downstream consumers can build their own projections from the same event stream

In Smart Maple's aggregator projects, we use event sourcing for price change tracking. Every price update is an event with the source, timestamp, and confidence score. The current price is the most recent accepted event, but historical analysis can trace exactly when prices changed and correlate changes with external market events.

Snapshot and Compaction

As events accumulate, replaying from the beginning becomes expensive. Snapshots capture the current state at regular intervals, enabling replay from the snapshot rather than the beginning of time. Kafka's log compaction retains only the most recent value per key, providing an efficient snapshot of current state without a separate snapshot mechanism.

Balance snapshot frequency against storage cost and replay time. For most applications, daily snapshots with event retention for 30 days provides adequate recovery options without excessive storage.

Staleness Management

Knowing when data is stale is as important as keeping data fresh. A system that silently serves six-month-old data is worse than one that clearly indicates data age.

Freshness Scoring

Assign a freshness score to each record based on time since last update and the record's expected update frequency:

def freshness_score(last_updated, expected_interval_hours):
    age_hours = (datetime.now() - last_updated).total_seconds() / 3600
    return math.exp(-age_hours / expected_interval_hours)

A score near 1.0 indicates fresh data; near 0.0 indicates stale data. Records with scores below defined thresholds trigger re-sync attempts or user-facing staleness indicators.

Staleness Actions

Define escalating responses to data age:

Staleness Level	Action
Slightly stale (1–2x expected interval)	Queue priority re-sync
Moderately stale (2–5x expected interval)	Show "data may be outdated" indicator
Severely stale (5–10x expected interval)	Hide record or show age warning prominently
Expired (10x+ expected interval)	Archive or remove from active display

User-facing freshness indicators ("Price updated 4 hours ago") build trust even when perfect real-time synchronization is not achievable for every data source.

Operational Monitoring

Sync Lag Metrics

Track end-to-end synchronization lag per source: the time between when a change occurs at the source and when it is reflected in the destination. Alert thresholds should match business requirements:

• Payment transactions: alert if lag exceeds 30 seconds
• Inventory levels: alert if lag exceeds 5 minutes
• Product descriptions: alert if lag exceeds 1 hour

Dead Letter Queue Growth

Monitor DLQ message count and growth rate. A spike in DLQ messages indicates a systematic processing failure — often a schema change at the source, an authentication expiry, or a downstream system outage. DLQs that grow silently for weeks indicate a monitoring gap.

Reconciliation Jobs

Periodic reconciliation compares source and destination states independently of the sync pipeline. Find records that exist in the source but not the destination, records with mismatched values, and records that should have been deleted but were not. Run reconciliation daily for critical data paths; treat reconciliation failures as pipeline incidents.

Conclusion

Real-time data synchronization is not a single technology — it is a collection of patterns applied to the specific relationship between each pair of systems in your architecture. Log-based CDC (Debezium + Kafka) is the right choice when source databases can be configured for logical replication and sub-second lag is required. Query-based incremental sync handles external APIs and legacy systems that cannot expose transaction logs. Webhook integration provides push-based updates for cloud services that support them. Conflict resolution strategy must be chosen before the first bidirectional sync is deployed — retrofitting it after data divergence has occurred is significantly more expensive.

Design your synchronization architecture around the staleness tolerance of each data type. Not everything needs sub-second synchronization. Price feeds and inventory levels may; product descriptions and category hierarchies typically do not. Matching synchronization frequency to actual business requirements avoids over-engineering while ensuring the data your users see is fresh enough to be actionable.