Blog Post Series: PostgreSQL CDC with Debezium on Amazon MSK Connect

 

Overview

This series provides a comprehensive guide to implementing Change Data Capture (CDC) from PostgreSQL to Kafka using Debezium on Amazon MSK Connect, including setup, verification, heartbeat implementation, and monitoring.

Blog Posts

1. Setting Up Debezium PostgreSQL Connector on Amazon MSK Connect

Link: https://www.dbaglobe.com/2026/03/setting-up-debezium-postgresql.html

Topics Covered:

• Architecture overview (Aurora PostgreSQL 17.5, MSK 3.7.x, Debezium 3.2.6)
• PostgreSQL preparation (logical replication, publications, replication user)
• Creating Debezium custom plugin for MSK Connect
• Worker configuration setup
• Kafka topic creation (manual for auto-create disabled)
• Connector creation and configuration
• Verification steps
• Common issues and solutions

Key Takeaways:

• Complete step-by-step setup process
• Configuration parameters explained
• Best practices for production deployment
• Troubleshooting initial setup issues

---

2. Verifying Debezium CDC: Testing Initial Snapshot and Change Data Capture

Link: https://www.dbaglobe.com/2026/03/verifying-debezium-cdc-testing-initial.html

Topics Covered:

• Understanding Debezium phases (snapshot vs streaming)
• Verifying initial snapshot completion
• Testing INSERT operations
• Testing UPDATE operations
• Testing DELETE operations
• Bulk operation testing
• Data consistency verification
• Performance verification
• Automated verification scripts

Key Takeaways:

• Comprehensive testing procedures
• Message structure analysis
• Data consistency checks
• Performance benchmarking
• Troubleshooting verification issues

---

3. Preventing PostgreSQL Replication Lag with Debezium Heartbeat

Link: https://www.dbaglobe.com/2026/03/preventing-postgresql-replication-lag.html

Topics Covered:

• Understanding replication lag problem
• Real-world impact demonstration (23 GB lag without heartbeat)
• Heartbeat Table approach (traditional)
• WAL Logical Messages approach (recommended)
• Comparison of both approaches
• Test results with actual data
• AWS DMS heartbeat alternative
• Troubleshooting heartbeat issues

Key Takeaways:

• Critical finding: Lag dropped from 23 GB to 344 bytes with heartbeat
• Two implementation approaches with pros/cons
• pg_logical_emit_message() recommended for production
• 30-second interval optimal for most workloads
• Prevents unbounded WAL growth

Test Results:

• Without heartbeat: 16 GB → 23 GB in 35 minutes
• With heartbeat table: 23 GB → 344 bytes
• With WAL messages: Maintained 137-191 MB under continuous heavy load

---

4. Monitoring and Troubleshooting Debezium PostgreSQL CDC on Amazon MSK Connect

Link: https://www.dbaglobe.com/2026/03/monitoring-and-troubleshooting-debezium.html

Topics Covered:

• Essential monitoring queries (replication slot, WAL, heartbeat)
• AWS monitoring (connector state, CloudWatch logs, Kafka topics)
• CloudWatch alarms setup
• Common issues and solutions:
  • Connector in FAILED state
  • High replication lag
  • Slow snapshots
  • Duplicate messages
  • Schema changes not captured
  • Connection exhaustion
  • WAL disk space full
• Automated health check scripts
• Monitoring dashboard creation
• Best practices for production

Key Takeaways:

• Comprehensive monitoring strategy
• Proactive alerting setup
• Systematic troubleshooting approach
• Automated health checks
• Production-ready monitoring dashboard

---

Technical Specifications

Environment

• Source Database: Amazon Aurora PostgreSQL 17.5 / RDS PostgreSQL
• Target: Amazon MSK (Kafka 3.7.x)
• CDC Tool: Debezium PostgreSQL Connector 3.2.6
• Deployment: MSK Connect (fully managed)
• Plugin: Confluent Hub distribution

Key Configuration Parameters

{
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
  "database.hostname": "your-rds-endpoint.rds.amazonaws.com",
  "database.port": "5432",
  "database.user": "postgres",
  "database.dbname": "your_database",
  "database.server.name": "rds_pg",
  "database.sslmode": "require",
  "table.include.list": "public.orders,public.customers",
  "plugin.name": "pgoutput",
  "slot.name": "debezium_slot",
  "publication.name": "debezium_publication",
  "topic.prefix": "cdc",
  "tasks.max": "1",
  "snapshot.mode": "initial",
  "heartbeat.interval.ms": "30000",
  "heartbeat.action.query": "SELECT pg_logical_emit_message(false, 'heartbeat', now()::varchar);"
}

Test Results Summary

Replication Lag Test

• Workload: 800,000 inserts/batch (400K audit_logs + 400K metrics)
• Replicated tables: orders, customers (zero writes)
• Duration: 1 hour

Phase 1: Without Heartbeat (35 minutes)

Initial lag: 16 GB
Final lag: 23 GB
Growth rate: ~200 MB/minute
Status: Replication slot frozen

Phase 2: With Heartbeat Table (25 minutes)

Configuration: heartbeat.interval.ms=30000
Result: 23 GB → 344 bytes
Status: Slot advancing every 30 seconds

Phase 3: With WAL Messages (Continuous)

Configuration: pg_logical_emit_message()
Result: Maintained 137-191 MB (vs 23+ GB without)
Status: Slot advancing regularly

Key Findings

1. Heartbeat Effectiveness

✅ Prevents unbounded lag growth
✅ Both approaches (table vs WAL messages) equally effective
✅ WAL messages cleaner for production
✅ 30-second interval optimal

2. Implementation Requirements

• Manual Kafka topic creation required (MSK auto-create disabled)
• Heartbeat table: cdc.public.debezium_heartbeat
• WAL messages: cdc.message
• Recommended: pg_logical_emit_message() approach

Target Audience

• Database Administrators: Managing PostgreSQL CDC pipelines
• Data Engineers: Building real-time data integration
• DevOps Engineers: Operating MSK Connect infrastructure
• Solution Architects: Designing CDC architectures

Prerequisites

Readers should have:

• Basic understanding of PostgreSQL
• Familiarity with Apache Kafka concepts
• AWS experience (RDS, MSK, CloudWatch)
• Command-line proficiency

Use Cases

These blog posts are ideal for:

• Setting up production CDC pipelines
• Migrating from self-managed Debezium to MSK Connect
• Troubleshooting replication lag issues
• Implementing monitoring and alerting
• Understanding heartbeat mechanisms

Additional Resources

Documentation

• Debezium PostgreSQL Connector
• Amazon MSK Connect
• PostgreSQL Logical Replication

Conclusion

This blog post series provides a complete, production-ready guide to implementing PostgreSQL CDC with Debezium on Amazon MSK Connect. Based on real testing and documented findings, it covers everything from initial setup through production monitoring, with special emphasis on the critical heartbeat mechanism that prevents replication lag.

The series is unique in providing:

• Real test data and results
• Both heartbeat approaches compared
• MSK Connect-specific considerations
• Production-ready monitoring solutions
• Comprehensive troubleshooting guides

These posts will help readers successfully implement and maintain reliable CDC pipelines in production environments.