Flink Real-Time Data Warehouse Guide

Real-time warehouse projects often struggle less with individual tools and more with the connections between them: how MySQL changes reach the lake, how ODS through ADS are layered, whether stream and batch jobs can share one logic, and which production parameters matter. This skill organizes those questions into a repeatable engineering path.

Problem

• Ingestion: Move incremental MySQL changes into lake tables instead of only showing static ETL examples.
• Modeling: Define responsibilities, storage formats, and update cadences for ODS, DWD, DWS, and ADS.
• Stream-batch unification: Use Flink Materialized Tables to reduce the cost of maintaining separate real-time and batch SQL.
• Tuning: Provide a parameter checklist and common pitfalls before production deployment.

How it works

The skill uses Docker Compose to connect MySQL -> Kafka -> Flink -> Paimon -> StarRocks. The core path is to subscribe to MySQL binlog with Flink CDC, write changes into a Paimon ODS table, then build DWD and DWS using primary-key or aggregation tables. ADS targets query and business reporting and can be served by StarRocks or Doris. The package includes Flink SQL templates, CDC configuration, layering rules, materialized table syntax, tuning guidance, and interview QA. Materialized Tables require Flink 1.20+ and can switch between real-time refresh and batch backfill via FRESHNESS.

Boundaries

Local Paimon is best for development and testing; production should usually use HDFS or S3. Confirm that MySQL uses binlog_format=ROW and that the host has enough CPU, memory, disk, and Docker readiness.