Big Data Management and Applications

Problem it solves

When a request mentions real-time dashboards, data-lake migration, or Spark tuning but lacks data volume, latency, and team stack, the answer can collapse into a tool list. This skill first constrains the inputs: business context, focus area, stack preference, and whether the question is theoretical, implementation, or design.

How it works and where it stops

• Collection and storage: It compares Kafka, Flume, DataX, HDFS, Ceph, Iceberg, Hudi, and Delta Lake for ingestion and persistence choices.
• Processing and query: It maps Spark, Flink, Kafka Streams, Airflow, ClickHouse, Doris, Trino, and Druid to batch, stream, scheduling, and OLAP concerns.
• Governance and security: It brings in Atlas, DataHub, Great Expectations, and Ranger for catalog, lineage, quality, and access control.

It expects actionable output: trade-offs, tool comparisons, key parameters, text architecture diagrams, and code examples when needed. Boundaries: it is closer to engineering consulting and solution design than production operations, cost accounting, or benchmarking. For emerging components or private implementations, verify against official docs and community cases.