Skip to content

blog🔗

  • in blog
  • 5 min read

Semi-Structured Data in Apache Iceberg: Meet the Variant Type

Semi-structured data, such as JSON-like documents whose fields differ from row to row, has always been a poor fit for table formats built around fixed schemas. Iceberg v3 adds the Variant type for exactly this data: a single column can hold values of arbitrary, evolving shape, stored in a compact binary form that engines read and write consistently.

This is the first post in a series on Variant in Apache Iceberg. It covers what Variant is, why it exists, and how it fits into an Iceberg table. The next post covers shredding, the technique that stores frequently accessed Variant fields as typed, columnar data. Variant is stored in Parquet, Avro, and ORC; shredding is currently available only in Parquet.

  • in blog
  • 9 min read

Accelerating Apache Spark Queries (and Iceberg Rust Development) with Apache DataFusion Comet

Apache Iceberg's ecosystem spans multiple query engines and language implementations that work together to give users a consistent experience across the data lakehouse. This post explores one integration within that ecosystem, Iceberg Rust and Apache DataFusion Comet, and the two benefits their relationship brings. Comet accelerates Apache Spark's reads over Iceberg tables by running them natively through Iceberg Rust. That same integration turns Iceberg Java's nearly 10,000 Spark tests into a differential-testing harness whose benefits run both ways: Iceberg Rust gets exercised against a broad corpus of real-world scenarios, and the comparison has even caught bugs in Iceberg Java. The resulting fixes land upstream and benefit every project built on these libraries, not just Comet, as the Iceberg and DataFusion communities build on each other's strengths.