Package org.apache.iceberg.flink.sink.dynamic

Class DynamicIcebergSink

java.lang.Object
org.apache.iceberg.flink.sink.dynamic.DynamicIcebergSink
All Implemented Interfaces:
Serializable, org.apache.flink.api.connector.sink2.Sink<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal>, org.apache.flink.api.connector.sink2.SupportsCommitter<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>, org.apache.flink.streaming.api.connector.sink2.SupportsPostCommitTopology<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>, org.apache.flink.streaming.api.connector.sink2.SupportsPreCommitTopology<org.apache.iceberg.flink.sink.dynamic.DynamicWriteResult,org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>, org.apache.flink.streaming.api.connector.sink2.SupportsPreWriteTopology<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal>
@Experimental public class DynamicIcebergSink extends Object implements org.apache.flink.api.connector.sink2.Sink<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal>, org.apache.flink.streaming.api.connector.sink2.SupportsPreWriteTopology<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal>, org.apache.flink.api.connector.sink2.SupportsCommitter<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>, org.apache.flink.streaming.api.connector.sink2.SupportsPreCommitTopology<org.apache.iceberg.flink.sink.dynamic.DynamicWriteResult,org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>, org.apache.flink.streaming.api.connector.sink2.SupportsPostCommitTopology<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>
Dynamic version of the IcebergSink which supports:
  • Writing to any number of tables (No more 1:1 sink/topic relationship).
  • Creating and updating tables based on the user-supplied routing.
  • Updating the schema and partition spec of tables based on the user-supplied specification.
See Also:

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    DynamicIcebergSink.Builder<T>
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    addPostCommitTopology(org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>> committables)
     
    org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>>
    addPreCommitTopology(org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<org.apache.iceberg.flink.sink.dynamic.DynamicWriteResult>> writeResults)
     
    org.apache.flink.streaming.api.datastream.DataStream<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal>
    addPreWriteTopology(org.apache.flink.streaming.api.datastream.DataStream<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal> inputDataStream)
     
    org.apache.flink.api.connector.sink2.Committer<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>
    createCommitter(org.apache.flink.api.connector.sink2.CommitterInitContext context)
     
    org.apache.flink.api.connector.sink2.SinkWriter<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal>
    createWriter(org.apache.flink.api.connector.sink2.WriterInitContext context)
     
    static <T> DynamicIcebergSink.Builder<T>
    forInput(org.apache.flink.streaming.api.datastream.DataStream<T> input)
    Initialize a IcebergSink.Builder to export the data from input data stream with RowDatas into iceberg table.
    org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>
    getCommittableSerializer()
     
    org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.iceberg.flink.sink.dynamic.DynamicWriteResult>
    getWriteResultSerializer()
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • createWriter

      public org.apache.flink.api.connector.sink2.SinkWriter<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal> createWriter(org.apache.flink.api.connector.sink2.WriterInitContext context)
      Specified by:
      createWriter in interface org.apache.flink.api.connector.sink2.Sink<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal>

    • createCommitter

      public org.apache.flink.api.connector.sink2.Committer<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable> createCommitter(org.apache.flink.api.connector.sink2.CommitterInitContext context)
      Specified by:
      createCommitter in interface org.apache.flink.api.connector.sink2.SupportsCommitter<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>

    • getCommittableSerializer

      public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable> getCommittableSerializer()
      Specified by:
      getCommittableSerializer in interface org.apache.flink.api.connector.sink2.SupportsCommitter<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>

    • addPostCommitTopology

      public void addPostCommitTopology(org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>> committables)
      Specified by:
      addPostCommitTopology in interface org.apache.flink.streaming.api.connector.sink2.SupportsPostCommitTopology<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>

    • addPreWriteTopology

      public org.apache.flink.streaming.api.datastream.DataStream<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal> addPreWriteTopology(org.apache.flink.streaming.api.datastream.DataStream<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal> inputDataStream)
      Specified by:
      addPreWriteTopology in interface org.apache.flink.streaming.api.connector.sink2.SupportsPreWriteTopology<org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternal>

    • addPreCommitTopology

      public org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>> addPreCommitTopology(org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<org.apache.iceberg.flink.sink.dynamic.DynamicWriteResult>> writeResults)
      Specified by:
      addPreCommitTopology in interface org.apache.flink.streaming.api.connector.sink2.SupportsPreCommitTopology<org.apache.iceberg.flink.sink.dynamic.DynamicWriteResult,org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>

    • getWriteResultSerializer

      public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.iceberg.flink.sink.dynamic.DynamicWriteResult> getWriteResultSerializer()
      Specified by:
      getWriteResultSerializer in interface org.apache.flink.streaming.api.connector.sink2.SupportsPreCommitTopology<org.apache.iceberg.flink.sink.dynamic.DynamicWriteResult,org.apache.iceberg.flink.sink.dynamic.DynamicCommittable>

    • forInput

      public static <T> DynamicIcebergSink.Builder<T> forInput(org.apache.flink.streaming.api.datastream.DataStream<T> input)
      Initialize a IcebergSink.Builder to export the data from input data stream with RowDatas into iceberg table.
      Parameters:
      input - the source input data stream with RowDatas.
      Returns:
      IcebergSink.Builder to connect the iceberg table.