Package org.apache.iceberg.flink.maintenance.api

Class RewriteDataFiles.Builder

java.lang.Object

org.apache.iceberg.flink.maintenance.api.MaintenanceTaskBuilder<RewriteDataFiles.Builder>

org.apache.iceberg.flink.maintenance.api.RewriteDataFiles.Builder

Enclosing class:

RewriteDataFiles

public static class RewriteDataFiles.Builder
extends MaintenanceTaskBuilder<RewriteDataFiles.Builder>

Constructor Summary

Constructors

Constructor	Description
Builder()

Method Summary

Modifier and Type	Method	Description
RewriteDataFiles.Builder	config(RewriteDataFilesConfig rewriteDataFilesConfig)	Configures the properties for the rewriter.
RewriteDataFiles.Builder	deleteFileThreshold(int deleteFileThreshold)	Configures the minimum delete file number for a file after a rewrite is always initiated.
RewriteDataFiles.Builder	filter(Expression newFilter)	A user provided filter for determining which files will be considered by the rewrite strategy.
RewriteDataFiles.Builder	maxFileGroupSizeBytes(long maxFileGroupSizeBytes)	Configures the group size for rewriting.
RewriteDataFiles.Builder	maxFileSizeBytes(long maxFileSizeBytes)	Configures the max file size considered for rewriting.
RewriteDataFiles.Builder	maxFilesToRewrite(int maxFilesToRewrite)	Configures max files to rewrite.
RewriteDataFiles.Builder	maxRewriteBytes(long newMaxRewriteBytes)	Configures the maximum byte size of the rewrites for one scheduled compaction.
RewriteDataFiles.Builder	minFileSizeBytes(long minFileSizeBytes)	Configures the min file size considered for rewriting.
RewriteDataFiles.Builder	minInputFiles(int minInputFiles)	Configures the minimum file number after a rewrite is always initiated.
RewriteDataFiles.Builder	partialProgressEnabled(boolean newPartialProgressEnabled)	Allows committing compacted data files in batches.
RewriteDataFiles.Builder	partialProgressMaxCommits(int newPartialProgressMaxCommits)	Configures the size of batches if partialProgressEnabled.
RewriteDataFiles.Builder	rewriteAll(boolean rewriteAll)	Overrides other options and forces rewriting of all provided files.
RewriteDataFiles.Builder	targetFileSizeBytes(long targetFileSizeBytes)	Configures the target file size.

Methods inherited from class org.apache.iceberg.flink.maintenance.api.MaintenanceTaskBuilder

index, operatorName, parallelism, parallelism, scheduleOnCommitCount, scheduleOnDataFileCount, scheduleOnDataFileSize, scheduleOnEqDeleteFileCount, scheduleOnEqDeleteRecordCount, scheduleOnInterval, scheduleOnPosDeleteFileCount, scheduleOnPosDeleteRecordCount, slotSharingGroup, slotSharingGroup, tableLoader, tableName, taskName, uidSuffix, uidSuffix

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Builder

public Builder()

Method Details

partialProgressEnabled

public RewriteDataFiles.Builder partialProgressEnabled(boolean newPartialProgressEnabled)

Allows committing compacted data files in batches. See RewriteDataFiles.PARTIAL_PROGRESS_ENABLED for more details.

Parameters:

newPartialProgressEnabled - to enable partial commits

partialProgressMaxCommits

public RewriteDataFiles.Builder partialProgressMaxCommits(int newPartialProgressMaxCommits)

Configures the size of batches if partialProgressEnabled. See RewriteDataFiles.PARTIAL_PROGRESS_MAX_COMMITS for more details.

Parameters:

newPartialProgressMaxCommits - to target number of the commits per run

maxRewriteBytes

public RewriteDataFiles.Builder maxRewriteBytes(long newMaxRewriteBytes)

Configures the maximum byte size of the rewrites for one scheduled compaction. This could be used to limit the resources used by the compaction.

Parameters:

newMaxRewriteBytes - to limit the size of the rewrites

targetFileSizeBytes

public RewriteDataFiles.Builder targetFileSizeBytes(long targetFileSizeBytes)

Configures the target file size. See RewriteDataFiles.TARGET_FILE_SIZE_BYTES for more details.

Parameters:

targetFileSizeBytes - target file size

minFileSizeBytes

public RewriteDataFiles.Builder minFileSizeBytes(long minFileSizeBytes)

Configures the min file size considered for rewriting. See SizeBasedFileRewritePlanner.MIN_FILE_SIZE_BYTES for more details.

Parameters:

minFileSizeBytes - min file size

maxFileSizeBytes

public RewriteDataFiles.Builder maxFileSizeBytes(long maxFileSizeBytes)

Configures the max file size considered for rewriting. See SizeBasedFileRewritePlanner.MAX_FILE_SIZE_BYTES for more details.

Parameters:

maxFileSizeBytes - max file size

minInputFiles

public RewriteDataFiles.Builder minInputFiles(int minInputFiles)

Configures the minimum file number after a rewrite is always initiated. See description see SizeBasedFileRewritePlanner.MIN_INPUT_FILES for more details.

Parameters:

minInputFiles - min file number

deleteFileThreshold

public RewriteDataFiles.Builder deleteFileThreshold(int deleteFileThreshold)

Configures the minimum delete file number for a file after a rewrite is always initiated. See BinPackRewriteFilePlanner.DELETE_FILE_THRESHOLD for more details.

Parameters:

deleteFileThreshold - min delete file number

rewriteAll

public RewriteDataFiles.Builder rewriteAll(boolean rewriteAll)

Overrides other options and forces rewriting of all provided files.

Parameters:

rewriteAll - enables a full rewrite

maxFileGroupSizeBytes

public RewriteDataFiles.Builder maxFileGroupSizeBytes(long maxFileGroupSizeBytes)

Configures the group size for rewriting. See SizeBasedFileRewritePlanner.MAX_FILE_GROUP_SIZE_BYTES for more details.

Parameters:

maxFileGroupSizeBytes - file group size for rewrite

maxFilesToRewrite

public RewriteDataFiles.Builder maxFilesToRewrite(int maxFilesToRewrite)

Configures max files to rewrite. See BinPackRewriteFilePlanner.MAX_FILES_TO_REWRITE for more details.

Parameters:

maxFilesToRewrite - maximum files to rewrite

filter

public RewriteDataFiles.Builder filter(Expression newFilter)

A user provided filter for determining which files will be considered by the rewrite strategy.

Parameters:

newFilter - the filter expression to apply

Returns:

this for method chaining

config

public RewriteDataFiles.Builder config(RewriteDataFilesConfig rewriteDataFilesConfig)

Configures the properties for the rewriter.

Parameters:

rewriteDataFilesConfig - properties for the rewriter