Package org.apache.iceberg

Class BaseRewriteManifests

java.lang.Object

org.apache.iceberg.BaseRewriteManifests

All Implemented Interfaces:

PendingUpdate<Snapshot>, RewriteManifests, SnapshotUpdate<RewriteManifests>

public class BaseRewriteManifests
extends java.lang.Object
implements RewriteManifests

Method Summary

Modifier and Type	Method	Description
RewriteManifests	addManifest(ManifestFile manifest)	Adds a manifest file to the table.
Snapshot	apply()	Apply the pending changes and return the uncommitted changes for validation.
java.util.List<ManifestFile>	apply(TableMetadata base, Snapshot snapshot)	Apply the update's changes to the given metadata and snapshot.
protected boolean	canInheritSnapshotId()
protected void	cleanAll()
protected void	cleanUncommitted(java.util.Set<ManifestFile> committed)	Clean up any uncommitted manifests that were created.
protected boolean	cleanupAfterCommit()
RewriteManifests	clusterBy(java.util.function.Function<DataFile,java.lang.Object> func)	Groups an existing DataFile by a cluster key produced by a function.
void	commit()	Apply the pending changes and commit.
protected CommitMetrics	commitMetrics()
protected TableMetadata	current()
protected void	deleteFile(java.lang.String path)
RewriteManifests	deleteManifest(ManifestFile manifest)	Deletes a manifest file from the table.
ThisT	deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback)	Set a callback to delete files instead of the table's default.
protected OutputFile	manifestListPath()
protected ManifestReader<DeleteFile>	newDeleteManifestReader(ManifestFile manifest)
protected ManifestWriter<DeleteFile>	newDeleteManifestWriter(PartitionSpec spec)
protected EncryptedOutputFile	newManifestOutputFile()
protected ManifestReader<DataFile>	newManifestReader(ManifestFile manifest)
protected ManifestWriter<DataFile>	newManifestWriter(PartitionSpec spec)
protected RollingManifestWriter<DeleteFile>	newRollingDeleteManifestWriter(PartitionSpec spec)
protected RollingManifestWriter<DataFile>	newRollingManifestWriter(PartitionSpec spec)
protected java.lang.String	operation()	A string that describes the action that produced the new snapshot.
protected TableMetadata	refresh()
protected ThisT	reportWith(MetricsReporter newReporter)
RewriteManifests	rewriteIf(java.util.function.Predicate<ManifestFile> pred)	Determines which existing ManifestFile for the table should be rewritten.
ThisT	scanManifestsWith(java.util.concurrent.ExecutorService executorService)	Use a particular executor to scan manifests.
protected RewriteManifests	self()
RewriteManifests	set(java.lang.String property, java.lang.String value)	Set a summary property in the snapshot produced by this update.
protected long	snapshotId()
ThisT	stageOnly()	Called to stage a snapshot in table metadata, but not update the current snapshot id.
protected java.util.Map<java.lang.String,java.lang.String>	summary()
protected java.lang.String	targetBranch()
protected void	targetBranch(java.lang.String branch)	A setter for the target branch on which snapshot producer operation should be performed
protected void	validate(TableMetadata currentMetadata, Snapshot snapshot)	Validate the current metadata.
protected java.util.concurrent.ExecutorService	workerPool()
protected java.util.List<ManifestFile>	writeDataManifests(java.util.Collection<DataFile> files, java.lang.Long dataSeq, PartitionSpec spec)
protected java.util.List<ManifestFile>	writeDataManifests(java.util.Collection<DataFile> files, PartitionSpec spec)
protected java.util.List<ManifestFile>	writeDeleteManifests(java.util.Collection<DeleteFile> files, PartitionSpec spec)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.PendingUpdate

apply, commit, updateEvent

Methods inherited from interface org.apache.iceberg.SnapshotUpdate

deleteWith, scanManifestsWith, stageOnly, toBranch

Method Detail

self

protected RewriteManifests self()

operation

protected java.lang.String operation()

A string that describes the action that produced the new snapshot.

Returns:

a string operation

set

public RewriteManifests set(java.lang.String property,
                            java.lang.String value)

Description copied from interface: SnapshotUpdate

Set a summary property in the snapshot produced by this update.

Specified by:

set in interface SnapshotUpdate<RewriteManifests>

Parameters:

property - a String property name

value - a String property value

Returns:

this for method chaining

summary

protected java.util.Map<java.lang.String,java.lang.String> summary()

clusterBy

public RewriteManifests clusterBy(java.util.function.Function<DataFile,java.lang.Object> func)

Description copied from interface: RewriteManifests

Groups an existing DataFile by a cluster key produced by a function. The cluster key will determine which data file will be associated with a particular manifest. All data files with the same cluster key will be written to the same manifest (unless the file is large and split into multiple files). Manifests deleted via RewriteManifests.deleteManifest(ManifestFile) or added via RewriteManifests.addManifest(ManifestFile) are ignored during the rewrite process.

Specified by:

clusterBy in interface RewriteManifests

Parameters:

func - Function used to cluster data files to manifests.

Returns:

this for method chaining

rewriteIf

public RewriteManifests rewriteIf(java.util.function.Predicate<ManifestFile> pred)

Description copied from interface: RewriteManifests

Determines which existing ManifestFile for the table should be rewritten. Manifests that do not match the predicate are kept as-is. If this is not called and no predicate is set, then all manifests will be rewritten.

Specified by:

rewriteIf in interface RewriteManifests

Parameters:

pred - Predicate used to determine which manifests to rewrite. If true then the manifest file will be included for rewrite. If false then the manifest is kept as-is.

Returns:

this for method chaining

deleteManifest

public RewriteManifests deleteManifest(ManifestFile manifest)

Description copied from interface: RewriteManifests

Deletes a manifest file from the table.

Specified by:

deleteManifest in interface RewriteManifests

Parameters:

manifest - a manifest to delete

Returns:

this for method chaining

addManifest

public RewriteManifests addManifest(ManifestFile manifest)

Description copied from interface: RewriteManifests

Adds a manifest file to the table. The added manifest cannot contain new or deleted files.

By default, the manifest will be rewritten to ensure all entries have explicit snapshot IDs. In that case, it is always the responsibility of the caller to manage the lifecycle of the original manifest.

If manifest entries are allowed to inherit the snapshot ID assigned on commit, the manifest should never be deleted manually if the commit succeeds as it will become part of the table metadata and will be cleaned up on expiry. If the manifest gets merged with others while preparing a new snapshot, it will be deleted automatically if this operation is successful. If the commit fails, the manifest will never be deleted and it is up to the caller whether to delete or reuse it.

Specified by:

addManifest in interface RewriteManifests

Parameters:

manifest - a manifest to add

Returns:

this for method chaining

apply

public java.util.List<ManifestFile> apply(TableMetadata base,
                                          Snapshot snapshot)

Apply the update's changes to the given metadata and snapshot. Return the new manifest list.

Parameters:

base - the base table metadata to apply changes to

snapshot - snapshot to apply the changes to

Returns:

a manifest list for the new snapshot.

cleanUncommitted

protected void cleanUncommitted(java.util.Set<ManifestFile> committed)

Clean up any uncommitted manifests that were created.

Manifests may not be committed if apply is called more because a commit conflict has occurred. Implementations may keep around manifests because the same changes will be made by both apply calls. This method instructs the implementation to clean up those manifests and passes the paths of the manifests that were actually committed.

Parameters:

committed - a set of manifest paths that were actually committed

stageOnly

public ThisT stageOnly()

Description copied from interface: SnapshotUpdate

Called to stage a snapshot in table metadata, but not update the current snapshot id.

Specified by:

stageOnly in interface SnapshotUpdate<ThisT>

Returns:

this for method chaining

scanManifestsWith

public ThisT scanManifestsWith(java.util.concurrent.ExecutorService executorService)

Description copied from interface: SnapshotUpdate

Use a particular executor to scan manifests. The default worker pool will be used by default.

Specified by:

scanManifestsWith in interface SnapshotUpdate<ThisT>

Parameters:

executorService - the provided executor

Returns:

this for method chaining

commitMetrics

protected CommitMetrics commitMetrics()

reportWith

protected ThisT reportWith(MetricsReporter newReporter)

targetBranch

protected void targetBranch(java.lang.String branch)

A setter for the target branch on which snapshot producer operation should be performed

Parameters:

branch - to set as target branch

targetBranch

protected java.lang.String targetBranch()

workerPool

protected java.util.concurrent.ExecutorService workerPool()

deleteWith

public ThisT deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback)

Description copied from interface: SnapshotUpdate

Set a callback to delete files instead of the table's default.

Specified by:

deleteWith in interface SnapshotUpdate<ThisT>

Parameters:

deleteCallback - a String consumer used to delete locations.

Returns:

this for method chaining

validate

protected void validate(TableMetadata currentMetadata,
                        Snapshot snapshot)

Validate the current metadata.

Child operations can override this to add custom validation.

Parameters:

currentMetadata - current table metadata to validate

snapshot - ending snapshot on the lineage which is being validated

apply

public Snapshot apply()

Description copied from interface: PendingUpdate

Apply the pending changes and return the uncommitted changes for validation.

This does not result in a permanent update.

Specified by:

apply in interface PendingUpdate<ThisT>

Returns:

the uncommitted changes that would be committed by calling PendingUpdate.commit()

current

protected TableMetadata current()

refresh

protected TableMetadata refresh()

commit

public void commit()

Description copied from interface: PendingUpdate

Apply the pending changes and commit.

Changes are committed by calling the underlying table's commit method.

Once the commit is successful, the updated table will be refreshed.

Specified by:

commit in interface PendingUpdate<ThisT>

cleanAll

protected void cleanAll()

deleteFile

protected void deleteFile(java.lang.String path)

manifestListPath

protected OutputFile manifestListPath()

newManifestOutputFile

protected EncryptedOutputFile newManifestOutputFile()

newManifestWriter

protected ManifestWriter<DataFile> newManifestWriter(PartitionSpec spec)

newDeleteManifestWriter

protected ManifestWriter<DeleteFile> newDeleteManifestWriter(PartitionSpec spec)

newRollingManifestWriter

protected RollingManifestWriter<DataFile> newRollingManifestWriter(PartitionSpec spec)

newRollingDeleteManifestWriter

protected RollingManifestWriter<DeleteFile> newRollingDeleteManifestWriter(PartitionSpec spec)

newManifestReader

protected ManifestReader<DataFile> newManifestReader(ManifestFile manifest)

newDeleteManifestReader

protected ManifestReader<DeleteFile> newDeleteManifestReader(ManifestFile manifest)

snapshotId

protected long snapshotId()

canInheritSnapshotId

protected boolean canInheritSnapshotId()

cleanupAfterCommit

protected boolean cleanupAfterCommit()

writeDataManifests

protected java.util.List<ManifestFile> writeDataManifests(java.util.Collection<DataFile> files,
                                                          PartitionSpec spec)

writeDataManifests

protected java.util.List<ManifestFile> writeDataManifests(java.util.Collection<DataFile> files,
                                                          java.lang.Long dataSeq,
                                                          PartitionSpec spec)

writeDeleteManifests

protected java.util.List<ManifestFile> writeDeleteManifests(java.util.Collection<DeleteFile> files,
                                                            PartitionSpec spec)