org.apache.iceberg.actions

Class Spark3MigrateAction

java.lang.Object

org.apache.iceberg.actions.Spark3MigrateAction

All Implemented Interfaces:

Action<java.lang.Long>, CreateAction

public class Spark3MigrateAction
extends java.lang.Object

Takes a Spark table in the sourceCatalog and attempts to transform it into an Iceberg Table in the same location with the same identifier. Once complete the identifier which previously referred to a non-iceberg table will refer to the newly migrated iceberg table.

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.util.List<java.lang.String>	EXCLUDED_PROPERTIES
protected static java.lang.String	ICEBERG_METADATA_FOLDER
protected static java.lang.String	LOCATION

Constructor Summary

Constructors

Constructor and Description
Spark3MigrateAction(org.apache.spark.sql.SparkSession spark, org.apache.spark.sql.connector.catalog.CatalogPlugin sourceCatalog, org.apache.spark.sql.connector.catalog.Identifier sourceTableName)

Method Summary

Modifier and Type	Method and Description
protected java.util.Map<java.lang.String,java.lang.String>	additionalProperties()
protected org.apache.spark.sql.connector.catalog.TableCatalog	checkSourceCatalog(org.apache.spark.sql.connector.catalog.CatalogPlugin catalog)
protected org.apache.spark.sql.connector.catalog.StagingTableCatalog	destCatalog()
protected org.apache.spark.sql.connector.catalog.Identifier	destTableIdent()
protected void	ensureNameMappingPresent(Table table)
java.lang.Long	execute() Executes this action.
protected java.lang.String	getMetadataLocation(Table table)
protected org.apache.spark.sql.connector.catalog.TableCatalog	sourceCatalog()
protected org.apache.spark.sql.connector.catalog.Identifier	sourceTableIdent()
protected java.lang.String	sourceTableLocation()
protected org.apache.spark.sql.SparkSession	spark()
protected StagedSparkTable	stageDestTable()
protected java.util.Map<java.lang.String,java.lang.String>	targetTableProps()
protected org.apache.spark.sql.catalyst.catalog.CatalogTable	v1SourceTable()
CreateAction	withProperties(java.util.Map<java.lang.String,java.lang.String> properties) Adds additional properties to the newly created Iceberg table.
CreateAction	withProperty(java.lang.String key, java.lang.String value) Adds an additional property to the newly created Iceberg table.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOCATION

protected static final java.lang.String LOCATION

See Also:

Constant Field Values

ICEBERG_METADATA_FOLDER

protected static final java.lang.String ICEBERG_METADATA_FOLDER

See Also:

Constant Field Values

EXCLUDED_PROPERTIES

protected static final java.util.List<java.lang.String> EXCLUDED_PROPERTIES

Constructor Detail

Spark3MigrateAction

public Spark3MigrateAction(org.apache.spark.sql.SparkSession spark,
                           org.apache.spark.sql.connector.catalog.CatalogPlugin sourceCatalog,
                           org.apache.spark.sql.connector.catalog.Identifier sourceTableName)

Method Detail

execute

public java.lang.Long execute()

Description copied from interface: Action

Executes this action.

Returns:

the result of this action

targetTableProps

protected java.util.Map<java.lang.String,java.lang.String> targetTableProps()

checkSourceCatalog

protected org.apache.spark.sql.connector.catalog.TableCatalog checkSourceCatalog(org.apache.spark.sql.connector.catalog.CatalogPlugin catalog)

withProperties

public CreateAction withProperties(java.util.Map<java.lang.String,java.lang.String> properties)

Description copied from interface: CreateAction

Adds additional properties to the newly created Iceberg table. Any properties with the same key name will be overwritten.

Specified by:

withProperties in interface CreateAction

Parameters:

properties - a map of properties to be included

Returns:

this for chaining

withProperty

public CreateAction withProperty(java.lang.String key,
                                 java.lang.String value)

Description copied from interface: CreateAction

Adds an additional property to the newly created Iceberg table. Any properties with the same key name will be overwritten.

Specified by:

withProperty in interface CreateAction

Parameters:

key - the key of the property to add

value - the value of the property to add

Returns:

this for chaining

spark

protected org.apache.spark.sql.SparkSession spark()

sourceTableLocation

protected java.lang.String sourceTableLocation()

v1SourceTable

protected org.apache.spark.sql.catalyst.catalog.CatalogTable v1SourceTable()

sourceCatalog

protected org.apache.spark.sql.connector.catalog.TableCatalog sourceCatalog()

sourceTableIdent

protected org.apache.spark.sql.connector.catalog.Identifier sourceTableIdent()

destCatalog

protected org.apache.spark.sql.connector.catalog.StagingTableCatalog destCatalog()

destTableIdent

protected org.apache.spark.sql.connector.catalog.Identifier destTableIdent()

additionalProperties

protected java.util.Map<java.lang.String,java.lang.String> additionalProperties()

stageDestTable

protected StagedSparkTable stageDestTable()

ensureNameMappingPresent

protected void ensureNameMappingPresent(Table table)

getMetadataLocation

protected java.lang.String getMetadataLocation(Table table)