Package org.apache.iceberg

Class CatalogUtil

java.lang.Object

org.apache.iceberg.CatalogUtil

public class CatalogUtil
extends java.lang.Object

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	ICEBERG_CATALOG_GLUE
static java.lang.String	ICEBERG_CATALOG_HADOOP
static java.lang.String	ICEBERG_CATALOG_HIVE
static java.lang.String	ICEBERG_CATALOG_JDBC
static java.lang.String	ICEBERG_CATALOG_NESSIE
static java.lang.String	ICEBERG_CATALOG_REST
static java.lang.String	ICEBERG_CATALOG_TYPE	Shortcut catalog property to load a catalog implementation through a short type name, instead of specifying a full java class through CatalogProperties.CATALOG_IMPL.
static java.lang.String	ICEBERG_CATALOG_TYPE_GLUE
static java.lang.String	ICEBERG_CATALOG_TYPE_HADOOP
static java.lang.String	ICEBERG_CATALOG_TYPE_HIVE
static java.lang.String	ICEBERG_CATALOG_TYPE_JDBC
static java.lang.String	ICEBERG_CATALOG_TYPE_NESSIE
static java.lang.String	ICEBERG_CATALOG_TYPE_REST

Method Summary

Modifier and Type	Method	Description
static Catalog	buildIcebergCatalog(java.lang.String name, java.util.Map<java.lang.String,java.lang.String> options, java.lang.Object conf)	Build an Iceberg Catalog based on a map of catalog properties and optional Hadoop configuration.
static void	configureHadoopConf(java.lang.Object maybeConfigurable, java.lang.Object conf)	Dynamically detects whether an object is a Hadoop Configurable and calls setConf.
static void	deleteFiles(FileIO io, java.lang.Iterable<java.lang.String> files, java.lang.String type, boolean concurrent)	Helper to delete files.
static void	deleteRemovedMetadataFiles(FileIO io, TableMetadata base, TableMetadata metadata)	Deletes the oldest metadata files if TableProperties.METADATA_DELETE_AFTER_COMMIT_ENABLED is true.
static void	dropTableData(FileIO io, TableMetadata metadata)	Drops all data and metadata files referenced by TableMetadata.
static void	dropViewMetadata(FileIO io, ViewMetadata metadata)	Drops view metadata files referenced by ViewMetadata.
static java.lang.String	fullTableName(java.lang.String catalogName, TableIdentifier identifier)
static Catalog	loadCatalog(java.lang.String impl, java.lang.String catalogName, java.util.Map<java.lang.String,java.lang.String> properties, java.lang.Object hadoopConf)	Load a custom catalog implementation.
static FileIO	loadFileIO(java.lang.String impl, java.util.Map<java.lang.String,java.lang.String> properties, java.lang.Object hadoopConf)	Load a custom FileIO implementation.
static MetricsReporter	loadMetricsReporter(java.util.Map<java.lang.String,java.lang.String> properties)	Load a custom MetricsReporter implementation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ICEBERG_CATALOG_TYPE

public static final java.lang.String ICEBERG_CATALOG_TYPE

Shortcut catalog property to load a catalog implementation through a short type name, instead of specifying a full java class through CatalogProperties.CATALOG_IMPL. Currently the following type to implementation mappings are supported:

• hive: org.apache.iceberg.hive.HiveCatalog
• hadoop: org.apache.iceberg.hadoop.HadoopCatalog

See Also:

Constant Field Values

ICEBERG_CATALOG_TYPE_HADOOP

public static final java.lang.String ICEBERG_CATALOG_TYPE_HADOOP

See Also:

Constant Field Values

ICEBERG_CATALOG_TYPE_HIVE

public static final java.lang.String ICEBERG_CATALOG_TYPE_HIVE

See Also:

Constant Field Values

ICEBERG_CATALOG_TYPE_REST

public static final java.lang.String ICEBERG_CATALOG_TYPE_REST

See Also:

Constant Field Values

ICEBERG_CATALOG_TYPE_GLUE

public static final java.lang.String ICEBERG_CATALOG_TYPE_GLUE

See Also:

Constant Field Values

ICEBERG_CATALOG_TYPE_NESSIE

public static final java.lang.String ICEBERG_CATALOG_TYPE_NESSIE

See Also:

Constant Field Values

ICEBERG_CATALOG_TYPE_JDBC

public static final java.lang.String ICEBERG_CATALOG_TYPE_JDBC

See Also:

Constant Field Values

ICEBERG_CATALOG_HADOOP

public static final java.lang.String ICEBERG_CATALOG_HADOOP

See Also:

Constant Field Values

ICEBERG_CATALOG_HIVE

public static final java.lang.String ICEBERG_CATALOG_HIVE

See Also:

Constant Field Values

ICEBERG_CATALOG_REST

public static final java.lang.String ICEBERG_CATALOG_REST

See Also:

Constant Field Values

ICEBERG_CATALOG_GLUE

public static final java.lang.String ICEBERG_CATALOG_GLUE

See Also:

Constant Field Values

ICEBERG_CATALOG_NESSIE

public static final java.lang.String ICEBERG_CATALOG_NESSIE

See Also:

Constant Field Values

ICEBERG_CATALOG_JDBC

public static final java.lang.String ICEBERG_CATALOG_JDBC

See Also:

Constant Field Values

Method Detail

dropTableData

public static void dropTableData(FileIO io,
                                 TableMetadata metadata)

Drops all data and metadata files referenced by TableMetadata.

This should be called by dropTable implementations to clean up table files once the table has been dropped in the metastore.

Parameters:

io - a FileIO to use for deletes

metadata - the last valid TableMetadata instance for a dropped table.

dropViewMetadata

public static void dropViewMetadata(FileIO io,
                                    ViewMetadata metadata)

Drops view metadata files referenced by ViewMetadata.

This should be called by dropView implementations

Parameters:

io - a FileIO to use for deletes

metadata - the last valid ViewMetadata instance for a dropped view.

deleteFiles

public static void deleteFiles(FileIO io,
                               java.lang.Iterable<java.lang.String> files,
                               java.lang.String type,
                               boolean concurrent)

Helper to delete files. Bulk deletion is used if possible.

Parameters:

io - FileIO for deletes

files - files to delete

type - type of files being deleted

concurrent - controls concurrent deletion. Only applicable for non-bulk FileIO

loadCatalog

public static Catalog loadCatalog(java.lang.String impl,
                                  java.lang.String catalogName,
                                  java.util.Map<java.lang.String,java.lang.String> properties,
                                  java.lang.Object hadoopConf)

Load a custom catalog implementation.

The catalog must have a no-arg constructor. If the class implements Configurable, a Hadoop config will be passed using Configurable.setConf. Catalog.initialize(String catalogName, Map options) is called to complete the initialization.

Parameters:

impl - catalog implementation full class name

catalogName - catalog name

properties - catalog properties

hadoopConf - hadoop configuration if needed

Returns:

initialized catalog object

Throws:

java.lang.IllegalArgumentException - if no-arg constructor not found or error during initialization

buildIcebergCatalog

public static Catalog buildIcebergCatalog(java.lang.String name,
                                          java.util.Map<java.lang.String,java.lang.String> options,
                                          java.lang.Object conf)

Build an Iceberg Catalog based on a map of catalog properties and optional Hadoop configuration.

This method examines both the ICEBERG_CATALOG_TYPE and CatalogProperties.CATALOG_IMPL properties to determine the catalog implementation to load. If nothing is specified for both properties, Hive catalog will be loaded by default.

Parameters:

name - catalog name

options - catalog properties

conf - a Hadoop Configuration

Returns:

initialized catalog

loadFileIO

public static FileIO loadFileIO(java.lang.String impl,
                                java.util.Map<java.lang.String,java.lang.String> properties,
                                java.lang.Object hadoopConf)

Load a custom FileIO implementation.

The implementation must have a no-arg constructor. If the class implements Configurable, a Hadoop config will be passed using Configurable.setConf. FileIO.initialize(Map properties) is called to complete the initialization.

Parameters:

impl - full class name of a custom FileIO implementation

properties - used to initialize the FileIO implementation

hadoopConf - a hadoop Configuration

Returns:

FileIO class

Throws:

java.lang.IllegalArgumentException - if class path not found or right constructor not found or the loaded class cannot be cast to the given interface type

configureHadoopConf

public static void configureHadoopConf(java.lang.Object maybeConfigurable,
                                       java.lang.Object conf)

Dynamically detects whether an object is a Hadoop Configurable and calls setConf.

Parameters:

maybeConfigurable - an object that may be Configurable

conf - a Configuration

loadMetricsReporter

public static MetricsReporter loadMetricsReporter(java.util.Map<java.lang.String,java.lang.String> properties)

Load a custom MetricsReporter implementation.

The implementation must have a no-arg constructor.

Parameters:

properties - catalog properties which contains class name of a custom MetricsReporter implementation

Returns:

An initialized MetricsReporter.

Throws:

java.lang.IllegalArgumentException - if class path not found or right constructor not found or the loaded class cannot be cast to the given interface type

fullTableName

public static java.lang.String fullTableName(java.lang.String catalogName,
                                             TableIdentifier identifier)

deleteRemovedMetadataFiles

public static void deleteRemovedMetadataFiles(FileIO io,
                                              TableMetadata base,
                                              TableMetadata metadata)

Deletes the oldest metadata files if TableProperties.METADATA_DELETE_AFTER_COMMIT_ENABLED is true.

Parameters:

io - FileIO instance to use for deletes

base - table metadata on which previous versions were based

metadata - new table metadata with updated previous versions