Package org.apache.iceberg.hadoop
Class HadoopCatalog
- java.lang.Object
-
- org.apache.iceberg.BaseMetastoreCatalog
-
- org.apache.iceberg.hadoop.HadoopCatalog
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,org.apache.hadoop.conf.Configurable
,Catalog
,SupportsNamespaces
public class HadoopCatalog extends BaseMetastoreCatalog implements java.io.Closeable, SupportsNamespaces, org.apache.hadoop.conf.Configurable
HadoopCatalog provides a way to use table names like db.table to work with path-based tables under a common location. It uses a specified directory under a specified filesystem as the warehouse directory, and organizes multiple levels directories that mapped to the database, namespace and the table respectively. The HadoopCatalog takes a location as the warehouse directory. When creating a table such as $db.$tbl, it creates $db/$tbl directory under the warehouse directory, and put the table metadata into that directory. The HadoopCatalog now supportsCatalog.createTable(org.apache.iceberg.catalog.TableIdentifier, org.apache.iceberg.Schema, org.apache.iceberg.PartitionSpec, java.lang.String, java.util.Map<java.lang.String, java.lang.String>)
,Catalog.dropTable(org.apache.iceberg.catalog.TableIdentifier)
, theCatalog.renameTable(org.apache.iceberg.catalog.TableIdentifier, org.apache.iceberg.catalog.TableIdentifier)
is not supported yet. Note: The HadoopCatalog requires that the underlying file system supports atomic rename.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.iceberg.BaseMetastoreCatalog
BaseMetastoreCatalog.BaseMetastoreCatalogTableBuilder
-
Nested classes/interfaces inherited from interface org.apache.iceberg.catalog.Catalog
Catalog.TableBuilder
-
-
Constructor Summary
Constructors Constructor Description HadoopCatalog()
HadoopCatalog(org.apache.hadoop.conf.Configuration conf)
Deprecated.please use the no-arg constructor, setConf and initialize to construct the catalog.HadoopCatalog(org.apache.hadoop.conf.Configuration conf, java.lang.String warehouseLocation)
Deprecated.please use the no-arg constructor, setConf and initialize to construct the catalog.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description Catalog.TableBuilder
buildTable(TableIdentifier identifier, Schema schema)
Instantiate a builder to either create a table or start a create/replace transaction.void
close()
void
createNamespace(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> meta)
Create a namespace in the catalog.protected java.lang.String
defaultWarehouseLocation(TableIdentifier tableIdentifier)
boolean
dropNamespace(Namespace namespace)
Drop a namespace.boolean
dropTable(TableIdentifier identifier, boolean purge)
Drop a table; optionally delete data and metadata files.org.apache.hadoop.conf.Configuration
getConf()
void
initialize(java.lang.String name, java.util.Map<java.lang.String,java.lang.String> properties)
Initialize a catalog given a custom name and a map of catalog properties.protected boolean
isValidIdentifier(TableIdentifier identifier)
java.util.List<Namespace>
listNamespaces(Namespace namespace)
List namespaces from the namespace.java.util.List<TableIdentifier>
listTables(Namespace namespace)
Return all the identifiers under this namespace.java.util.Map<java.lang.String,java.lang.String>
loadNamespaceMetadata(Namespace namespace)
Load metadata properties for a namespace.java.lang.String
name()
Return the name for this catalog.protected TableOperations
newTableOps(TableIdentifier identifier)
boolean
removeProperties(Namespace namespace, java.util.Set<java.lang.String> properties)
Remove a set of metadata from a namespace in the catalog.void
renameTable(TableIdentifier from, TableIdentifier to)
Rename a table.void
setConf(org.apache.hadoop.conf.Configuration conf)
boolean
setProperties(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> properties)
Apply a set of metadata to a namespace in the catalog.java.lang.String
toString()
-
Methods inherited from class org.apache.iceberg.BaseMetastoreCatalog
createTable, fullTableName, loadTable, newCreateTableTransaction, newReplaceTableTransaction
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.apache.iceberg.catalog.Catalog
createTable, createTable, createTable, dropTable, newCreateTableTransaction, newCreateTableTransaction, newCreateTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, tableExists
-
Methods inherited from interface org.apache.iceberg.catalog.SupportsNamespaces
createNamespace, listNamespaces, namespaceExists
-
-
-
-
Constructor Detail
-
HadoopCatalog
public HadoopCatalog()
-
HadoopCatalog
@Deprecated public HadoopCatalog(org.apache.hadoop.conf.Configuration conf, java.lang.String warehouseLocation)
Deprecated.please use the no-arg constructor, setConf and initialize to construct the catalog. Will be removed in v0.13.0The constructor of the HadoopCatalog. It uses the passed location as its warehouse directory.- Parameters:
conf
- The Hadoop configurationwarehouseLocation
- The location used as warehouse directory
-
HadoopCatalog
@Deprecated public HadoopCatalog(org.apache.hadoop.conf.Configuration conf)
Deprecated.please use the no-arg constructor, setConf and initialize to construct the catalog. Will be removed in v0.13.0The constructor of the HadoopCatalog. It gets the value offs.defaultFS
property from the passed Hadoop configuration as its default file system, and use the default directoryiceberg/warehouse
as the warehouse directory.- Parameters:
conf
- The Hadoop configuration
-
-
Method Detail
-
initialize
public void initialize(java.lang.String name, java.util.Map<java.lang.String,java.lang.String> properties)
Description copied from interface:Catalog
Initialize a catalog given a custom name and a map of catalog properties.A custom Catalog implementation must have a no-arg constructor. A compute engine like Spark or Flink will first initialize the catalog without any arguments, and then call this method to complete catalog initialization with properties passed into the engine.
- Specified by:
initialize
in interfaceCatalog
- Parameters:
name
- a custom name for the catalogproperties
- catalog properties
-
name
public java.lang.String name()
Description copied from interface:Catalog
Return the name for this catalog.
-
listTables
public java.util.List<TableIdentifier> listTables(Namespace namespace)
Description copied from interface:Catalog
Return all the identifiers under this namespace.- Specified by:
listTables
in interfaceCatalog
- Parameters:
namespace
- a namespace- Returns:
- a list of identifiers for tables
-
isValidIdentifier
protected boolean isValidIdentifier(TableIdentifier identifier)
- Overrides:
isValidIdentifier
in classBaseMetastoreCatalog
-
newTableOps
protected TableOperations newTableOps(TableIdentifier identifier)
- Specified by:
newTableOps
in classBaseMetastoreCatalog
-
defaultWarehouseLocation
protected java.lang.String defaultWarehouseLocation(TableIdentifier tableIdentifier)
- Specified by:
defaultWarehouseLocation
in classBaseMetastoreCatalog
-
dropTable
public boolean dropTable(TableIdentifier identifier, boolean purge)
Description copied from interface:Catalog
Drop a table; optionally delete data and metadata files.If purge is set to true the implementation should delete all data and metadata files.
-
renameTable
public void renameTable(TableIdentifier from, TableIdentifier to)
Description copied from interface:Catalog
Rename a table.- Specified by:
renameTable
in interfaceCatalog
- Parameters:
from
- identifier of the table to renameto
- new table name
-
createNamespace
public void createNamespace(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> meta)
Description copied from interface:SupportsNamespaces
Create a namespace in the catalog.- Specified by:
createNamespace
in interfaceSupportsNamespaces
- Parameters:
namespace
- a multi-part namespacemeta
- a string Map of properties for the given namespace
-
listNamespaces
public java.util.List<Namespace> listNamespaces(Namespace namespace)
Description copied from interface:SupportsNamespaces
List namespaces from the namespace.For example, if table a.b.t exists, use 'SELECT NAMESPACE IN a' this method must return Namepace.of("a","b")
Namespace
.- Specified by:
listNamespaces
in interfaceSupportsNamespaces
- Returns:
- a List of namespace
Namespace
names
-
dropNamespace
public boolean dropNamespace(Namespace namespace)
Description copied from interface:SupportsNamespaces
Drop a namespace. If the namespace exists and was dropped, this will return true.- Specified by:
dropNamespace
in interfaceSupportsNamespaces
- Parameters:
namespace
- a namespace.Namespace
- Returns:
- true if the namespace was dropped, false otherwise.
-
setProperties
public boolean setProperties(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> properties)
Description copied from interface:SupportsNamespaces
Apply a set of metadata to a namespace in the catalog.- Specified by:
setProperties
in interfaceSupportsNamespaces
- Parameters:
namespace
- a namespace.Namespace
properties
- a collection of metadata to apply to the namespace
-
removeProperties
public boolean removeProperties(Namespace namespace, java.util.Set<java.lang.String> properties)
Description copied from interface:SupportsNamespaces
Remove a set of metadata from a namespace in the catalog.- Specified by:
removeProperties
in interfaceSupportsNamespaces
- Parameters:
namespace
- a namespace.Namespace
properties
- a collection of metadata to apply to the namespace
-
loadNamespaceMetadata
public java.util.Map<java.lang.String,java.lang.String> loadNamespaceMetadata(Namespace namespace)
Description copied from interface:SupportsNamespaces
Load metadata properties for a namespace.- Specified by:
loadNamespaceMetadata
in interfaceSupportsNamespaces
- Parameters:
namespace
- a namespace.Namespace
- Returns:
- a string map of properties for the given namespace
-
close
public void close() throws java.io.IOException
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Throws:
java.io.IOException
-
toString
public java.lang.String toString()
- Overrides:
toString
in classBaseMetastoreCatalog
-
buildTable
public Catalog.TableBuilder buildTable(TableIdentifier identifier, Schema schema)
Description copied from interface:Catalog
Instantiate a builder to either create a table or start a create/replace transaction.- Specified by:
buildTable
in interfaceCatalog
- Overrides:
buildTable
in classBaseMetastoreCatalog
- Parameters:
identifier
- a table identifierschema
- a schema- Returns:
- the builder to create a table or start a create/replace transaction
-
setConf
public void setConf(org.apache.hadoop.conf.Configuration conf)
- Specified by:
setConf
in interfaceorg.apache.hadoop.conf.Configurable
-
getConf
public org.apache.hadoop.conf.Configuration getConf()
- Specified by:
getConf
in interfaceorg.apache.hadoop.conf.Configurable
-
-