Package org.apache.iceberg.io

Class ResolvingFileIO

java.lang.Object

org.apache.iceberg.io.ResolvingFileIO

All Implemented Interfaces:

java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable, org.apache.hadoop.conf.Configurable, HadoopConfigurable, DelegateFileIO, FileIO, SupportsBulkOperations, SupportsPrefixOperations

public class ResolvingFileIO
extends java.lang.Object
implements HadoopConfigurable, DelegateFileIO

FileIO implementation that uses location scheme to choose the correct FileIO implementation. Delegate FileIO implementations must implement the DelegateFileIO mixin interface, otherwise initialization will fail.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor	Description
ResolvingFileIO()	No-arg constructor to load the FileIO dynamically.

Method Summary

Modifier and Type	Method	Description
void	close()	Close File IO to release underlying resources.
void	deleteFile(java.lang.String location)	Delete the file at the given path.
void	deleteFiles(java.lang.Iterable<java.lang.String> pathsToDelete)	Delete the files at the given paths.
void	deletePrefix(java.lang.String prefix)	Delete all files under a prefix.
protected void	finalize()
org.apache.hadoop.conf.Configuration	getConf()
void	initialize(java.util.Map<java.lang.String,java.lang.String> newProperties)	Initialize File IO from catalog properties.
java.lang.Class<?>	ioClass(java.lang.String location)
java.lang.Iterable<FileInfo>	listPrefix(java.lang.String prefix)	Return an iterable of all files under a prefix.
InputFile	newInputFile(java.lang.String location)	Get a InputFile instance to read bytes from the file at the given path.
InputFile	newInputFile(java.lang.String location, long length)	Get a InputFile instance to read bytes from the file at the given path, with a known file length.
OutputFile	newOutputFile(java.lang.String location)	Get a OutputFile instance to write bytes to the file at the given path.
java.util.Map<java.lang.String,java.lang.String>	properties()	Returns the property map used to configure this FileIO
void	serializeConfWith(java.util.function.Function<org.apache.hadoop.conf.Configuration,SerializableSupplier<org.apache.hadoop.conf.Configuration>> confSerializer)	Take a function that serializes Hadoop configuration into a supplier.
void	setConf(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.io.FileIO

deleteFile, deleteFile, newInputFile, newInputFile, newInputFile

Constructor Detail

ResolvingFileIO

public ResolvingFileIO()

No-arg constructor to load the FileIO dynamically.

All fields are initialized by calling initialize(Map) later.

Method Detail

newInputFile

public InputFile newInputFile(java.lang.String location)

Description copied from interface: FileIO

Get a InputFile instance to read bytes from the file at the given path.

Specified by:

newInputFile in interface FileIO

newInputFile

public InputFile newInputFile(java.lang.String location,
                              long length)

Description copied from interface: FileIO

Get a InputFile instance to read bytes from the file at the given path, with a known file length.

Specified by:

newInputFile in interface FileIO

newOutputFile

public OutputFile newOutputFile(java.lang.String location)

Description copied from interface: FileIO

Get a OutputFile instance to write bytes to the file at the given path.

Specified by:

newOutputFile in interface FileIO

deleteFile

public void deleteFile(java.lang.String location)

Description copied from interface: FileIO

Delete the file at the given path.

Specified by:

deleteFile in interface FileIO

deleteFiles

public void deleteFiles(java.lang.Iterable<java.lang.String> pathsToDelete)
                 throws BulkDeletionFailureException

Description copied from interface: SupportsBulkOperations

Delete the files at the given paths.

Specified by:

deleteFiles in interface SupportsBulkOperations

Parameters:

pathsToDelete - The paths to delete

Throws:

BulkDeletionFailureException - in case of failure to delete at least 1 file

properties

public java.util.Map<java.lang.String,java.lang.String> properties()

Description copied from interface: FileIO

Returns the property map used to configure this FileIO

Specified by:

properties in interface FileIO

initialize

public void initialize(java.util.Map<java.lang.String,java.lang.String> newProperties)

Description copied from interface: FileIO

Initialize File IO from catalog properties.

Specified by:

initialize in interface FileIO

Parameters:

newProperties - catalog properties

close

public void close()

Description copied from interface: FileIO

Close File IO to release underlying resources.

Calling this method is only required when this FileIO instance is no longer expected to be used, and the resources it holds need to be explicitly released to avoid resource leaks.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface FileIO

serializeConfWith

public void serializeConfWith(java.util.function.Function<org.apache.hadoop.conf.Configuration,SerializableSupplier<org.apache.hadoop.conf.Configuration>> confSerializer)

Description copied from interface: HadoopConfigurable

Take a function that serializes Hadoop configuration into a supplier. An implementation is supposed to pass in its current Hadoop configuration into this function, and the result can be safely serialized for future use.

Specified by:

serializeConfWith in interface HadoopConfigurable

Parameters:

confSerializer - A function that takes Hadoop configuration and returns a serializable supplier of it.

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

ioClass

public java.lang.Class<?> ioClass(java.lang.String location)

finalize

protected void finalize()
                 throws java.lang.Throwable

Overrides:

finalize in class java.lang.Object

Throws:

java.lang.Throwable

listPrefix

public java.lang.Iterable<FileInfo> listPrefix(java.lang.String prefix)

Description copied from interface: SupportsPrefixOperations

Return an iterable of all files under a prefix.

Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.

Specified by:

listPrefix in interface SupportsPrefixOperations

Parameters:

prefix - prefix to list

Returns:

iterable of file information

deletePrefix

public void deletePrefix(java.lang.String prefix)

Description copied from interface: SupportsPrefixOperations

Delete all files under a prefix.

Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.

Specified by:

deletePrefix in interface SupportsPrefixOperations

Parameters:

prefix - prefix to delete