Package org.apache.iceberg.arrow.vectorized

Class ArrowReader

java.lang.Object

org.apache.iceberg.io.CloseableGroup

org.apache.iceberg.arrow.vectorized.ArrowReader

All Implemented Interfaces:

Closeable, AutoCloseable

public class ArrowReader
extends CloseableGroup

Vectorized reader that returns an iterator of ColumnarBatch. See open(CloseableIterable) ()} to learn about the behavior of the iterator.

The following Iceberg data types are supported and have been tested:

• Iceberg: Types.BooleanType, Arrow: Types.MinorType.BIT
• Iceberg: Types.IntegerType, Arrow: Types.MinorType.INT
• Iceberg: Types.LongType, Arrow: Types.MinorType.BIGINT
• Iceberg: Types.FloatType, Arrow: Types.MinorType.FLOAT4
• Iceberg: Types.DoubleType, Arrow: Types.MinorType.FLOAT8
• Iceberg: Types.DecimalType, Arrow: Types.MinorType.DECIMAL
• Iceberg: Types.StringType, Arrow: Types.MinorType.VARCHAR
• Iceberg: Types.TimestampType (both with and without timezone), Arrow: Types.MinorType.TIMESTAMPMICRO and Types.MinorType.TIMESTAMPMICROTZ
• Iceberg: Types.TimestampNanoType (both with and without timezone), Arrow: Types.MinorType.TIMESTAMPNANO and Types.MinorType.TIMESTAMPNANOTZ
• Iceberg: Types.BinaryType, Arrow: Types.MinorType.VARBINARY
• Iceberg: Types.FixedType, Arrow: Types.MinorType.FIXEDSIZEBINARY
• Iceberg: Types.DateType, Arrow: Types.MinorType.DATEDAY
• Iceberg: Types.TimeType, Arrow: Types.MinorType.TIMEMICRO
• Iceberg: Types.UUIDType, Arrow: Types.MinorType.FIXEDSIZEBINARY(16)

Features that don't work in this implementation:

• Type promotion: In case of type promotion, the Arrow vector corresponding to the data type in the parquet file is returned instead of the data type in the latest schema. See https://github.com/apache/iceberg/issues/2483.
• Columns with constant values are physically encoded as a dictionary. The Arrow vector type is int32 instead of the type as per the schema. See https://github.com/apache/iceberg/issues/2484.
• Data types: Types.ListType, Types.MapType and Types.StructType See https://github.com/apache/iceberg/issues/2485 and https://github.com/apache/iceberg/issues/2486.
• Delete files are not supported. See https://github.com/apache/iceberg/issues/2487.

Constructor Summary

Constructors

Constructor	Description
ArrowReader(TableScan scan, int batchSize, boolean reuseContainers)	Create a new instance of the reader.

Method Summary

Modifier and Type	Method	Description
void	close()	Close all the registered resources.
CloseableIterator<ColumnarBatch>	open(CloseableIterable<CombinedScanTask> tasks)	Returns a new iterator of ColumnarBatch objects.

Methods inherited from class org.apache.iceberg.io.CloseableGroup

addCloseable, addCloseable, setSuppressCloseFailure

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ArrowReader

public ArrowReader(TableScan scan,
 int batchSize,
 boolean reuseContainers)

Create a new instance of the reader.

Parameters:

scan - the table scan object.

batchSize - the maximum number of rows per Arrow batch.

reuseContainers - whether to reuse Arrow vectors when iterating through the data. If set to false, every Iterator.next() call creates new instances of Arrow vectors. If set to true, the Arrow vectors in the previous Iterator.next() may be reused for the data returned in the current Iterator.next(). This option avoids allocating memory again and again. Irrespective of the value of reuseContainers, the Arrow vectors in the previous Iterator.next() call are closed before creating new instances if the current Iterator.next().

Method Details

open

public CloseableIterator<ColumnarBatch> open(CloseableIterable<CombinedScanTask> tasks)

Returns a new iterator of ColumnarBatch objects.

Note that the reader owns the ColumnarBatch objects and takes care of closing them. The caller should not hold onto a ColumnarBatch or try to close them.

If reuseContainers is false, the Arrow vectors in the previous ColumnarBatch are closed before returning the next ColumnarBatch object. This implies that the caller should either use the ColumnarBatch or transfer the ownership of ColumnarBatch before getting the next ColumnarBatch.

If reuseContainers is true, the Arrow vectors in the previous ColumnarBatch may be reused for the next ColumnarBatch. This implies that the caller should either use the ColumnarBatch or deep copy the ColumnarBatch before getting the next ColumnarBatch.

This method works for only when the following conditions are true:

1. At least one column is queried,
2. There are no delete files, and
3. Supported data types are queried (see SUPPORTED_TYPES).

When any of these conditions fail, an UnsupportedOperationException is thrown.

close

public void close()
           throws IOException

Description copied from class: CloseableGroup

Close all the registered resources. Close method of each resource will only be called once. Checked exception from AutoCloseable will be wrapped to runtime exception.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Overrides:

close in class CloseableGroup

Throws:

IOException