Package org.apache.iceberg.types

Class TypeUtil

  • public class TypeUtil
extends java.lang.Object

    • Method Detail

      • project

        public static Schema project​(Schema schema,
                             java.util.Set<java.lang.Integer> fieldIds)
        Project extracts particular fields from a schema by ID.

        Unlike select(Schema, Set), project will pick out only the fields enumerated. Structs that are explicitly projected are empty unless sub-fields are explicitly projected. Maps and lists cannot be explicitly selected in fieldIds.

        Parameters:
        schema - to project fields from
        fieldIds - list of explicit fields to extract
        Returns:
        the schema with all fields fields not selected removed

      • select

        public static Schema select​(Schema schema,
                            java.util.Set<java.lang.Integer> fieldIds)

      • getProjectedIds

        public static java.util.Set<java.lang.Integer> getProjectedIds​(Schema schema)

      • getProjectedIds

        public static java.util.Set<java.lang.Integer> getProjectedIds​(Type type)

      • selectNot

        public static Schema selectNot​(Schema schema,
                               java.util.Set<java.lang.Integer> fieldIds)

      • indexByName

        public static java.util.Map<java.lang.String,​java.lang.Integer> indexByName​(Types.StructType struct)

      • indexNameById

        public static java.util.Map<java.lang.Integer,​java.lang.String> indexNameById​(Types.StructType struct)

      • indexQuotedNameById

        public static java.util.Map<java.lang.Integer,​java.lang.String> indexQuotedNameById​(Types.StructType struct,
                                                                                          java.util.function.Function<java.lang.String,​java.lang.String> quotingFunc)

      • indexByLowerCaseName

        public static java.util.Map<java.lang.String,​java.lang.Integer> indexByLowerCaseName​(Types.StructType struct)
        Creates a mapping from lower-case field names to their corresponding field IDs.

        This method iterates over the fields of the provided struct and maps each field's name (converted to lower-case) to its ID. If two fields have the same lower-case name, an `IllegalArgumentException` is thrown.

        Parameters:
        struct - the struct type whose fields are to be indexed
        Returns:
        a map where the keys are lower-case field names and the values are field IDs
        Throws:
        java.lang.IllegalArgumentException - if two fields have the same lower-case name

      • indexParents

        public static java.util.Map<java.lang.Integer,​java.lang.Integer> indexParents​(Types.StructType struct)

      • assignFreshIds

        public static Type assignFreshIds​(Type type,
                                  TypeUtil.NextID nextId)
        Assigns fresh ids from the nextId function for all fields in a type.
        Parameters:
        type - a type
        nextId - an id assignment function
        Returns:
        an structurally identical type with new ids assigned by the nextId function

      • assignFreshIds

        public static Schema assignFreshIds​(Schema schema,
                                    TypeUtil.NextID nextId)
        Assigns fresh ids from the nextId function for all fields in a schema.
        Parameters:
        schema - a schema
        nextId - an id assignment function
        Returns:
        a structurally identical schema with new ids assigned by the nextId function

      • assignFreshIds

        public static Schema assignFreshIds​(int schemaId,
                                    Schema schema,
                                    TypeUtil.NextID nextId)
        Assigns fresh ids from the nextId function for all fields in a schema.
        Parameters:
        schemaId - an ID assigned to this schema
        schema - a schema
        nextId - an id assignment function
        Returns:
        a structurally identical schema with new ids assigned by the nextId function

      • assignFreshIds

        public static Schema assignFreshIds​(Schema schema,
                                    Schema baseSchema,
                                    TypeUtil.NextID nextId)
        Assigns ids to match a given schema, and fresh ids from the nextId function for all other fields.
        Parameters:
        schema - a schema
        baseSchema - a schema with existing IDs to copy by name
        nextId - an id assignment function
        Returns:
        a structurally identical schema with new ids assigned by the nextId function

      • refreshIdentifierFields

        public static java.util.Set<java.lang.Integer> refreshIdentifierFields​(Types.StructType freshSchema,
                                                                       Schema baseSchema)
        Get the identifier fields in the fresh schema based on the identifier fields in the base schema.
        Parameters:
        freshSchema - fresh schema
        baseSchema - base schema
        Returns:
        identifier fields in the fresh schema

      • assignIncreasingFreshIds

        public static Schema assignIncreasingFreshIds​(Schema schema)
        Assigns strictly increasing fresh ids for all fields in a schema, starting from 1.
        Parameters:
        schema - a schema
        Returns:
        a structurally identical schema with new ids assigned strictly increasing from 1

      • reassignIds

        public static Schema reassignIds​(Schema schema,
                                 Schema idSourceSchema)
        Reassigns ids in a schema from another schema.

        Ids are determined by field names. If a field in the schema cannot be found in the source schema, this will throw IllegalArgumentException.

        This will not alter a schema's structure, nullability, or types.

        Parameters:
        schema - the schema to have ids reassigned
        idSourceSchema - the schema from which field ids will be used
        Returns:
        an structurally identical schema with field ids matching the source schema
        Throws:
        java.lang.IllegalArgumentException - if a field cannot be found (by name) in the source schema

      • reassignDoc

        public static Schema reassignDoc​(Schema schema,
                                 Schema docSourceSchema)
        Reassigns doc in a schema from another schema.

        Doc are determined by field id. If a field in the schema cannot be found in the source schema, this will throw IllegalArgumentException.

        This will not alter a schema's structure, nullability, or types.

        Parameters:
        schema - the schema to have doc reassigned
        docSourceSchema - the schema from which field doc will be used
        Returns:
        an structurally identical schema with field ids matching the source schema
        Throws:
        java.lang.IllegalArgumentException - if a field cannot be found (by id) in the source schema

      • reassignIds

        public static Schema reassignIds​(Schema schema,
                                 Schema idSourceSchema,
                                 boolean caseSensitive)
        Reassigns ids in a schema from another schema.

        Ids are determined by field names. If a field in the schema cannot be found in the source schema, this will throw IllegalArgumentException.

        This will not alter a schema's structure, nullability, or types.

        Parameters:
        schema - the schema to have ids reassigned
        idSourceSchema - the schema from which field ids will be used
        Returns:
        an structurally identical schema with field ids matching the source schema
        Throws:
        java.lang.IllegalArgumentException - if a field cannot be found (by name) in the source schema

      • reassignOrRefreshIds

        public static Schema reassignOrRefreshIds​(Schema schema,
                                          Schema idSourceSchema)

      • reassignOrRefreshIds

        public static Schema reassignOrRefreshIds​(Schema schema,
                                          Schema idSourceSchema,
                                          boolean caseSensitive)

      • assignIds

        public static Type assignIds​(Type type,
                             TypeUtil.GetID getId)
        Assigns fresh ids from the getId function for all fields in a type.
        Parameters:
        type - a type
        getId - an id assignment function
        Returns:
        an structurally identical type with new ids assigned by the getId function

      • find

        public static Type find​(Schema schema,
                        java.util.function.Predicate<Type> predicate)

      • validateWriteSchema

        public static void validateWriteSchema​(Schema tableSchema,
                                       Schema writeSchema,
                                       java.lang.Boolean checkNullability,
                                       java.lang.Boolean checkOrdering)
        Check whether we could write the iceberg table with the user-provided write schema.
        Parameters:
        tableSchema - the table schema written in iceberg meta data.
        writeSchema - the user-provided write schema.
        checkNullability - If true, not allow to write optional values to a required field.
        checkOrdering - If true, not allow input schema to have different ordering than table schema.

      • validateSchema

        public static void validateSchema​(java.lang.String context,
                                  Schema expectedSchema,
                                  Schema providedSchema,
                                  boolean checkNullability,
                                  boolean checkOrdering)
        Validates whether the provided schema is compatible with the expected schema.
        Parameters:
        context - the schema context (e.g. row ID)
        expectedSchema - the expected schema
        providedSchema - the provided schema
        checkNullability - whether to check field nullability
        checkOrdering - whether to check field ordering

      • estimateSize

        public static int estimateSize​(Types.NestedField field)
        Estimates the number of bytes a value for a given field may occupy in memory.

        This method approximates the memory size based on heuristics and the internal Java representation defined by Type.TypeID. It is important to note that the actual size might differ from this estimation. The method is designed to handle a variety of data types, including primitive types, strings, and nested types such as structs, maps, and lists.

        Parameters:
        field - a field for which to estimate the size
        Returns:
        the estimated size in bytes of the field's value in memory

      • visit

        public static <T> T visit​(Type type,
                          TypeUtil.CustomOrderSchemaVisitor<T> visitor)
        Used to traverse types with traversals other than post-order.

        This passes a Supplier to each visitor method that returns the result of traversing child types. Structs are passed an Iterable that traverses child fields during iteration.

        An example use is assigning column IDs, which should be done with a pre-order traversal.

        Type Parameters:
        T - the type returned by the visitor
        Parameters:
        type - a type to traverse with a visitor
        visitor - a custom order visitor
        Returns:
        the result of traversing the given type with the visitor

      • decimalRequiredBytes

        public static int decimalRequiredBytes​(int precision)