Class BaseDeleteReachableFilesSparkAction

    • Constructor Detail

      • BaseDeleteReachableFilesSparkAction

        public BaseDeleteReachableFilesSparkAction​(org.apache.spark.sql.SparkSession spark,
                                                   java.lang.String metadataLocation)
    • Method Detail

      • deleteWith

        public DeleteReachableFiles deleteWith​(java.util.function.Consumer<java.lang.String> deleteFunc)
        Description copied from interface: DeleteReachableFiles
        Passes an alternative delete implementation that will be used for files.
        Specified by:
        deleteWith in interface DeleteReachableFiles
        Parameters:
        deleteFunc - a function that will be called to delete files. The function accepts path to file as an argument.
        Returns:
        this for method chaining
      • executeDeleteWith

        public DeleteReachableFiles executeDeleteWith​(java.util.concurrent.ExecutorService executorService)
        Description copied from interface: DeleteReachableFiles
        Passes an alternative executor service that will be used for files removal.

        If this method is not called, files will be deleted in the current thread.

        Specified by:
        executeDeleteWith in interface DeleteReachableFiles
        Parameters:
        executorService - the service to use
        Returns:
        this for method chaining
      • buildOtherMetadataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildOtherMetadataFileDF​(Table table)
      • spark

        protected org.apache.spark.sql.SparkSession spark()
      • sparkContext

        protected org.apache.spark.api.java.JavaSparkContext sparkContext()
      • option

        public ThisT option​(java.lang.String name,
                            java.lang.String value)
        Description copied from interface: Action
        Configures this action with an extra option.

        Certain actions allow users to control internal details of their execution via options.

        Specified by:
        option in interface Action<ThisT,​R>
        Parameters:
        name - an option name
        value - an option value
        Returns:
        this for method chaining
      • options

        public ThisT options​(java.util.Map<java.lang.String,​java.lang.String> newOptions)
        Description copied from interface: Action
        Configures this action with extra options.

        Certain actions allow users to control internal details of their execution via options.

        Specified by:
        options in interface Action<ThisT,​R>
        Parameters:
        newOptions - a map of extra options
        Returns:
        this for method chaining
      • options

        protected java.util.Map<java.lang.String,​java.lang.String> options()
      • withJobGroupInfo

        protected <T> T withJobGroupInfo​(JobGroupInfo info,
                                         java.util.function.Supplier<T> supplier)
      • newJobGroupInfo

        protected JobGroupInfo newJobGroupInfo​(java.lang.String groupId,
                                               java.lang.String desc)
      • buildValidDataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidDataFileDF​(Table table)
      • buildManifestFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestFileDF​(Table table)
      • buildManifestListDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestListDF​(Table table)
      • buildValidMetadataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidMetadataFileDF​(Table table)
      • loadMetadataTable

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> loadMetadataTable​(Table table,
                                                                                           MetadataTableType type)