Package org.apache.flink.api.java.hadoop.mapred

Class HadoopOutputFormatBase<K,​V,​T>

  • Type Parameters:
    K - Type of Key
    V - Type of Value
    T - Record type.
    All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.io.FinalizeOnMaster, org.apache.flink.api.common.io.OutputFormat<T>
    Direct Known Subclasses:
    HadoopOutputFormat
    @Internal
public abstract class HadoopOutputFormatBase<K,​V,​T>
extends HadoopOutputFormatCommonBase<T>
implements org.apache.flink.api.common.io.FinalizeOnMaster
    Common base for the mapred HadoopOutputFormat wrappers. There are implementations for Java and Scala.
    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from interface org.apache.flink.api.common.io.FinalizeOnMaster

        org.apache.flink.api.common.io.FinalizeOnMaster.FinalizationContext

      • Nested classes/interfaces inherited from interface org.apache.flink.api.common.io.OutputFormat

        org.apache.flink.api.common.io.OutputFormat.InitializationContext

    • Constructor Summary

      Constructors 
      Constructor Description
      HadoopOutputFormatBase​(org.apache.hadoop.mapred.OutputFormat<K,​V> mapredOutputFormat, org.apache.hadoop.mapred.JobConf job)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()
      commit the task by moving the output file out from the temporary directory.
      void configure​(org.apache.flink.configuration.Configuration parameters)  
      void finalizeGlobal​(org.apache.flink.api.common.io.FinalizeOnMaster.FinalizationContext context)  
      org.apache.hadoop.mapred.JobConf getJobConf()  
      void open​(org.apache.flink.api.common.io.OutputFormat.InitializationContext context)
      create the temporary output file for hadoop RecordWriter.

      • Methods inherited from class org.apache.flink.api.common.io.RichOutputFormat

        getRuntimeContext, setRuntimeContext

      • Methods inherited from interface org.apache.flink.api.common.io.OutputFormat

        writeRecord

    • Field Detail

      • OPEN_MUTEX

        protected static final Object OPEN_MUTEX

      • CONFIGURE_MUTEX

        protected static final Object CONFIGURE_MUTEX

      • CLOSE_MUTEX

        protected static final Object CLOSE_MUTEX

      • jobConf

        protected org.apache.hadoop.mapred.JobConf jobConf

      • mapredOutputFormat

        protected org.apache.hadoop.mapred.OutputFormat<K,​V> mapredOutputFormat

      • recordWriter

        protected transient org.apache.hadoop.mapred.RecordWriter<K,​V> recordWriter

      • outputCommitter

        protected transient org.apache.hadoop.mapred.OutputCommitter outputCommitter

      • context

        protected transient org.apache.hadoop.mapred.TaskAttemptContext context

    • Constructor Detail

      • HadoopOutputFormatBase

        public HadoopOutputFormatBase​(org.apache.hadoop.mapred.OutputFormat<K,​V> mapredOutputFormat,
                              org.apache.hadoop.mapred.JobConf job)

    • Method Detail

      • getJobConf

        public org.apache.hadoop.mapred.JobConf getJobConf()

      • configure

        public void configure​(org.apache.flink.configuration.Configuration parameters)
        Specified by:
        configure in interface org.apache.flink.api.common.io.OutputFormat<K>

      • open

        public void open​(org.apache.flink.api.common.io.OutputFormat.InitializationContext context)
          throws IOException
        create the temporary output file for hadoop RecordWriter.
        Specified by:
        open in interface org.apache.flink.api.common.io.OutputFormat<K>
        Parameters:
        context - The initialization context.
        Throws:
        IOException

      • close

        public void close()
           throws IOException
        commit the task by moving the output file out from the temporary directory.
        Specified by:
        close in interface org.apache.flink.api.common.io.OutputFormat<K>
        Throws:
        IOException

      • finalizeGlobal

        public void finalizeGlobal​(org.apache.flink.api.common.io.FinalizeOnMaster.FinalizationContext context)
                    throws IOException
        Specified by:
        finalizeGlobal in interface org.apache.flink.api.common.io.FinalizeOnMaster
        Throws:
        IOException