Package org.apache.flink.formats.avro

Class AvroInputFormat<E>

  • Type Parameters:
    E - the type of the result Avro record. If you specify GenericRecord then the result will be returned as a GenericRecord, so you do not have to know the schema ahead of time.
    All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,​org.apache.flink.api.java.tuple.Tuple2<Long,​Long>>, org.apache.flink.api.common.io.InputFormat<E,​org.apache.flink.core.fs.FileInputSplit>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<E>, org.apache.flink.core.io.InputSplitSource<org.apache.flink.core.fs.FileInputSplit>
    public class AvroInputFormat<E>
extends org.apache.flink.api.common.io.FileInputFormat<E>
implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<E>, org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,​org.apache.flink.api.java.tuple.Tuple2<Long,​Long>>
    Provides a FileInputFormat for Avro records.
    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

        org.apache.flink.api.common.io.FileInputFormat.FileBaseStatistics, org.apache.flink.api.common.io.FileInputFormat.InputSplitOpenThread

    • Field Summary

      • Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

        currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

    • Constructor Summary

      Constructors 
      Constructor Description
      AvroInputFormat​(org.apache.flink.core.fs.Path filePath, Class<E> type)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.flink.api.java.tuple.Tuple2<Long,​Long> getCurrentState()  
      org.apache.flink.api.common.typeinfo.TypeInformation<E> getProducedType()  
      long getRecordsReadFromBlock()  
      E nextRecord​(E reuseValue)  
      void open​(org.apache.flink.core.fs.FileInputSplit split)  
      boolean reachedEnd()  
      void reopen​(org.apache.flink.core.fs.FileInputSplit split, org.apache.flink.api.java.tuple.Tuple2<Long,​Long> state)  
      void setReuseAvroValue​(boolean reuseAvroValue)
      Sets the flag whether to reuse the Avro value instance for all records.
      void setUnsplittable​(boolean unsplittable)
      If set, the InputFormat will only read entire files.
      boolean supportsMultiPaths()  

      • Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

        acceptFile, close, configure, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFilePaths, getFileStats, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNestedFileEnumeration, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, getStatistics, getSupportedCompressionFormats, registerInflaterInputStreamFactory, setFilePath, setFilePath, setFilePaths, setFilePaths, setFilesFilter, setMinSplitSize, setNestedFileEnumeration, setNumSplits, setOpenTimeout, testForUnsplittable, toString

      • Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

        closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

    • Constructor Detail

      • AvroInputFormat

        public AvroInputFormat​(org.apache.flink.core.fs.Path filePath,
                       Class<E> type)

    • Method Detail

      • setReuseAvroValue

        public void setReuseAvroValue​(boolean reuseAvroValue)
        Sets the flag whether to reuse the Avro value instance for all records. By default, the input format reuses the Avro value.
        Parameters:
        reuseAvroValue - True, if the input format should reuse the Avro value instance, false otherwise.

      • setUnsplittable

        public void setUnsplittable​(boolean unsplittable)
        If set, the InputFormat will only read entire files.

      • getProducedType

        public org.apache.flink.api.common.typeinfo.TypeInformation<E> getProducedType()
        Specified by:
        getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<E>

      • open

        public void open​(org.apache.flink.core.fs.FileInputSplit split)
          throws IOException
        Specified by:
        open in interface org.apache.flink.api.common.io.InputFormat<E,​org.apache.flink.core.fs.FileInputSplit>
        Overrides:
        open in class org.apache.flink.api.common.io.FileInputFormat<E>
        Throws:
        IOException

      • reachedEnd

        public boolean reachedEnd()
                   throws IOException
        Specified by:
        reachedEnd in interface org.apache.flink.api.common.io.InputFormat<E,​org.apache.flink.core.fs.FileInputSplit>
        Throws:
        IOException

      • getRecordsReadFromBlock

        public long getRecordsReadFromBlock()

      • nextRecord

        public E nextRecord​(E reuseValue)
             throws IOException
        Specified by:
        nextRecord in interface org.apache.flink.api.common.io.InputFormat<E,​org.apache.flink.core.fs.FileInputSplit>
        Throws:
        IOException

      • supportsMultiPaths

        public boolean supportsMultiPaths()
        Overrides:
        supportsMultiPaths in class org.apache.flink.api.common.io.FileInputFormat<E>

      • getCurrentState

        public org.apache.flink.api.java.tuple.Tuple2<Long,​Long> getCurrentState()
                                                                        throws IOException
        Specified by:
        getCurrentState in interface org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,​org.apache.flink.api.java.tuple.Tuple2<Long,​Long>>
        Throws:
        IOException

      • reopen

        public void reopen​(org.apache.flink.core.fs.FileInputSplit split,
                   org.apache.flink.api.java.tuple.Tuple2<Long,​Long> state)
            throws IOException
        Specified by:
        reopen in interface org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,​org.apache.flink.api.java.tuple.Tuple2<Long,​Long>>
        Throws:
        IOException