Package org.apache.flink.core.fs

Class FSDataOutputStream

  • All Implemented Interfaces:
    Closeable, Flushable, AutoCloseable
    Direct Known Subclasses:
    FSDataOutputStreamWrapper, LocalDataOutputStream, RecoverableFsDataOutputStream, RefCountedFSOutputStream
    @Public
public abstract class FSDataOutputStream
extends OutputStream
    An output stream to a file that is created via a FileSystem. This class extends the base OutputStream with some additional important methods.

    Data Persistence Guarantees

    These streams are used to persistently store data, both for results of streaming applications and for fault tolerance and recovery. It is therefore crucial that the persistence semantics of these streams are well defined.

    Please refer to the class-level docs of FileSystem for the definition of data persistence via Flink's FileSystem abstraction and the FSDataOutputStream.

    Thread Safety

    Implementations of the FSDataOutputStream are generally not assumed to be thread safe. Instances of FSDataOutputStream should not be passed between threads, because there are no guarantees about the order of visibility of operations across threads.

    See Also:
    FileSystem, FSDataInputStream

    • Constructor Detail

      • FSDataOutputStream

        public FSDataOutputStream()

    • Method Detail

      • getPos

        public abstract long getPos()
                     throws IOException
        Gets the position of the stream (non-negative), defined as the number of bytes from the beginning of the file to the current writing position. The position corresponds to the zero-based index of the next byte that will be written.

        This method must report accurately report the current position of the stream. Various components of the high-availability and recovery logic rely on the accurate

        Returns:
        The current position in the stream, defined as the number of bytes from the beginning of the file to the current writing position.
        Throws:
        IOException - Thrown if an I/O error occurs while obtaining the position from the stream implementation.

      • flush

        public abstract void flush()
                    throws IOException
        Flushes the stream, writing any data currently buffered in stream implementation to the proper output stream. After this method has been called, the stream implementation must not hold onto any buffered data any more.

        A completed flush does not mean that the data is necessarily persistent. Data persistence can is only assumed after calls to close() or sync().

        Implementation note: This overrides the method defined in OutputStream as abstract to force implementations of the FSDataOutputStream to implement this method directly.

        Specified by:
        flush in interface Flushable
        Overrides:
        flush in class OutputStream
        Throws:
        IOException - Thrown if an I/O error occurs while flushing the stream.

      • sync

        public abstract void sync()
                   throws IOException
        Flushes the data all the way to the persistent non-volatile storage (for example disks). The method behaves similar to the fsync function, forcing all data to be persistent on the devices.
        Throws:
        IOException - Thrown if an I/O error occurs

      • close

        public abstract void close()
                    throws IOException
        Closes the output stream. After this method returns, the implementation must guarantee that all data written to the stream is persistent/visible, as defined in the class-level docs.

        The above implies that the method must block until persistence can be guaranteed. For example for distributed replicated file systems, the method must block until the replication quorum has been reached. If the calling thread is interrupted in the process, it must fail with an IOException to indicate that persistence cannot be guaranteed.

        If this method throws an exception, the data in the stream cannot be assumed to be persistent.

        Implementation note: This overrides the method defined in OutputStream as abstract to force implementations of the FSDataOutputStream to implement this method directly.

        Specified by:
        close in interface AutoCloseable
        Specified by:
        close in interface Closeable
        Overrides:
        close in class OutputStream
        Throws:
        IOException - Thrown, if an error occurred while closing the stream or guaranteeing that the data is persistent.