Package org.apache.flink.connector.base.sink.writer

Interface RequestBuffer<RequestEntryT extends Serializable>

  • Type Parameters:
    RequestEntryT - The type of request entries being buffered.
    All Known Implementing Classes:
    DequeRequestBuffer
    @PublicEvolving
public interface RequestBuffer<RequestEntryT extends Serializable>
    A flexible wrapper interface for managing buffered request entries in an async sink. This allows sink implementations to define and optimize their own data structures for request buffering.

    RequestEntryWrapper is buffered instead of raw request entries (like InputT) to support metadata tracking (e.g., entry size, retry priority). This makes it easier to manage retries and batch sizing without burdening the sink logic.

    Sink developers can provide custom implementations of this interface (e.g., circular buffer, priority queue) to control how entries are buffered.

    • Method Detail

      • add

        void add​(RequestEntryWrapper<RequestEntryT> entry,
         boolean prioritize)
        Adds an entry (RequestEntryWrapper<RequestEntryT>) to the buffer. Implementations can decide how to store the entry.
        Parameters:
        entry - The request entry to add.
        prioritize - If true, the entry should be prioritized (e.g. retried before others) required to maintain ordering on retries.

      • poll

        RequestEntryWrapper<RequestEntryT> poll()
        Retrieves and removes the next available request entry from the buffer. The removal order is determined by the implementation.
        Returns:
        The removed request entry, or null if the buffer is empty.

      • peek

        RequestEntryWrapper<RequestEntryT> peek()
        Retrieves, but does not remove, the next available request entry from the buffer. This allows checking the next request before processing.
        Returns:
        The next request entry, or null if the buffer is empty.

      • isEmpty

        boolean isEmpty()
        Checks whether the buffer is empty. Useful for determining if there are pending entries before flushing.
        Returns:
        True if the buffer contains no entries, false otherwise.

      • size

        int size()
        Returns the number of request entries currently in the buffer. Can be used for batching decisions.
        Returns:
        The total number of buffered entries.

      • getBufferedState

        Collection<RequestEntryWrapper<RequestEntryT>> getBufferedState()
        Retrieves all buffered request entries as a collection. Implementations should return a snapshot of the buffer for checkpointing.

        The returned collection:

        • Must preserve the order in which entries were added (FIFO).
        • Must not modify or clear the internal buffer.
        Returns:
        A collection of all buffered request entries.

      • totalSizeInBytes

        long totalSizeInBytes()
        Returns the total size of all buffered request entries in bytes.

        Tracks the cumulative size of all elements in bufferedRequestEntries to facilitate the criterion for flushing after maxBatchSizeInBytes is reached.

        Returns:
        The total buffered size in bytes.