Package org.apache.flink.api.connector.source

Interface SplitEnumeratorContext<SplitT extends SourceSplit>

  • Type Parameters:
    SplitT - the type of the splits.
    @Public
public interface SplitEnumeratorContext<SplitT extends SourceSplit>
    A context class for the SplitEnumerator. This class serves the following purposes:
    1. Host information necessary for the SplitEnumerator to make split assignment decisions.
    2. Accept and track the split assignment from the enumerator.
    3. Provide a managed threading model so the split enumerators do not need to create their own internal threads.

    • Method Detail

      • metricGroup

        org.apache.flink.metrics.groups.SplitEnumeratorMetricGroup metricGroup()

      • sendEventToSourceReader

        void sendEventToSourceReader​(int subtaskId,
                             SourceEvent event)
        Send a source event to a source reader. The source reader is identified by its subtask id.
        Parameters:
        subtaskId - the subtask id of the source reader to send this event to.
        event - the source event to send.

      • sendEventToSourceReader

        default void sendEventToSourceReader​(int subtaskId,
                                     int attemptNumber,
                                     SourceEvent event)
        Send a source event to a source reader. The source reader is identified by its subtask id and attempt number. It is similar to sendEventToSourceReader(int, SourceEvent) but it is aware of the subtask execution attempt to send this event to.

        The SplitEnumerator must invoke this method instead of sendEventToSourceReader(int, SourceEvent) if it is used in cases that a subtask can have multiple concurrent execution attempts, e.g. if speculative execution is enabled. Otherwise an error will be thrown when the split enumerator tries to send a custom source event.

        Parameters:
        subtaskId - the subtask id of the source reader to send this event to.
        attemptNumber - the attempt number of the source reader to send this event to.
        event - the source event to send.

      • currentParallelism

        int currentParallelism()
        Get the current parallelism of this Source. Note that due to auto-scaling, the parallelism may change over time. Therefore the SplitEnumerator should not cache the return value of this method, but always invoke this method to get the latest parallelism.
        Returns:
        the parallelism of the Source.

      • registeredReaders

        Map<Integer,​ReaderInfo> registeredReaders()
        Get the currently registered readers. The mapping is from subtask id to the reader info.

        Note that if a subtask has multiple concurrent attempts, the map will contain the earliest attempt of that subtask. This is for compatibility purpose. It's recommended to use registeredReadersOfAttempts() instead.

        Returns:
        the currently registered readers.

      • registeredReadersOfAttempts

        default Map<Integer,​Map<Integer,​ReaderInfo>> registeredReadersOfAttempts()
        Get the currently registered readers of all the subtask attempts. The mapping is from subtask id to a map which maps an attempt to its reader info.
        Returns:
        the currently registered readers.

      • assignSplits

        void assignSplits​(SplitsAssignment<SplitT> newSplitAssignments)
        Assign the splits.
        Parameters:
        newSplitAssignments - the new split assignments to add.

      • assignSplit

        default void assignSplit​(SplitT split,
                         int subtask)
        Assigns a single split.

        When assigning multiple splits, it is more efficient to assign all of them in a single call to the assignSplits(SplitsAssignment) method.

        Parameters:
        split - The new split
        subtask - The index of the operator's parallel subtask that shall receive the split.

      • signalNoMoreSplits

        void signalNoMoreSplits​(int subtask)
        Signals a subtask that it will not receive any further split.
        Parameters:
        subtask - The index of the operator's parallel subtask that shall be signaled it will not receive any further split.

      • callAsync

        <T> void callAsync​(Callable<T> callable,
                   java.util.function.BiConsumer<T,​Throwable> handler)
        Invoke the callable and handover the return value to the handler which will be executed by the source coordinator. When this method is invoked multiple times, The Callable s may be executed in a thread pool concurrently.

        It is important to make sure that the callable does not modify any shared state, especially the states that will be a part of the SplitEnumerator.snapshotState(long). Otherwise, there might be unexpected behavior.

        Note that an exception thrown from the handler would result in failing the job.

        Parameters:
        callable - a callable to call.
        handler - a handler that handles the return value of or the exception thrown from the callable.

      • callAsync

        <T> void callAsync​(Callable<T> callable,
                   java.util.function.BiConsumer<T,​Throwable> handler,
                   long initialDelayMillis,
                   long periodMillis)
        Invoke the given callable periodically and handover the return value to the handler which will be executed by the source coordinator. When this method is invoked multiple times, The Callables may be executed in a thread pool concurrently.

        It is important to make sure that the callable does not modify any shared state, especially the states that will be a part of the SplitEnumerator.snapshotState(long). Otherwise, there might be unexpected behavior.

        Note that an exception thrown from the handler would result in failing the job.

        Parameters:
        callable - the callable to call.
        handler - a handler that handles the return value of or the exception thrown from the callable.
        initialDelayMillis - the initial delay of calling the callable, in milliseconds.
        periodMillis - the period between two invocations of the callable, in milliseconds.

      • runInCoordinatorThread

        void runInCoordinatorThread​(Runnable runnable)
        Invoke the given runnable in the source coordinator thread.

        This can be useful when the enumerator needs to execute some action (like assignSplits) triggered by some external events. E.g., Watermark from another source advanced and this source now be able to assign splits to awaiting readers. The trigger can be initiated from the coordinator thread of the other source. Instead of using lock for thread safety, this API allows to run such externally triggered action in the coordinator thread. Hence, we can ensure all enumerator actions are serialized in the single coordinator thread.

        It is important that the runnable does not block.

        Parameters:
        runnable - a runnable to execute