Package org.apache.flink.runtime.dispatcher

Class Dispatcher

  • All Implemented Interfaces:
    AutoCloseable, DispatcherGateway, org.apache.flink.runtime.rpc.FencedRpcGateway<DispatcherId>, org.apache.flink.runtime.rpc.RpcGateway, RestfulGateway, org.apache.flink.util.AutoCloseableAsync
    Direct Known Subclasses:
    MiniDispatcher, StandaloneDispatcher
    public abstract class Dispatcher
extends org.apache.flink.runtime.rpc.FencedRpcEndpoint<DispatcherId>
implements DispatcherGateway
    Base class for the Dispatcher component. The Dispatcher component is responsible for receiving job submissions, persisting them, spawning JobManagers to execute the jobs and to recover them in case of a master failure. Furthermore, it knows about the state of the Flink session cluster.

    • Field Detail

      • CLIENT_ALIVENESS_CHECK_DURATION

        @VisibleForTesting
public static final org.apache.flink.configuration.ConfigOption<java.time.Duration> CLIENT_ALIVENESS_CHECK_DURATION

    • Method Detail

      • onStart

        public void onStart()
             throws Exception
        Overrides:
        onStart in class org.apache.flink.runtime.rpc.RpcEndpoint
        Throws:
        Exception

      • onStop

        public CompletableFuture<Void> onStop()
        Overrides:
        onStop in class org.apache.flink.runtime.rpc.RpcEndpoint

      • submitJob

        public CompletableFuture<Acknowledge> submitJob​(JobGraph jobGraph,
                                                org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: DispatcherGateway
        Submit a job to the dispatcher.
        Specified by:
        submitJob in interface DispatcherGateway
        Parameters:
        jobGraph - JobGraph to submit
        timeout - RPC timeout
        Returns:
        A future acknowledge if the submission succeeded

      • listJobs

        public CompletableFuture<Collection<org.apache.flink.api.common.JobID>> listJobs​(org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: DispatcherGateway
        List the current set of submitted jobs.
        Specified by:
        listJobs in interface DispatcherGateway
        Parameters:
        timeout - RPC timeout
        Returns:
        A future collection of currently submitted jobs

      • disposeSavepoint

        public CompletableFuture<Acknowledge> disposeSavepoint​(String savepointPath,
                                                       org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: RestfulGateway
        Dispose the given savepoint.
        Specified by:
        disposeSavepoint in interface RestfulGateway
        Parameters:
        savepointPath - identifying the savepoint to dispose
        timeout - RPC timeout
        Returns:
        A future acknowledge if the disposal succeeded

      • cancelJob

        public CompletableFuture<Acknowledge> cancelJob​(org.apache.flink.api.common.JobID jobId,
                                                org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: RestfulGateway
        Cancel the given job.
        Specified by:
        cancelJob in interface RestfulGateway
        Parameters:
        jobId - identifying the job to cancel
        timeout - of the operation
        Returns:
        A future acknowledge if the cancellation succeeded

      • requestJobStatus

        public CompletableFuture<org.apache.flink.api.common.JobStatus> requestJobStatus​(org.apache.flink.api.common.JobID jobId,
                                                                                 org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: RestfulGateway
        Request the JobStatus of the given job.
        Specified by:
        requestJobStatus in interface RestfulGateway
        Parameters:
        jobId - identifying the job for which to retrieve the JobStatus
        timeout - for the asynchronous operation
        Returns:
        A future to the JobStatus of the given job

      • requestJobResult

        public CompletableFuture<JobResult> requestJobResult​(org.apache.flink.api.common.JobID jobId,
                                                     org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: RestfulGateway
        Requests the JobResult of a job specified by the given jobId.
        Specified by:
        requestJobResult in interface RestfulGateway
        Parameters:
        jobId - identifying the job for which to retrieve the JobResult.
        timeout - for the asynchronous operation
        Returns:
        Future which is completed with the job's JobResult once the job has finished

      • requestThreadDump

        public CompletableFuture<ThreadDumpInfo> requestThreadDump​(org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: RestfulGateway
        Requests the thread dump from the JobManager.
        Specified by:
        requestThreadDump in interface RestfulGateway
        Parameters:
        timeout - timeout of the asynchronous operation
        Returns:
        Future containing the thread dump information

      • triggerCheckpoint

        public CompletableFuture<Acknowledge> triggerCheckpoint​(AsynchronousJobOperationKey operationKey,
                                                        org.apache.flink.core.execution.CheckpointType checkpointType,
                                                        org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: RestfulGateway
        Triggers a checkpoint with the given savepoint directory as a target.
        Specified by:
        triggerCheckpoint in interface RestfulGateway
        Parameters:
        operationKey - the key of the operation, for deduplication purposes
        checkpointType - checkpoint backup type (configured / full / incremental)
        timeout - Timeout for the asynchronous operation
        Returns:
        A future to the external pointer of the savepoint.

      • triggerCheckpointAndGetCheckpointID

        public CompletableFuture<Long> triggerCheckpointAndGetCheckpointID​(org.apache.flink.api.common.JobID jobID,
                                                                   org.apache.flink.core.execution.CheckpointType checkpointType,
                                                                   org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: DispatcherGateway
        Triggers a checkpoint, returning a future that completes with the checkpoint id when it is complete.
        Specified by:
        triggerCheckpointAndGetCheckpointID in interface DispatcherGateway
        Parameters:
        jobID - the job id
        checkpointType - checkpoint type of this checkpoint (configured / full / incremental)
        timeout - Timeout for the asynchronous operation
        Returns:
        Future which is completed once the operation is triggered successfully

      • triggerSavepoint

        public CompletableFuture<Acknowledge> triggerSavepoint​(AsynchronousJobOperationKey operationKey,
                                                       String targetDirectory,
                                                       org.apache.flink.core.execution.SavepointFormatType formatType,
                                                       TriggerSavepointMode savepointMode,
                                                       org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: RestfulGateway
        Triggers a savepoint with the given savepoint directory as a target, returning a future that completes when the operation is started.
        Specified by:
        triggerSavepoint in interface RestfulGateway
        Parameters:
        operationKey - the key of the operation, for deduplication purposes
        targetDirectory - Target directory for the savepoint.
        formatType - Binary format of the savepoint.
        savepointMode - context of the savepoint operation
        timeout - Timeout for the asynchronous operation
        Returns:
        Future which is completed once the operation is triggered successfully

      • triggerSavepointAndGetLocation

        public CompletableFuture<String> triggerSavepointAndGetLocation​(org.apache.flink.api.common.JobID jobId,
                                                                String targetDirectory,
                                                                org.apache.flink.core.execution.SavepointFormatType formatType,
                                                                TriggerSavepointMode savepointMode,
                                                                org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: DispatcherGateway
        Triggers a savepoint with the given savepoint directory as a target, returning a future that completes with the savepoint location when it is complete.
        Specified by:
        triggerSavepointAndGetLocation in interface DispatcherGateway
        Parameters:
        jobId - the job id
        targetDirectory - Target directory for the savepoint.
        formatType - Binary format of the savepoint.
        savepointMode - context of the savepoint operation
        timeout - Timeout for the asynchronous operation
        Returns:
        Future which is completed once the operation is triggered successfully

      • stopWithSavepoint

        public CompletableFuture<Acknowledge> stopWithSavepoint​(AsynchronousJobOperationKey operationKey,
                                                        String targetDirectory,
                                                        org.apache.flink.core.execution.SavepointFormatType formatType,
                                                        TriggerSavepointMode savepointMode,
                                                        org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: RestfulGateway
        Stops the job with a savepoint, returning a future that completes when the operation is started.
        Specified by:
        stopWithSavepoint in interface RestfulGateway
        Parameters:
        operationKey - key of the operation, for deduplication
        targetDirectory - Target directory for the savepoint.
        formatType - Binary format of the savepoint.
        savepointMode - context of the savepoint operation
        timeout - for the rpc call
        Returns:
        Future which is completed once the operation is triggered successfully

      • stopWithSavepointAndGetLocation

        public CompletableFuture<String> stopWithSavepointAndGetLocation​(org.apache.flink.api.common.JobID jobId,
                                                                 String targetDirectory,
                                                                 org.apache.flink.core.execution.SavepointFormatType formatType,
                                                                 TriggerSavepointMode savepointMode,
                                                                 org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: DispatcherGateway
        Stops the job with a savepoint, returning a future that completes with the savepoint location when the savepoint is completed.
        Specified by:
        stopWithSavepointAndGetLocation in interface DispatcherGateway
        Parameters:
        jobId - the job id
        targetDirectory - Target directory for the savepoint.
        savepointMode - context of the savepoint operation
        timeout - for the rpc call
        Returns:
        Future which is completed with the savepoint location once it is completed

      • deliverCoordinationRequestToCoordinator

        public CompletableFuture<CoordinationResponse> deliverCoordinationRequestToCoordinator​(org.apache.flink.api.common.JobID jobId,
                                                                                       OperatorID operatorId,
                                                                                       org.apache.flink.util.SerializedValue<CoordinationRequest> serializedRequest,
                                                                                       org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: RestfulGateway
        Deliver a coordination request to a specified coordinator and return the response.
        Specified by:
        deliverCoordinationRequestToCoordinator in interface RestfulGateway
        Parameters:
        jobId - identifying the job which the coordinator belongs to
        operatorId - identifying the coordinator to receive the request
        serializedRequest - serialized request to deliver
        timeout - RPC timeout
        Returns:
        A future containing the response. The response will fail with a FlinkException if the task is not running, or no operator/coordinator exists for the given ID, or the coordinator cannot handle client events.

      • reportJobClientHeartbeat

        public CompletableFuture<Void> reportJobClientHeartbeat​(org.apache.flink.api.common.JobID jobId,
                                                        long expiredTimestamp,
                                                        org.apache.flink.api.common.time.Time timeout)
        Description copied from interface: RestfulGateway
        The client reports the heartbeat to the dispatcher for aliveness.
        Specified by:
        reportJobClientHeartbeat in interface RestfulGateway

      • updateJobResourceRequirements

        public CompletableFuture<Acknowledge> updateJobResourceRequirements​(org.apache.flink.api.common.JobID jobId,
                                                                    JobResourceRequirements jobResourceRequirements)
        Description copied from interface: RestfulGateway
        Update job resource requirements for a given job. When the returned future is complete the requirements have been updated and were persisted in HA, but the job may not have been rescaled (yet).
        Specified by:
        updateJobResourceRequirements in interface RestfulGateway
        Parameters:
        jobId - job the given requirements belong to
        jobResourceRequirements - new resource requirements for the job
        Returns:
        Future which is completed successfully when requirements are updated

      • runPostJobGloballyTerminated

        protected void runPostJobGloballyTerminated​(org.apache.flink.api.common.JobID jobId,
                                            org.apache.flink.api.common.JobStatus jobStatus)

      • onFatalError

        protected void onFatalError​(Throwable throwable)

      • jobReachedTerminalState

        @VisibleForTesting
protected CompletableFuture<org.apache.flink.runtime.dispatcher.Dispatcher.CleanupJobState> jobReachedTerminalState​(ExecutionGraphInfo executionGraphInfo)

      • onRemovedJobGraph

        public CompletableFuture<Void> onRemovedJobGraph​(org.apache.flink.api.common.JobID jobId)