Package org.apache.flink.runtime.scheduler.adaptivebatch

Class SpeculativeScheduler

java.lang.Object

org.apache.flink.runtime.scheduler.SchedulerBase

org.apache.flink.runtime.scheduler.DefaultScheduler

org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler

org.apache.flink.runtime.scheduler.adaptivebatch.SpeculativeScheduler

All Implemented Interfaces:

AutoCloseable, CheckpointScheduling, GlobalFailureHandler, SchedulerNG, SchedulerOperations, SlowTaskDetectorListener, org.apache.flink.util.AutoCloseableAsync

public class SpeculativeScheduler
extends AdaptiveBatchScheduler
implements SlowTaskDetectorListener

The speculative scheduler.

Field Summary

Fields inherited from class org.apache.flink.runtime.scheduler.DefaultScheduler

executionDeployer, executionSlotAllocator, log

Fields inherited from class org.apache.flink.runtime.scheduler.SchedulerBase

executionVertexVersioner, inputsLocationsRetriever, jobManagerJobMetricGroup, operatorCoordinatorHandler, stateLocationRetriever

Constructor Summary

Constructors

Constructor

Description

SpeculativeScheduler(org.slf4j.Logger log, JobGraph jobGraph, Executor ioExecutor, org.apache.flink.configuration.Configuration jobMasterConfiguration, java.util.function.Consumer<org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor> startUpAction, org.apache.flink.util.concurrent.ScheduledExecutor delayExecutor, ClassLoader userCodeLoader, CheckpointsCleaner checkpointsCleaner, CheckpointRecoveryFactory checkpointRecoveryFactory, JobManagerJobMetricGroup jobManagerJobMetricGroup, SchedulingStrategyFactory schedulingStrategyFactory, FailoverStrategy.Factory failoverStrategyFactory, RestartBackoffTimeStrategy restartBackoffTimeStrategy, ExecutionOperations executionOperations, ExecutionVertexVersioner executionVertexVersioner, ExecutionSlotAllocatorFactory executionSlotAllocatorFactory, long initializationTimestamp, org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor, JobStatusListener jobStatusListener, Collection<org.apache.flink.core.failure.FailureEnricher> failureEnrichers, ExecutionGraphFactory executionGraphFactory, ShuffleMaster<?> shuffleMaster, org.apache.flink.api.common.time.Time rpcTimeout, VertexParallelismAndInputInfosDecider vertexParallelismAndInputInfosDecider, int defaultMaxParallelism, BlocklistOperations blocklistOperations, org.apache.flink.configuration.JobManagerOptions.HybridPartitionDataConsumeConstraint hybridPartitionDataConsumeConstraint, Map<JobVertexID,ForwardGroup> forwardGroupsByJobVertexId)

Method Summary

Modifier and Type	Method	Description
CompletableFuture<Void>	closeAsync()
SpeculativeExecutionVertex	getExecutionVertex(ExecutionVertexID executionVertexId)
protected void	handleTaskFailure(Execution failedExecution, Throwable error)
void	notifySlowTasks(Map<ExecutionVertexID,Collection<ExecutionAttemptID>> slowTasks)	Notify detected slow tasks.
protected void	onTaskFailed(Execution execution)
protected void	onTaskFinished(Execution execution, IOMetrics ioMetrics)
protected void	resetForNewExecution(ExecutionVertexID executionVertexId)
protected void	startSchedulingInternal()

Methods inherited from class org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler

allocateSlotsAndDeploy, computeVertexParallelismStoreForDynamicGraph, getMarkPartitionFinishedStrategy, initializeVerticesIfPossible

Methods inherited from class org.apache.flink.runtime.scheduler.DefaultScheduler

cancelAllPendingSlotRequestsForVertex, cancelAllPendingSlotRequestsInternal, cancelExecution, createFailureHandlingResultSnapshot, getNumberOfRestarts, handleGlobalFailure, recordTaskFailure

Methods inherited from class org.apache.flink.runtime.scheduler.SchedulerBase

acknowledgeCheckpoint, archiveFromFailureHandlingResult, archiveGlobalFailure, cancel, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, declineCheckpoint, deliverCoordinationRequestToCoordinator, deliverOperatorEventToCoordinator, failJob, getDefaultMaxParallelism, getExceptionHistory, getExecutionGraph, getExecutionJobVertex, getJobGraph, getJobTerminationFuture, getMainThreadExecutor, getResultPartitionAvailabilityChecker, getSchedulingTopology, notifyEndOfData, notifyKvStateRegistered, notifyKvStateUnregistered, registerJobMetrics, reportCheckpointMetrics, requestCheckpointStats, requestJob, requestJobDetails, requestJobStatus, requestKvStateLocation, requestNextInputSplit, requestPartitionState, resetForNewExecutions, restoreState, setGlobalFailureCause, startCheckpointScheduler, startScheduling, stopCheckpointScheduler, stopWithSavepoint, transitionExecutionGraphState, transitionToRunning, transitionToScheduled, triggerCheckpoint, triggerSavepoint, updateAccumulators, updateTaskExecutionState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Methods inherited from interface org.apache.flink.runtime.scheduler.SchedulerNG

requestJobResourceRequirements, updateJobResourceRequirements, updateTaskExecutionState

Constructor Detail

SpeculativeScheduler

public SpeculativeScheduler(org.slf4j.Logger log,
                            JobGraph jobGraph,
                            Executor ioExecutor,
                            org.apache.flink.configuration.Configuration jobMasterConfiguration,
                            java.util.function.Consumer<org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor> startUpAction,
                            org.apache.flink.util.concurrent.ScheduledExecutor delayExecutor,
                            ClassLoader userCodeLoader,
                            CheckpointsCleaner checkpointsCleaner,
                            CheckpointRecoveryFactory checkpointRecoveryFactory,
                            JobManagerJobMetricGroup jobManagerJobMetricGroup,
                            SchedulingStrategyFactory schedulingStrategyFactory,
                            FailoverStrategy.Factory failoverStrategyFactory,
                            RestartBackoffTimeStrategy restartBackoffTimeStrategy,
                            ExecutionOperations executionOperations,
                            ExecutionVertexVersioner executionVertexVersioner,
                            ExecutionSlotAllocatorFactory executionSlotAllocatorFactory,
                            long initializationTimestamp,
                            org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor,
                            JobStatusListener jobStatusListener,
                            Collection<org.apache.flink.core.failure.FailureEnricher> failureEnrichers,
                            ExecutionGraphFactory executionGraphFactory,
                            ShuffleMaster<?> shuffleMaster,
                            org.apache.flink.api.common.time.Time rpcTimeout,
                            VertexParallelismAndInputInfosDecider vertexParallelismAndInputInfosDecider,
                            int defaultMaxParallelism,
                            BlocklistOperations blocklistOperations,
                            org.apache.flink.configuration.JobManagerOptions.HybridPartitionDataConsumeConstraint hybridPartitionDataConsumeConstraint,
                            Map<JobVertexID,ForwardGroup> forwardGroupsByJobVertexId)
                     throws Exception

Throws:

Exception

Method Detail

startSchedulingInternal

protected void startSchedulingInternal()

Overrides:

startSchedulingInternal in class AdaptiveBatchScheduler

closeAsync

public CompletableFuture<Void> closeAsync()

Specified by:

closeAsync in interface org.apache.flink.util.AutoCloseableAsync

Overrides:

closeAsync in class SchedulerBase

getExecutionVertex

public SpeculativeExecutionVertex getExecutionVertex(ExecutionVertexID executionVertexId)

Overrides:

getExecutionVertex in class SchedulerBase

onTaskFinished

protected void onTaskFinished(Execution execution,
                              IOMetrics ioMetrics)

Overrides:

onTaskFinished in class AdaptiveBatchScheduler

onTaskFailed

protected void onTaskFailed(Execution execution)

Overrides:

onTaskFailed in class DefaultScheduler

handleTaskFailure

protected void handleTaskFailure(Execution failedExecution,
                                 @Nullable
                                 Throwable error)

Overrides:

handleTaskFailure in class DefaultScheduler

resetForNewExecution

protected void resetForNewExecution(ExecutionVertexID executionVertexId)

Overrides:

resetForNewExecution in class AdaptiveBatchScheduler

notifySlowTasks

public void notifySlowTasks(Map<ExecutionVertexID,Collection<ExecutionAttemptID>> slowTasks)

Description copied from interface: SlowTaskDetectorListener

Notify detected slow tasks.

Specified by:

notifySlowTasks in interface SlowTaskDetectorListener

Parameters:

slowTasks - the map of execution vertices and their execution attempts which are detected as slow.