Package org.apache.flink.runtime.scheduler.adaptivebatch

Class AdaptiveBatchScheduler

java.lang.Object

org.apache.flink.runtime.scheduler.SchedulerBase

org.apache.flink.runtime.scheduler.DefaultScheduler

org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler

All Implemented Interfaces:

AutoCloseable, CheckpointScheduling, GlobalFailureHandler, SchedulerNG, SchedulerOperations, org.apache.flink.util.AutoCloseableAsync

Direct Known Subclasses:

SpeculativeScheduler

public class AdaptiveBatchScheduler
extends DefaultScheduler

This scheduler decides the parallelism of JobVertex according to the data volume it consumes. A dynamically built up ExecutionGraph is used for this purpose.

Field Summary

Fields inherited from class org.apache.flink.runtime.scheduler.DefaultScheduler

executionDeployer, executionSlotAllocator, log

Fields inherited from class org.apache.flink.runtime.scheduler.SchedulerBase

executionVertexVersioner, inputsLocationsRetriever, jobInfo, jobManagerJobMetricGroup, operatorCoordinatorHandler, stateLocationRetriever

Constructor Summary

Constructors

Constructor

Description

AdaptiveBatchScheduler(org.slf4j.Logger log, JobGraph jobGraph, Executor ioExecutor, org.apache.flink.configuration.Configuration jobMasterConfiguration, java.util.function.Consumer<org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor> startUpAction, org.apache.flink.util.concurrent.ScheduledExecutor delayExecutor, ClassLoader userCodeLoader, CheckpointsCleaner checkpointsCleaner, CheckpointRecoveryFactory checkpointRecoveryFactory, JobManagerJobMetricGroup jobManagerJobMetricGroup, SchedulingStrategyFactory schedulingStrategyFactory, FailoverStrategy.Factory failoverStrategyFactory, RestartBackoffTimeStrategy restartBackoffTimeStrategy, ExecutionOperations executionOperations, ExecutionVertexVersioner executionVertexVersioner, ExecutionSlotAllocatorFactory executionSlotAllocatorFactory, long initializationTimestamp, org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor, JobStatusListener jobStatusListener, Collection<org.apache.flink.core.failure.FailureEnricher> failureEnrichers, ExecutionGraphFactory executionGraphFactory, ShuffleMaster<?> shuffleMaster, org.apache.flink.api.common.time.Time rpcTimeout, VertexParallelismAndInputInfosDecider vertexParallelismAndInputInfosDecider, int defaultMaxParallelism, org.apache.flink.configuration.JobManagerOptions.HybridPartitionDataConsumeConstraint hybridPartitionDataConsumeConstraint, Map<JobVertexID,ForwardGroup> forwardGroupsByJobVertexId)

Method Summary

Modifier and Type	Method	Description
void	allocateSlotsAndDeploy(List<ExecutionVertexID> verticesToDeploy)	Allocate slots and deploy the vertex when slots are returned.
List<CompletableFuture<Integer>>	computeDynamicSourceParallelism()
static VertexParallelismStore	computeVertexParallelismStoreForDynamicGraph(Iterable<JobVertex> vertices, int defaultMaxParallelism)	Compute the VertexParallelismStore for all given vertices in a dynamic graph, which will set defaults and ensure that the returned store contains valid parallelisms, with the configured default max parallelism.
protected MarkPartitionFinishedStrategy	getMarkPartitionFinishedStrategy()
void	initializeVerticesIfPossible()
protected void	onTaskFinished(Execution execution, IOMetrics ioMetrics)
protected void	resetForNewExecution(ExecutionVertexID executionVertexId)
protected void	startSchedulingInternal()

Methods inherited from class org.apache.flink.runtime.scheduler.DefaultScheduler

cancelAllPendingSlotRequestsForVertex, cancelAllPendingSlotRequestsInternal, cancelExecution, createFailureHandlingResultSnapshot, getNumberOfRestarts, handleGlobalFailure, handleTaskFailure, onTaskFailed, recordTaskFailure

Methods inherited from class org.apache.flink.runtime.scheduler.SchedulerBase

acknowledgeCheckpoint, archiveFromFailureHandlingResult, archiveGlobalFailure, cancel, closeAsync, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, declineCheckpoint, deliverCoordinationRequestToCoordinator, deliverOperatorEventToCoordinator, failJob, getDefaultMaxParallelism, getExceptionHistory, getExecutionGraph, getExecutionJobVertex, getExecutionVertex, getJobGraph, getJobTerminationFuture, getMainThreadExecutor, getResultPartitionAvailabilityChecker, getSchedulingTopology, notifyEndOfData, notifyKvStateRegistered, notifyKvStateUnregistered, registerJobMetrics, reportCheckpointMetrics, reportInitializationMetrics, requestCheckpointStats, requestJob, requestJobStatus, requestKvStateLocation, requestNextInputSplit, requestPartitionState, resetForNewExecutions, restoreState, setGlobalFailureCause, startCheckpointScheduler, startScheduling, stopCheckpointScheduler, stopWithSavepoint, transitionExecutionGraphState, transitionToRunning, transitionToScheduled, triggerCheckpoint, triggerSavepoint, updateAccumulators, updateTaskExecutionState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Methods inherited from interface org.apache.flink.runtime.scheduler.SchedulerNG

requestJobResourceRequirements, updateJobResourceRequirements, updateTaskExecutionState

Constructor Detail

AdaptiveBatchScheduler

public AdaptiveBatchScheduler(org.slf4j.Logger log,
                              JobGraph jobGraph,
                              Executor ioExecutor,
                              org.apache.flink.configuration.Configuration jobMasterConfiguration,
                              java.util.function.Consumer<org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor> startUpAction,
                              org.apache.flink.util.concurrent.ScheduledExecutor delayExecutor,
                              ClassLoader userCodeLoader,
                              CheckpointsCleaner checkpointsCleaner,
                              CheckpointRecoveryFactory checkpointRecoveryFactory,
                              JobManagerJobMetricGroup jobManagerJobMetricGroup,
                              SchedulingStrategyFactory schedulingStrategyFactory,
                              FailoverStrategy.Factory failoverStrategyFactory,
                              RestartBackoffTimeStrategy restartBackoffTimeStrategy,
                              ExecutionOperations executionOperations,
                              ExecutionVertexVersioner executionVertexVersioner,
                              ExecutionSlotAllocatorFactory executionSlotAllocatorFactory,
                              long initializationTimestamp,
                              org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor,
                              JobStatusListener jobStatusListener,
                              Collection<org.apache.flink.core.failure.FailureEnricher> failureEnrichers,
                              ExecutionGraphFactory executionGraphFactory,
                              ShuffleMaster<?> shuffleMaster,
                              org.apache.flink.api.common.time.Time rpcTimeout,
                              VertexParallelismAndInputInfosDecider vertexParallelismAndInputInfosDecider,
                              int defaultMaxParallelism,
                              org.apache.flink.configuration.JobManagerOptions.HybridPartitionDataConsumeConstraint hybridPartitionDataConsumeConstraint,
                              Map<JobVertexID,ForwardGroup> forwardGroupsByJobVertexId)
                       throws Exception

Throws:

Exception

Method Detail

startSchedulingInternal

protected void startSchedulingInternal()

Overrides:

startSchedulingInternal in class DefaultScheduler

onTaskFinished

protected void onTaskFinished(Execution execution,
                              IOMetrics ioMetrics)

Overrides:

onTaskFinished in class DefaultScheduler

allocateSlotsAndDeploy

public void allocateSlotsAndDeploy(List<ExecutionVertexID> verticesToDeploy)

Description copied from interface: SchedulerOperations

Allocate slots and deploy the vertex when slots are returned. Vertices will be deployed only after all of them have been assigned slots. The given order will be respected, i.e. tasks with smaller indices will be deployed earlier. Only vertices in CREATED state will be accepted. Errors will happen if scheduling Non-CREATED vertices.

Specified by:

allocateSlotsAndDeploy in interface SchedulerOperations

Overrides:

allocateSlotsAndDeploy in class DefaultScheduler

Parameters:

verticesToDeploy - The execution vertices to deploy

resetForNewExecution

protected void resetForNewExecution(ExecutionVertexID executionVertexId)

Overrides:

resetForNewExecution in class SchedulerBase

getMarkPartitionFinishedStrategy

protected MarkPartitionFinishedStrategy getMarkPartitionFinishedStrategy()

Overrides:

getMarkPartitionFinishedStrategy in class SchedulerBase

computeDynamicSourceParallelism

public List<CompletableFuture<Integer>> computeDynamicSourceParallelism()

initializeVerticesIfPossible

@VisibleForTesting
public void initializeVerticesIfPossible()

computeVertexParallelismStoreForDynamicGraph

@VisibleForTesting
public static VertexParallelismStore computeVertexParallelismStoreForDynamicGraph(Iterable<JobVertex> vertices,
                                                                                  int defaultMaxParallelism)

Compute the VertexParallelismStore for all given vertices in a dynamic graph, which will set defaults and ensure that the returned store contains valid parallelisms, with the configured default max parallelism.

Parameters:

vertices - the vertices to compute parallelism for

defaultMaxParallelism - the global default max parallelism

Returns:

the computed parallelism store