Package org.apache.flink.runtime.io.network.partition.hybrid

Interface HsSpillingStrategy

All Known Implementing Classes:

HsFullSpillingStrategy, HsSelectiveSpillingStrategy

public interface HsSpillingStrategy

Spilling strategy for hybrid shuffle mode.

Note: decideActionWithGlobalInfo(org.apache.flink.runtime.io.network.partition.hybrid.HsSpillingInfoProvider) is usually expensive, in the sense of both the computation complexity of the strategy algorithm and the synchronization overhead for providing the global information. Thus, it should only be called when global information is needed.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	HsSpillingStrategy.Decision	This class represents the spill and release decision made by HsSpillingStrategy, in other words, which data is to be spilled and which data is to be released.

Method Summary

Modifier and Type	Method	Description
HsSpillingStrategy.Decision	decideActionWithGlobalInfo(HsSpillingInfoProvider spillingInfoProvider)	Make a decision based on global information.
Optional<HsSpillingStrategy.Decision>	onBufferConsumed(BufferIndexAndChannel consumedBuffer)	Make a decision when a buffer is consumed.
Optional<HsSpillingStrategy.Decision>	onBufferFinished(int numTotalUnSpillBuffers, int currentPoolSize)	Make a decision when a buffer becomes finished.
Optional<HsSpillingStrategy.Decision>	onMemoryUsageChanged(int numTotalRequestedBuffers, int currentPoolSize)	Make a decision when memory usage is changed.
HsSpillingStrategy.Decision	onResultPartitionClosed(HsSpillingInfoProvider spillingInfoProvider)	Make a decision when result partition is closed.

Method Detail

onMemoryUsageChanged

Optional<HsSpillingStrategy.Decision> onMemoryUsageChanged(int numTotalRequestedBuffers,
                                                           int currentPoolSize)

Make a decision when memory usage is changed.

Parameters:

numTotalRequestedBuffers - total number of buffers requested.

currentPoolSize - current value of buffer pool size.

Returns:

A HsSpillingStrategy.Decision based on the provided information, or Optional.empty() if the decision cannot be made, which indicates global information is needed.

onBufferFinished

Optional<HsSpillingStrategy.Decision> onBufferFinished(int numTotalUnSpillBuffers,
                                                       int currentPoolSize)

Make a decision when a buffer becomes finished.

Parameters:

numTotalUnSpillBuffers - total number of buffers not spill.

Returns:

A HsSpillingStrategy.Decision based on the provided information, or Optional.empty() if the decision cannot be made, which indicates global information is needed.

onBufferConsumed

Optional<HsSpillingStrategy.Decision> onBufferConsumed(BufferIndexAndChannel consumedBuffer)

Make a decision when a buffer is consumed.

Parameters:

consumedBuffer - the buffer that is consumed.

Returns:

A HsSpillingStrategy.Decision based on the provided information, or Optional.empty() if the decision cannot be made, which indicates global information is needed.

decideActionWithGlobalInfo

HsSpillingStrategy.Decision decideActionWithGlobalInfo(HsSpillingInfoProvider spillingInfoProvider)

Make a decision based on global information. Because this method will directly touch the HsSpillingInfoProvider, the caller should take care of the thread safety.

Parameters:

spillingInfoProvider - that provides information about the current status.

Returns:

A HsSpillingStrategy.Decision based on the global information.

onResultPartitionClosed

HsSpillingStrategy.Decision onResultPartitionClosed(HsSpillingInfoProvider spillingInfoProvider)

Make a decision when result partition is closed. Because this method will directly touch the HsSpillingInfoProvider, the caller should take care of the thread safety.

Parameters:

spillingInfoProvider - that provides information about the current status.

Returns:

A HsSpillingStrategy.Decision based on the global information.