Package org.apache.flink.runtime.io.network.partition.hybrid.tiered.common

Class TieredStorageConfiguration

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageConfiguration

public class TieredStorageConfiguration
extends Object

Configurations for the Tiered Storage.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	TieredStorageConfiguration.Builder	The builder for TieredStorageConfiguration.

Constructor Summary

Constructors

Constructor

Description

TieredStorageConfiguration(String remoteStorageBasePath, int tieredStorageBufferSize, int memoryTierExclusiveBuffers, int diskTierExclusiveBuffers, int remoteTierExclusiveBuffers, int accumulatorExclusiveBuffers, int memoryTierNumBytesPerSegment, int diskTierNumBytesPerSegment, int remoteTierNumBytesPerSegment, float numBuffersTriggerFlushRatio, int diskIOSchedulerMaxBuffersReadAhead, java.time.Duration diskIOSchedulerRequestTimeout, float minReserveDiskSpaceFraction, List<TierFactory> tierFactories, List<Integer> tierExclusiveBuffers)

Method Summary

Modifier and Type	Method	Description
static TieredStorageConfiguration.Builder	builder(int tieredStorageBufferSize, String remoteStorageBasePath)
static TieredStorageConfiguration.Builder	builder(String remoteStorageBasePath)
int	getAccumulatorExclusiveBuffers()	Get exclusive buffer number of accumulator.
java.time.Duration	getDiskIOSchedulerBufferRequestTimeout()	Maximum time to wait when requesting read buffers from the buffer pool before throwing an exception in DiskIOScheduler.
int	getDiskIOSchedulerMaxBuffersReadAhead()	The number of buffers to read ahead at most for each subpartition in DiskIOScheduler, which can be used to prevent other consumers from starving.
int	getDiskTierExclusiveBuffers()	Get exclusive buffer number of disk tier.
int	getDiskTierNumBytesPerSegment()	Get the segment size of disk tier.
List<Integer>	getEachTierExclusiveBufferNum()	Get exclusive buffer number of each tier.
int	getMemoryTierExclusiveBuffers()	Get exclusive buffer number of memory tier.
int	getMemoryTierNumBytesPerSegment()	Get the segment size of memory tier.
float	getMinReserveDiskSpaceFraction()	Minimum reserved disk space fraction in disk tier.
float	getNumBuffersTriggerFlushRatio()	When the number of buffers that have been requested exceeds this threshold, trigger the flushing operation in each TierProducerAgent.
String	getRemoteStorageBasePath()	Get the base path on remote storage.
int	getRemoteTierExclusiveBuffers()	Get exclusive buffer number of remote tier.
int	getRemoteTierNumBytesPerSegment()	Get the segment size of remote tier.
int	getTieredStorageBufferSize()	Get the buffer size in tiered storage.
List<TierFactory>	getTierFactories()
int	getTotalExclusiveBufferNum()	Get the total exclusive buffer number.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TieredStorageConfiguration

public TieredStorageConfiguration(String remoteStorageBasePath,
                                  int tieredStorageBufferSize,
                                  int memoryTierExclusiveBuffers,
                                  int diskTierExclusiveBuffers,
                                  int remoteTierExclusiveBuffers,
                                  int accumulatorExclusiveBuffers,
                                  int memoryTierNumBytesPerSegment,
                                  int diskTierNumBytesPerSegment,
                                  int remoteTierNumBytesPerSegment,
                                  float numBuffersTriggerFlushRatio,
                                  int diskIOSchedulerMaxBuffersReadAhead,
                                  java.time.Duration diskIOSchedulerRequestTimeout,
                                  float minReserveDiskSpaceFraction,
                                  List<TierFactory> tierFactories,
                                  List<Integer> tierExclusiveBuffers)

Method Detail

builder

public static TieredStorageConfiguration.Builder builder(String remoteStorageBasePath)

builder

public static TieredStorageConfiguration.Builder builder(int tieredStorageBufferSize,
                                                         String remoteStorageBasePath)

getRemoteStorageBasePath

public String getRemoteStorageBasePath()

Get the base path on remote storage.

Returns:

string if the remote storage path is configured otherwise null.

getTieredStorageBufferSize

public int getTieredStorageBufferSize()

Get the buffer size in tiered storage.

Returns:

the buffer size.

getMemoryTierExclusiveBuffers

public int getMemoryTierExclusiveBuffers()

Get exclusive buffer number of memory tier.

Returns:

the buffer number.

getDiskTierExclusiveBuffers

public int getDiskTierExclusiveBuffers()

Get exclusive buffer number of disk tier.

Returns:

the buffer number.

getRemoteTierExclusiveBuffers

public int getRemoteTierExclusiveBuffers()

Get exclusive buffer number of remote tier.

Returns:

the buffer number.

getAccumulatorExclusiveBuffers

public int getAccumulatorExclusiveBuffers()

Get exclusive buffer number of accumulator.

The buffer number is used to compare with the subpartition number to determine the type of BufferAccumulator.

If the exclusive buffer number is larger than (subpartitionNum + 1), the accumulator will use HashBufferAccumulator. If the exclusive buffer number is equal to or smaller than (subpartitionNum + 1), the accumulator will use SortBufferAccumulator

Returns:

the buffer number.

getMemoryTierNumBytesPerSegment

public int getMemoryTierNumBytesPerSegment()

Get the segment size of memory tier.

Returns:

segment size.

getDiskTierNumBytesPerSegment

public int getDiskTierNumBytesPerSegment()

Get the segment size of disk tier.

Returns:

segment size.

getRemoteTierNumBytesPerSegment

public int getRemoteTierNumBytesPerSegment()

Get the segment size of remote tier.

Returns:

segment size.

getNumBuffersTriggerFlushRatio

public float getNumBuffersTriggerFlushRatio()

When the number of buffers that have been requested exceeds this threshold, trigger the flushing operation in each TierProducerAgent.

Returns:

flush ratio.

getDiskIOSchedulerMaxBuffersReadAhead

public int getDiskIOSchedulerMaxBuffersReadAhead()

The number of buffers to read ahead at most for each subpartition in DiskIOScheduler, which can be used to prevent other consumers from starving.

Returns:

buffer number.

getDiskIOSchedulerBufferRequestTimeout

public java.time.Duration getDiskIOSchedulerBufferRequestTimeout()

Maximum time to wait when requesting read buffers from the buffer pool before throwing an exception in DiskIOScheduler.

Returns:

timeout duration.

getMinReserveDiskSpaceFraction

public float getMinReserveDiskSpaceFraction()

Minimum reserved disk space fraction in disk tier.

Returns:

the fraction.

getTotalExclusiveBufferNum

public int getTotalExclusiveBufferNum()

Get the total exclusive buffer number.

Returns:

the total exclusive buffer number.

getEachTierExclusiveBufferNum

public List<Integer> getEachTierExclusiveBufferNum()

Get exclusive buffer number of each tier.

Returns:

buffer number of each tier.

getTierFactories

public List<TierFactory> getTierFactories()