Package org.apache.flink.state.forst

Class ForStMemoryControllerUtils

java.lang.Object

org.apache.flink.state.forst.ForStMemoryControllerUtils

public class ForStMemoryControllerUtils
extends Object

Utils to create Cache and WriteBufferManager which are used to control total memory usage of ForSt.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	ForStMemoryControllerUtils.ForStMemoryFactory	Factory for Write Buffer Manager and Bock Cache.

Constructor Summary

Constructors

Constructor	Description
ForStMemoryControllerUtils()

Method Summary

Modifier and Type	Method	Description
static org.apache.flink.state.forst.ForStSharedResources	allocateForStSharedResources(long totalMemorySize, double writeBufferRatio, double highPriorityPoolRatio, boolean usingPartitionedIndexFilters, ForStMemoryControllerUtils.ForStMemoryFactory factory)	Allocate memory controllable ForSt shared resources.
static long	calculateActualCacheCapacity(long totalMemorySize, double writeBufferRatio)	Calculate the actual memory capacity of cache, which would be shared among ForSt instance(s).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ForStMemoryControllerUtils

public ForStMemoryControllerUtils()

Method Detail

allocateForStSharedResources

public static org.apache.flink.state.forst.ForStSharedResources allocateForStSharedResources(long totalMemorySize,
                                                                                             double writeBufferRatio,
                                                                                             double highPriorityPoolRatio,
                                                                                             boolean usingPartitionedIndexFilters,
                                                                                             ForStMemoryControllerUtils.ForStMemoryFactory factory)

Allocate memory controllable ForSt shared resources.

Parameters:

totalMemorySize - The total memory limit size.

writeBufferRatio - The ratio of total memory which is occupied by write buffer manager.

highPriorityPoolRatio - The high priority pool ratio of cache.

factory - creates Write Buffer Manager and Bock Cache

Returns:

memory controllable RocksDB shared resources.

calculateActualCacheCapacity

@VisibleForTesting
public static long calculateActualCacheCapacity(long totalMemorySize,
                                                double writeBufferRatio)

Calculate the actual memory capacity of cache, which would be shared among ForSt instance(s). We introduce this method because: a) We cannot create a strict capacity limit cache util FLINK-15532 resolved. b) Regardless of the memory usage of blocks pinned by ForSt iterators, which is difficult to calculate and only happened when we iterator entries in MapState, the overuse of memory is mainly occupied by at most half of the write buffer usage. (see the flush implementation of write buffer manager). Thus, we have four equations below: write_buffer_manager_memory = 1.5 * write_buffer_manager_capacity write_buffer_manager_memory = total_memory_size * write_buffer_ratio write_buffer_manager_memory + other_part = total_memory_size write_buffer_manager_capacity + other_part = cache_capacity And we would deduce the formula: cache_capacity = (3 - write_buffer_ratio) * total_memory_size / 3 write_buffer_manager_capacity = 2 * total_memory_size * write_buffer_ratio / 3

Parameters:

totalMemorySize - Total off-heap memory size reserved for ForSt instance(s).

writeBufferRatio - The ratio of total memory size which would be reserved for write buffer manager and its over-capacity part.

Returns:

The actual calculated cache capacity.