Package org.apache.flink.contrib.streaming.state

Class RocksDBIncrementalCheckpointUtils

  • public class RocksDBIncrementalCheckpointUtils
extends Object
    Utils for RocksDB Incremental Checkpoint.

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static boolean beforeThePrefixBytes​(byte[] bytes, byte[] prefixBytes)
      check whether the bytes is before prefixBytes in the character order.
      static <T extends org.apache.flink.runtime.state.KeyedStateHandle>
      T      		 chooseTheBestStateHandleForInitial​(List<T> restoreStateHandles, org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange, double overlapFractionThreshold)
      Choose the best state handle according to the stateHandleEvaluator(KeyedStateHandle, KeyGroupRange, double) to init the initial db.
      static void clipDBWithKeyGroupRange​(org.rocksdb.RocksDB db, List<org.rocksdb.ColumnFamilyHandle> columnFamilyHandles, org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange, org.apache.flink.runtime.state.KeyGroupRange currentKeyGroupRange, int keyGroupPrefixBytes, boolean useDeleteFilesInRange)
      The method to clip the db instance according to the target key group range using the RocksDB.delete(ColumnFamilyHandle, byte[]).
      static Optional<org.apache.flink.util.function.RunnableWithException> createRangeCompactionTaskIfNeeded​(org.rocksdb.RocksDB db, Collection<org.rocksdb.ColumnFamilyHandle> columnFamilyHandles, int keyGroupPrefixBytes, org.apache.flink.runtime.state.KeyGroupRange dbExpectedKeyGroupRange)
      Returns a range compaction task as runnable if any data in the SST files of the given DB exceeds the proclaimed key-group range.
      static <T extends org.apache.flink.runtime.state.KeyedStateHandle>
      int      		 findTheBestStateHandleForInitial​(List<T> restoreStateHandles, org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange, double overlapFractionThreshold)
      Choose the best state handle according to the stateHandleEvaluator(KeyedStateHandle, KeyGroupRange, double) to init the initial db from the given lists and returns its index.
      static boolean isSstDataInKeyGroupRange​(org.rocksdb.RocksDB db, int keyGroupPrefixBytes, org.apache.flink.runtime.state.KeyGroupRange dbExpectedKeyGroupRange)
      Returns true, if all entries in the sst files of the given DB is strictly within the expected key-group range for the DB.

    • Constructor Detail

      • RocksDBIncrementalCheckpointUtils

        public RocksDBIncrementalCheckpointUtils()

    • Method Detail

      • clipDBWithKeyGroupRange

        public static void clipDBWithKeyGroupRange​(@Nonnull
                                           org.rocksdb.RocksDB db,
                                           @Nonnull
                                           List<org.rocksdb.ColumnFamilyHandle> columnFamilyHandles,
                                           @Nonnull
                                           org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange,
                                           @Nonnull
                                           org.apache.flink.runtime.state.KeyGroupRange currentKeyGroupRange,
                                           @Nonnegative
                                           int keyGroupPrefixBytes,
                                           boolean useDeleteFilesInRange)
                                    throws org.rocksdb.RocksDBException
        The method to clip the db instance according to the target key group range using the RocksDB.delete(ColumnFamilyHandle, byte[]).
        Parameters:
        db - the RocksDB instance to be clipped.
        columnFamilyHandles - the column families in the db instance.
        targetKeyGroupRange - the target key group range.
        currentKeyGroupRange - the key group range of the db instance.
        keyGroupPrefixBytes - Number of bytes required to prefix the key groups.
        useDeleteFilesInRange - whether to call db.deleteFilesInRanges for the deleted ranges.
        Throws:
        org.rocksdb.RocksDBException

      • isSstDataInKeyGroupRange

        public static boolean isSstDataInKeyGroupRange​(org.rocksdb.RocksDB db,
                                               int keyGroupPrefixBytes,
                                               org.apache.flink.runtime.state.KeyGroupRange dbExpectedKeyGroupRange)
        Returns true, if all entries in the sst files of the given DB is strictly within the expected key-group range for the DB.
        Parameters:
        db - the DB to check.
        dbExpectedKeyGroupRange - the expected key-groups range of the DB.
        keyGroupPrefixBytes - the number of bytes used to serialize the key-group prefix of keys in the DB.

      • createRangeCompactionTaskIfNeeded

        public static Optional<org.apache.flink.util.function.RunnableWithException> createRangeCompactionTaskIfNeeded​(org.rocksdb.RocksDB db,
                                                                                                               Collection<org.rocksdb.ColumnFamilyHandle> columnFamilyHandles,
                                                                                                               int keyGroupPrefixBytes,
                                                                                                               org.apache.flink.runtime.state.KeyGroupRange dbExpectedKeyGroupRange)
        Returns a range compaction task as runnable if any data in the SST files of the given DB exceeds the proclaimed key-group range.
        Parameters:
        db - the DB to check and compact if needed.
        columnFamilyHandles - list of column families to check.
        keyGroupPrefixBytes - the number of bytes used to serialize the key-group prefix of keys in the DB.
        dbExpectedKeyGroupRange - the expected key-groups range of the DB.
        Returns:
        runnable that performs compaction upon execution if the key-groups range is exceeded. Otherwise, empty optional is returned.

      • beforeThePrefixBytes

        public static boolean beforeThePrefixBytes​(@Nonnull
                                           byte[] bytes,
                                           @Nonnull
                                           byte[] prefixBytes)
        check whether the bytes is before prefixBytes in the character order.

      • chooseTheBestStateHandleForInitial

        @Nullable
public static <T extends org.apache.flink.runtime.state.KeyedStateHandle> T chooseTheBestStateHandleForInitial​(@Nonnull
                                                                                                               List<T> restoreStateHandles,
                                                                                                               @Nonnull
                                                                                                               org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange,
                                                                                                               double overlapFractionThreshold)
        Choose the best state handle according to the stateHandleEvaluator(KeyedStateHandle, KeyGroupRange, double) to init the initial db.
        Type Parameters:
        T - the generic parameter type of the state handles.
        Parameters:
        restoreStateHandles - The candidate state handles.
        targetKeyGroupRange - The target key group range.
        overlapFractionThreshold - configured threshold for overlap.
        Returns:
        The best candidate or null if no candidate was a good fit.

      • findTheBestStateHandleForInitial

        public static <T extends org.apache.flink.runtime.state.KeyedStateHandle> int findTheBestStateHandleForInitial​(@Nonnull
                                                                                                               List<T> restoreStateHandles,
                                                                                                               @Nonnull
                                                                                                               org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange,
                                                                                                               double overlapFractionThreshold)
        Choose the best state handle according to the stateHandleEvaluator(KeyedStateHandle, KeyGroupRange, double) to init the initial db from the given lists and returns its index.
        Type Parameters:
        T - the generic parameter type of the state handles.
        Parameters:
        restoreStateHandles - The candidate state handles.
        targetKeyGroupRange - The target key group range.
        overlapFractionThreshold - configured threshold for overlap.
        Returns:
        the index of the best candidate handle in the list or -1 if the list was empty.