Package org.apache.flink.state.forst.restore

Class ForStIncrementalRestoreOperation<K>

java.lang.Object

org.apache.flink.state.forst.restore.ForStIncrementalRestoreOperation<K>

All Implemented Interfaces:

AutoCloseable, org.apache.flink.runtime.state.RestoreOperation<ForStRestoreResult>, ForStRestoreOperation

public class ForStIncrementalRestoreOperation<K>
extends Object
implements ForStRestoreOperation

Encapsulates the process of restoring a ForSt instance from an incremental snapshot.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ForStIncrementalRestoreOperation.RestoredDBInstance	Entity to hold the temporary RocksDB instance created for restore.

Constructor Summary

Constructors

Constructor

Description

ForStIncrementalRestoreOperation(String operatorIdentifier, org.apache.flink.runtime.state.KeyGroupRange keyGroupRange, int keyGroupPrefixBytes, org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry, ClassLoader userCodeClassLoader, Map<String,ForStOperationUtils.ForStKvStateInfo> kvStateInformation, org.apache.flink.runtime.state.StateSerializerProvider<K> keySerializerProvider, ForStResourceContainer optionsContainer, org.apache.flink.core.fs.Path forstBasePath, org.apache.flink.core.fs.Path instanceRocksDBPath, org.forstdb.DBOptions dbOptions, Function<String,org.forstdb.ColumnFamilyOptions> columnFamilyOptionsFactory, ForStNativeMetricOptions nativeMetricOptions, org.apache.flink.metrics.MetricGroup metricGroup, ForStDBTtlCompactFiltersManager ttlCompactFiltersManager, long writeBatchSize, Long writeBufferManagerCapacity, org.apache.flink.runtime.state.StateBackend.CustomInitializationMetrics customInitializationMetrics, Collection<org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle> restoreStateHandles, double overlapFractionThreshold, boolean useIngestDbRestoreMode, boolean useDeleteFilesInRange, org.apache.flink.core.execution.RecoveryClaimMode recoveryClaimMode)

Method Summary

Modifier and Type	Method	Description
void	close()
void	exportColumnFamilies(org.forstdb.RocksDB db, List<org.forstdb.ColumnFamilyHandle> columnFamilyHandles, List<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase> registeredStateMetaInfoBases, org.apache.flink.core.fs.Path exportBasePath, Map<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase.Key,List<org.forstdb.ExportImportFilesMetaData>> resultOutput)	Exports the data of the given column families in the given DB.
void	mergeStateHandlesWithClipAndIngest(List<StateHandleTransferSpec> keyedStateHandles, byte[] startKeyGroupPrefixBytes, byte[] stopKeyGroupPrefixBytes)	Restores the base DB by merging multiple state handles into one.
ForStRestoreResult	restore()	Root method that branches for different implementations of IncrementalKeyedStateHandle.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ForStIncrementalRestoreOperation

public ForStIncrementalRestoreOperation(String operatorIdentifier,
                                        org.apache.flink.runtime.state.KeyGroupRange keyGroupRange,
                                        int keyGroupPrefixBytes,
                                        org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry,
                                        ClassLoader userCodeClassLoader,
                                        Map<String,ForStOperationUtils.ForStKvStateInfo> kvStateInformation,
                                        org.apache.flink.runtime.state.StateSerializerProvider<K> keySerializerProvider,
                                        ForStResourceContainer optionsContainer,
                                        org.apache.flink.core.fs.Path forstBasePath,
                                        org.apache.flink.core.fs.Path instanceRocksDBPath,
                                        org.forstdb.DBOptions dbOptions,
                                        Function<String,org.forstdb.ColumnFamilyOptions> columnFamilyOptionsFactory,
                                        ForStNativeMetricOptions nativeMetricOptions,
                                        org.apache.flink.metrics.MetricGroup metricGroup,
                                        @Nonnull
                                        ForStDBTtlCompactFiltersManager ttlCompactFiltersManager,
                                        @Nonnegative
                                        long writeBatchSize,
                                        Long writeBufferManagerCapacity,
                                        org.apache.flink.runtime.state.StateBackend.CustomInitializationMetrics customInitializationMetrics,
                                        @Nonnull
                                        Collection<org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle> restoreStateHandles,
                                        double overlapFractionThreshold,
                                        boolean useIngestDbRestoreMode,
                                        boolean useDeleteFilesInRange,
                                        org.apache.flink.core.execution.RecoveryClaimMode recoveryClaimMode)

Method Detail

restore

public ForStRestoreResult restore()
                           throws Exception

Root method that branches for different implementations of IncrementalKeyedStateHandle.

Specified by:

restore in interface ForStRestoreOperation

Specified by:

restore in interface org.apache.flink.runtime.state.RestoreOperation<K>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

mergeStateHandlesWithClipAndIngest

public void mergeStateHandlesWithClipAndIngest(List<StateHandleTransferSpec> keyedStateHandles,
                                               byte[] startKeyGroupPrefixBytes,
                                               byte[] stopKeyGroupPrefixBytes)
                                        throws Exception

Restores the base DB by merging multiple state handles into one. This method first checks if all data to import is in the expected key-groups range and then uses import/export. Otherwise, this method falls back to copying the data using a temporary DB.

Parameters:

keyedStateHandles - the list of state handles to restore the base DB from.

startKeyGroupPrefixBytes - the min/start key of the key groups range as bytes.

stopKeyGroupPrefixBytes - the max+1/end key of the key groups range as bytes.

Throws:

Exception - on any restore error.

exportColumnFamilies

public void exportColumnFamilies(org.forstdb.RocksDB db,
                                 List<org.forstdb.ColumnFamilyHandle> columnFamilyHandles,
                                 List<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase> registeredStateMetaInfoBases,
                                 org.apache.flink.core.fs.Path exportBasePath,
                                 Map<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase.Key,List<org.forstdb.ExportImportFilesMetaData>> resultOutput)
                          throws org.forstdb.RocksDBException,
                                 IOException

Exports the data of the given column families in the given DB.

Parameters:

db - the DB to export from.

columnFamilyHandles - the column families to export.

registeredStateMetaInfoBases - meta information about the registered states in the DB.

exportBasePath - the path to which the export files go.

resultOutput - output parameter for the metadata of the export.

Throws:

org.forstdb.RocksDBException - on problems inside RocksDB.

IOException