Package org.apache.flink.contrib.streaming.state.snapshot

Class RocksDBSnapshotStrategyBase<K,R extends org.apache.flink.runtime.state.SnapshotResources>

java.lang.Object

org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase<K,R>

Type Parameters:

K - type of the backend keys.

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.runtime.state.SnapshotStrategy<org.apache.flink.runtime.state.KeyedStateHandle,RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources>

Direct Known Subclasses:

RocksIncrementalSnapshotStrategy, RocksNativeFullSnapshotStrategy

public abstract class RocksDBSnapshotStrategyBase<K,R extends org.apache.flink.runtime.state.SnapshotResources>
extends Object
implements org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.runtime.state.SnapshotStrategy<org.apache.flink.runtime.state.KeyedStateHandle,RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources>, AutoCloseable

Abstract base class for SnapshotStrategy implementations for RocksDB state backend.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources	A SnapshotResources for native rocksdb snapshot.
protected static class	RocksDBSnapshotStrategyBase.PreviousSnapshot	Previous snapshot with uploaded sst files.
protected class	RocksDBSnapshotStrategyBase.RocksDBSnapshotOperation	Common operation in native rocksdb snapshot result supplier.

Nested classes/interfaces inherited from interface org.apache.flink.runtime.state.SnapshotStrategy

org.apache.flink.runtime.state.SnapshotStrategy.SnapshotResultSupplier<S extends org.apache.flink.runtime.state.StateObject>

Field Summary

Fields

Modifier and Type	Field	Description
protected UUID	backendUID	The state handle ids of all sst files materialized in snapshots for previous checkpoints.
protected org.rocksdb.RocksDB	db	RocksDB instance from the backend.
protected static RocksDBSnapshotStrategyBase.PreviousSnapshot	EMPTY_PREVIOUS_SNAPSHOT
protected File	instanceBasePath	Base path of the RocksDB instance.
protected int	keyGroupPrefixBytes	Number of bytes in the key-group prefix.
protected org.apache.flink.runtime.state.KeyGroupRange	keyGroupRange	The key-group range for the task.
protected org.apache.flink.api.common.typeutils.TypeSerializer<K>	keySerializer	The key serializer of the backend.
protected LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo>	kvStateInformation	Key/Value state meta info from the backend.
protected String	localDirectoryName	The local directory name of the current snapshot strategy.
protected org.apache.flink.runtime.state.LocalRecoveryConfig	localRecoveryConfig	The configuration for local recovery.
protected org.apache.flink.util.ResourceGuard	rocksDBResourceGuard	Resource guard for the RocksDB instance.

Constructor Summary

Constructors

Constructor

Description

RocksDBSnapshotStrategyBase(String description, org.rocksdb.RocksDB db, org.apache.flink.util.ResourceGuard rocksDBResourceGuard, org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer, LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation, org.apache.flink.runtime.state.KeyGroupRange keyGroupRange, int keyGroupPrefixBytes, org.apache.flink.runtime.state.LocalRecoveryConfig localRecoveryConfig, File instanceBasePath, UUID backendUID)

Method Summary

Modifier and Type	Method	Description
protected void	cleanupIncompleteSnapshot(org.apache.flink.core.fs.CloseableRegistry tmpResourcesRegistry, org.apache.flink.runtime.state.SnapshotDirectory localBackupDirectory)
abstract void	close()
String	getDescription()
protected org.apache.flink.runtime.state.SnapshotResult<org.apache.flink.runtime.state.StreamStateHandle>	materializeMetaData(org.apache.flink.core.fs.CloseableRegistry snapshotCloseableRegistry, org.apache.flink.core.fs.CloseableRegistry tmpResourcesRegistry, List<org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshot> stateMetaInfoSnapshots, long checkpointId, org.apache.flink.runtime.state.CheckpointStreamFactory checkpointStreamFactory)
protected org.apache.flink.runtime.state.SnapshotDirectory	prepareLocalSnapshotDirectory(long checkpointId)
protected abstract RocksDBSnapshotStrategyBase.PreviousSnapshot	snapshotMetaData(long checkpointId, List<org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshot> stateMetaInfoSnapshots)
RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources	syncPrepareResources(long checkpointId)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.runtime.state.SnapshotStrategy

asyncSnapshot

Field Detail

db

@Nonnull
protected org.rocksdb.RocksDB db

RocksDB instance from the backend.

rocksDBResourceGuard

@Nonnull
protected final org.apache.flink.util.ResourceGuard rocksDBResourceGuard

Resource guard for the RocksDB instance.

keySerializer

@Nonnull
protected final org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer

The key serializer of the backend.

kvStateInformation

@Nonnull
protected final LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation

Key/Value state meta info from the backend.

keyGroupRange

@Nonnull
protected final org.apache.flink.runtime.state.KeyGroupRange keyGroupRange

The key-group range for the task.

keyGroupPrefixBytes

@Nonnegative
protected final int keyGroupPrefixBytes

Number of bytes in the key-group prefix.

localRecoveryConfig

@Nonnull
protected final org.apache.flink.runtime.state.LocalRecoveryConfig localRecoveryConfig

The configuration for local recovery.

instanceBasePath

@Nonnull
protected final File instanceBasePath

Base path of the RocksDB instance.

localDirectoryName

protected final String localDirectoryName

The local directory name of the current snapshot strategy.

backendUID

@Nonnull
protected final UUID backendUID

The state handle ids of all sst files materialized in snapshots for previous checkpoints.

EMPTY_PREVIOUS_SNAPSHOT

protected static final RocksDBSnapshotStrategyBase.PreviousSnapshot EMPTY_PREVIOUS_SNAPSHOT

Constructor Detail

RocksDBSnapshotStrategyBase

public RocksDBSnapshotStrategyBase(@Nonnull
                                   String description,
                                   @Nonnull
                                   org.rocksdb.RocksDB db,
                                   @Nonnull
                                   org.apache.flink.util.ResourceGuard rocksDBResourceGuard,
                                   @Nonnull
                                   org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer,
                                   @Nonnull
                                   LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation,
                                   @Nonnull
                                   org.apache.flink.runtime.state.KeyGroupRange keyGroupRange,
                                   @Nonnegative
                                   int keyGroupPrefixBytes,
                                   @Nonnull
                                   org.apache.flink.runtime.state.LocalRecoveryConfig localRecoveryConfig,
                                   @Nonnull
                                   File instanceBasePath,
                                   @Nonnull
                                   UUID backendUID)

Method Detail

getDescription

@Nonnull
public String getDescription()

syncPrepareResources

public RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources syncPrepareResources(long checkpointId)
                                                                                throws Exception

Specified by:

syncPrepareResources in interface org.apache.flink.runtime.state.SnapshotStrategy<K,R extends org.apache.flink.runtime.state.SnapshotResources>

Throws:

Exception

snapshotMetaData

protected abstract RocksDBSnapshotStrategyBase.PreviousSnapshot snapshotMetaData(long checkpointId,
                                                                                 @Nonnull
                                                                                 List<org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshot> stateMetaInfoSnapshots)

prepareLocalSnapshotDirectory

@Nonnull
protected org.apache.flink.runtime.state.SnapshotDirectory prepareLocalSnapshotDirectory(long checkpointId)
                                                                                  throws IOException

Throws:

IOException

cleanupIncompleteSnapshot

protected void cleanupIncompleteSnapshot(@Nonnull
                                         org.apache.flink.core.fs.CloseableRegistry tmpResourcesRegistry,
                                         @Nonnull
                                         org.apache.flink.runtime.state.SnapshotDirectory localBackupDirectory)

materializeMetaData

@Nonnull
protected org.apache.flink.runtime.state.SnapshotResult<org.apache.flink.runtime.state.StreamStateHandle> materializeMetaData(@Nonnull
                                                                                                                              org.apache.flink.core.fs.CloseableRegistry snapshotCloseableRegistry,
                                                                                                                              @Nonnull
                                                                                                                              org.apache.flink.core.fs.CloseableRegistry tmpResourcesRegistry,
                                                                                                                              @Nonnull
                                                                                                                              List<org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshot> stateMetaInfoSnapshots,
                                                                                                                              long checkpointId,
                                                                                                                              @Nonnull
                                                                                                                              org.apache.flink.runtime.state.CheckpointStreamFactory checkpointStreamFactory)
                                                                                                                       throws Exception

Throws:

Exception

close

public abstract void close()
                    throws IOException

Specified by:

close in interface AutoCloseable

Throws:

IOException