Package org.apache.flink.changelog.fs

Class DuplicatingStateChangeFsUploader

java.lang.Object

org.apache.flink.changelog.fs.AbstractStateChangeFsUploader

org.apache.flink.changelog.fs.DuplicatingStateChangeFsUploader

All Implemented Interfaces:

AutoCloseable, StateChangeUploader

public class DuplicatingStateChangeFsUploader
extends AbstractStateChangeFsUploader

A StateChangeFsUploader implementation that writes the changes to remote and local.

Local dstl files are only managed by TM side, LocalChangelogRegistry, TaskChangelogRegistry and ChangelogTaskLocalStateStore are responsible for managing them. Remote dstl files are managed by TM side and JM side, TaskChangelogRegistry is responsible for TM side, and SharedStateRegistry is responsible for JM side.

The total discard logic of local dstl files is:

1. Register files to TaskChangelogRegistry.startTracking(org.apache.flink.runtime.state.StreamStateHandle, long) on AbstractStateChangeFsUploader.upload(java.util.Collection<org.apache.flink.changelog.fs.StateChangeUploadScheduler.UploadTask>).
2. Store the meta of files into ChangelogTaskLocalStateStore by AsyncCheckpointRunnable#reportCompletedSnapshotStates().
3. Pass control of the file to LocalChangelogRegistry#register when ChangelogKeyedStateBackend#notifyCheckpointComplete() , files of the previous checkpoint will be deleted by LocalChangelogRegistry#discardUpToCheckpoint at the same time.
4. When ChangelogTruncateHelper#materialized() or ChangelogTruncateHelper#checkpointSubsumed() is called, TaskChangelogRegistry#notUsed is responsible for deleting local files.
5. When one checkpoint is aborted, the dstl files of this checkpoint will be deleted by LocalChangelogRegistry#prune in StateChangelogWriter.reset(org.apache.flink.runtime.state.changelog.SequenceNumber, org.apache.flink.runtime.state.changelog.SequenceNumber, long).

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.changelog.fs.StateChangeUploader

StateChangeUploader.UploadTasksResult

Field Summary

Fields inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader

bufferSize, compression, metrics

Constructor Summary

Constructors

Constructor	Description
DuplicatingStateChangeFsUploader(org.apache.flink.api.common.JobID jobID, org.apache.flink.core.fs.Path basePath, org.apache.flink.core.fs.FileSystem fileSystem, boolean compression, int bufferSize, ChangelogStorageMetricGroup metrics, TaskChangelogRegistry changelogRegistry, org.apache.flink.runtime.state.LocalRecoveryDirectoryProvider localRecoveryDirectoryProvider)

Method Summary

Modifier and Type	Method	Description
void	close()
org.apache.flink.changelog.fs.OutputStreamWithPos	prepareStream()

Methods inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader

generateFileName, upload

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DuplicatingStateChangeFsUploader

public DuplicatingStateChangeFsUploader(org.apache.flink.api.common.JobID jobID,
                                        org.apache.flink.core.fs.Path basePath,
                                        org.apache.flink.core.fs.FileSystem fileSystem,
                                        boolean compression,
                                        int bufferSize,
                                        ChangelogStorageMetricGroup metrics,
                                        TaskChangelogRegistry changelogRegistry,
                                        org.apache.flink.runtime.state.LocalRecoveryDirectoryProvider localRecoveryDirectoryProvider)

Method Detail

prepareStream

public org.apache.flink.changelog.fs.OutputStreamWithPos prepareStream()
                                                                throws IOException

Throws:

IOException

close

public void close()
           throws Exception

Throws:

Exception