Package org.apache.flink.core.fs.local
Class LocalRecoverableWriter
- java.lang.Object
-
- org.apache.flink.core.fs.local.LocalRecoverableWriter
-
- All Implemented Interfaces:
RecoverableWriter
@Internal public class LocalRecoverableWriter extends Object implements RecoverableWriter
ARecoverableWriterfor theLocalFileSystem.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from interface org.apache.flink.core.fs.RecoverableWriter
RecoverableWriter.CommitRecoverable, RecoverableWriter.ResumeRecoverable
-
-
Constructor Summary
Constructors Constructor Description LocalRecoverableWriter(LocalFileSystem fs)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description booleancleanupRecoverableState(RecoverableWriter.ResumeRecoverable resumable)Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.static FilegenerateStagingTempFilePath(File targetFile)SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable>getCommitRecoverableSerializer()The serializer for the CommitRecoverable types created in this writer.SimpleVersionedSerializer<RecoverableWriter.ResumeRecoverable>getResumeRecoverableSerializer()The serializer for the ResumeRecoverable types created in this writer.RecoverableFsDataOutputStreamopen(Path filePath)Opens a new recoverable stream to write to the given path.RecoverableFsDataOutputStreamrecover(RecoverableWriter.ResumeRecoverable recoverable)Resumes a recoverable stream consistently at the point indicated by the given ResumeRecoverable.RecoverableFsDataOutputStream.CommitterrecoverForCommit(RecoverableWriter.CommitRecoverable recoverable)Recovers a recoverable stream consistently at the point indicated by the given CommitRecoverable for finalizing and committing.booleanrequiresCleanupOfRecoverableState()Marks if the writer requires to do any additional cleanup/freeing of resources occupied as part of aRecoverableWriter.ResumeRecoverable, e.g.booleansupportsResume()Checks whether the writer and its streams support resuming (appending to) files after recovery (via theRecoverableWriter.recover(ResumeRecoverable)method).
-
-
-
Constructor Detail
-
LocalRecoverableWriter
public LocalRecoverableWriter(LocalFileSystem fs)
-
-
Method Detail
-
open
public RecoverableFsDataOutputStream open(Path filePath) throws IOException
Description copied from interface:RecoverableWriterOpens a new recoverable stream to write to the given path. Whether existing files will be overwritten is implementation specific and should not be relied upon.- Specified by:
openin interfaceRecoverableWriter- Parameters:
filePath- The path of the file/object to write to.- Returns:
- A new RecoverableFsDataOutputStream writing a new file/object.
- Throws:
IOException- Thrown if the stream could not be opened/initialized.
-
recover
public RecoverableFsDataOutputStream recover(RecoverableWriter.ResumeRecoverable recoverable) throws IOException
Description copied from interface:RecoverableWriterResumes a recoverable stream consistently at the point indicated by the given ResumeRecoverable. Future writes to the stream will continue / append the file as of that point.This method is optional and whether it is supported is indicated through the
RecoverableWriter.supportsResume()method.- Specified by:
recoverin interfaceRecoverableWriter- Parameters:
recoverable- The opaque handle with the recovery information.- Returns:
- A recoverable stream writing to the file/object as it was at the point when the ResumeRecoverable was created.
- Throws:
IOException- Thrown, if resuming fails.
-
requiresCleanupOfRecoverableState
public boolean requiresCleanupOfRecoverableState()
Description copied from interface:RecoverableWriterMarks if the writer requires to do any additional cleanup/freeing of resources occupied as part of aRecoverableWriter.ResumeRecoverable, e.g. temporarily files created or objects uploaded to external systems.In case cleanup is required, then
RecoverableWriter.cleanupRecoverableState(ResumeRecoverable)should be called.- Specified by:
requiresCleanupOfRecoverableStatein interfaceRecoverableWriter- Returns:
trueif cleanup is required,falseotherwise.
-
cleanupRecoverableState
public boolean cleanupRecoverableState(RecoverableWriter.ResumeRecoverable resumable) throws IOException
Description copied from interface:RecoverableWriterFrees up any resources that were previously occupied in order to be able to recover from a (potential) failure. These can be temporary files that were written to the filesystem or objects that were uploaded to S3.NOTE: This operation should not throw an exception, but return false if the cleanup did not happen for any reason.
- Specified by:
cleanupRecoverableStatein interfaceRecoverableWriter- Parameters:
resumable- TheRecoverableWriter.ResumeRecoverablewhose state we want to clean-up.- Returns:
trueif the resources were successfully freed,falseotherwise (e.g. the file to be deleted was not there for any reason - already deleted or never created).- Throws:
IOException
-
recoverForCommit
public RecoverableFsDataOutputStream.Committer recoverForCommit(RecoverableWriter.CommitRecoverable recoverable) throws IOException
Description copied from interface:RecoverableWriterRecovers a recoverable stream consistently at the point indicated by the given CommitRecoverable for finalizing and committing. This will publish the target file with exactly the data that was written up to the point then the CommitRecoverable was created.- Specified by:
recoverForCommitin interfaceRecoverableWriter- Parameters:
recoverable- The opaque handle with the recovery information.- Returns:
- A committer that publishes the target file.
- Throws:
IOException- Thrown, if recovery fails.
-
getCommitRecoverableSerializer
public SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable> getCommitRecoverableSerializer()
Description copied from interface:RecoverableWriterThe serializer for the CommitRecoverable types created in this writer. This serializer should be used to store the CommitRecoverable in checkpoint state or other forms of persistent state.- Specified by:
getCommitRecoverableSerializerin interfaceRecoverableWriter
-
getResumeRecoverableSerializer
public SimpleVersionedSerializer<RecoverableWriter.ResumeRecoverable> getResumeRecoverableSerializer()
Description copied from interface:RecoverableWriterThe serializer for the ResumeRecoverable types created in this writer. This serializer should be used to store the ResumeRecoverable in checkpoint state or other forms of persistent state.- Specified by:
getResumeRecoverableSerializerin interfaceRecoverableWriter
-
supportsResume
public boolean supportsResume()
Description copied from interface:RecoverableWriterChecks whether the writer and its streams support resuming (appending to) files after recovery (via theRecoverableWriter.recover(ResumeRecoverable)method).If true, then this writer supports the
RecoverableWriter.recover(ResumeRecoverable)method. If false, then that method may not be supported and streams can only be recovered viaRecoverableWriter.recoverForCommit(CommitRecoverable).- Specified by:
supportsResumein interfaceRecoverableWriter
-
-