Package org.apache.flink.formats.hadoop.bulk

Class HadoopPathBasedPartFileWriter.HadoopPathBasedBucketWriter<IN,BucketID>

java.lang.Object

org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriter.HadoopPathBasedBucketWriter<IN,BucketID>

All Implemented Interfaces:

org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,BucketID>

Enclosing class:

HadoopPathBasedPartFileWriter<IN,BucketID>

public static class HadoopPathBasedPartFileWriter.HadoopPathBasedBucketWriter<IN,BucketID>
extends Object
implements org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,BucketID>

Factory to create HadoopPathBasedPartFileWriter. This writer does not support snapshotting the in-progress files. For pending files, it stores the target path and the staging file path into the state.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter

org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter.PendingFile

Constructor Summary

Constructors

Constructor	Description
HadoopPathBasedBucketWriter(org.apache.hadoop.conf.Configuration configuration, HadoopPathBasedBulkWriter.Factory<IN> bulkWriterFactory, HadoopFileCommitterFactory fileCommitterFactory)

Method Summary

Modifier and Type	Method	Description
boolean	cleanupInProgressFileRecoverable(org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable)
org.apache.flink.streaming.api.functions.sink.filesystem.WriterProperties	getProperties()
HadoopPathBasedPartFileWriter<IN,BucketID>	openNewInProgressFile(BucketID bucketID, org.apache.flink.core.fs.Path flinkPath, long creationTime)
org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter.PendingFile	recoverPendingFile(org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable)
org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter<IN,BucketID>	resumeInProgressFileFrom(BucketID bucketID, org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.InProgressFileRecoverable inProgressFileSnapshot, long creationTime)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter

openNewCompactingFile

Constructor Detail

HadoopPathBasedBucketWriter

public HadoopPathBasedBucketWriter(org.apache.hadoop.conf.Configuration configuration,
                                   HadoopPathBasedBulkWriter.Factory<IN> bulkWriterFactory,
                                   HadoopFileCommitterFactory fileCommitterFactory)

Method Detail

openNewInProgressFile

public HadoopPathBasedPartFileWriter<IN,BucketID> openNewInProgressFile(BucketID bucketID,
                                                                              org.apache.flink.core.fs.Path flinkPath,
                                                                              long creationTime)
                                                                       throws IOException

Specified by:

openNewInProgressFile in interface org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,BucketID>

Throws:

IOException

recoverPendingFile

public org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter.PendingFile recoverPendingFile(org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable)
                                                                                                     throws IOException

Specified by:

recoverPendingFile in interface org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,BucketID>

Throws:

IOException

getProperties

public org.apache.flink.streaming.api.functions.sink.filesystem.WriterProperties getProperties()

Specified by:

getProperties in interface org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,BucketID>

resumeInProgressFileFrom

public org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter<IN,BucketID> resumeInProgressFileFrom(BucketID bucketID,
                                                                                                                                 org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.InProgressFileRecoverable inProgressFileSnapshot,
                                                                                                                                 long creationTime)

Specified by:

resumeInProgressFileFrom in interface org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,BucketID>

cleanupInProgressFileRecoverable

public boolean cleanupInProgressFileRecoverable(org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable)

Specified by:

cleanupInProgressFileRecoverable in interface org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,BucketID>