Package org.apache.flink.streaming.api.functions.sink.filesystem

Class HadoopPathBasedBulkFormatBuilder<IN,BucketID,T extends HadoopPathBasedBulkFormatBuilder<IN,BucketID,T>>

java.lang.Object

org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder<IN,BucketID,T>

org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder<IN,BucketID,T>

All Implemented Interfaces:

Serializable

@Internal
public class HadoopPathBasedBulkFormatBuilder<IN,BucketID,T extends HadoopPathBasedBulkFormatBuilder<IN,BucketID,T>>
extends org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder<IN,BucketID,T>

Buckets builder to create buckets that use HadoopPathBasedPartFileWriter.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder

DEFAULT_BUCKET_CHECK_INTERVAL

Constructor Summary

Constructors

Constructor	Description
HadoopPathBasedBulkFormatBuilder(org.apache.hadoop.fs.Path basePath, HadoopPathBasedBulkWriter.Factory<IN> writerFactory, HadoopFileCommitterFactory fileCommitterFactory, org.apache.hadoop.conf.Configuration configuration, org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner, org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.CheckpointRollingPolicy<IN,BucketID> policy, org.apache.flink.streaming.api.functions.sink.filesystem.BucketFactory<IN,BucketID> bucketFactory, org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig)
HadoopPathBasedBulkFormatBuilder(org.apache.hadoop.fs.Path basePath, HadoopPathBasedBulkWriter.Factory<IN> writerFactory, org.apache.hadoop.conf.Configuration configuration, org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner)

Method Summary

Modifier and Type	Method	Description
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets<IN,BucketID>	createBuckets(int subtaskIndex)
org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,BucketID>	createBucketWriter()
T	withBucketAssigner(org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner)
T	withBucketFactory(org.apache.flink.streaming.api.functions.sink.filesystem.BucketFactory<IN,BucketID> factory)
T	withConfiguration(org.apache.hadoop.conf.Configuration configuration)
T	withOutputFileConfig(org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig)
T	withRollingPolicy(org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.CheckpointRollingPolicy<IN,BucketID> rollingPolicy)

Methods inherited from class org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder

self

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopPathBasedBulkFormatBuilder

public HadoopPathBasedBulkFormatBuilder(org.apache.hadoop.fs.Path basePath,
                                        HadoopPathBasedBulkWriter.Factory<IN> writerFactory,
                                        org.apache.hadoop.conf.Configuration configuration,
                                        org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner)

HadoopPathBasedBulkFormatBuilder

public HadoopPathBasedBulkFormatBuilder(org.apache.hadoop.fs.Path basePath,
                                        HadoopPathBasedBulkWriter.Factory<IN> writerFactory,
                                        HadoopFileCommitterFactory fileCommitterFactory,
                                        org.apache.hadoop.conf.Configuration configuration,
                                        org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner,
                                        org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.CheckpointRollingPolicy<IN,BucketID> policy,
                                        org.apache.flink.streaming.api.functions.sink.filesystem.BucketFactory<IN,BucketID> bucketFactory,
                                        org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig)

Method Detail

withBucketAssigner

public T withBucketAssigner(org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner)

withRollingPolicy

public T withRollingPolicy(org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.CheckpointRollingPolicy<IN,BucketID> rollingPolicy)

withBucketFactory

public T withBucketFactory(org.apache.flink.streaming.api.functions.sink.filesystem.BucketFactory<IN,BucketID> factory)

withOutputFileConfig

public T withOutputFileConfig(org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig)

withConfiguration

public T withConfiguration(org.apache.hadoop.conf.Configuration configuration)

createBucketWriter

public org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,BucketID> createBucketWriter()

Specified by:

createBucketWriter in class org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder<IN,BucketID,T extends HadoopPathBasedBulkFormatBuilder<IN,BucketID,T>>

createBuckets

public org.apache.flink.streaming.api.functions.sink.filesystem.Buckets<IN,BucketID> createBuckets(int subtaskIndex)
                                                                                                  throws IOException

Specified by:

createBuckets in class org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder<IN,BucketID,T extends HadoopPathBasedBulkFormatBuilder<IN,BucketID,T>>

Throws:

IOException