io.confluent.telemetry.events.exporter.kafka

Class RandomBrokerPartitionSubsetPartitioner

java.lang.Object

io.confluent.telemetry.events.exporter.kafka.RandomBrokerPartitionSubsetPartitioner

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.kafka.clients.producer.Partitioner, org.apache.kafka.common.Configurable

public class RandomBrokerPartitionSubsetPartitioner
extends Object
implements org.apache.kafka.clients.producer.Partitioner

A partitioner that reduces the number of connections a producer needs to make by

1. Specifying a subset percentage of total partitions to produce to
2. Randomly connecting to brokers and taking all partitions on that broker until it has connected to the subset of partitions
3. Randomly produce to the recorded partitions

NOTE: This partitioner will not respect keys when determining the partition to be used. This is done so that keyed messages do not result in connections to every broker.

Field Summary

Fields

Modifier and Type	Field and Description
static double	DEFAULT_SUBSET_PARTITIONER_PARTITION_PERCENTAGE
static String	SUBSET_PARTITIONER_PARTITION_PERCENTAGE_CONFIG

Constructor Summary

Constructors

Constructor and Description
RandomBrokerPartitionSubsetPartitioner()

Method Summary

Modifier and Type	Method and Description
void	close()
void	configure(Map<String,?> configs)
void	onNewBatch(String topic, org.apache.kafka.common.Cluster cluster, int prevPartition) We need to deal with topology changes and redefine which partitions to write to.
int	partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, org.apache.kafka.common.Cluster cluster) Choose the partition for a record from partitions found on a limited number of brokers according to `subset.partitioner.partition.percentage`.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SUBSET_PARTITIONER_PARTITION_PERCENTAGE_CONFIG

public static final String SUBSET_PARTITIONER_PARTITION_PERCENTAGE_CONFIG

See Also:

Constant Field Values

DEFAULT_SUBSET_PARTITIONER_PARTITION_PERCENTAGE

public static final double DEFAULT_SUBSET_PARTITIONER_PARTITION_PERCENTAGE

See Also:

Constant Field Values

Constructor Detail

RandomBrokerPartitionSubsetPartitioner

public RandomBrokerPartitionSubsetPartitioner()

Method Detail

partition

public int partition(String topic,
                     Object key,
                     byte[] keyBytes,
                     Object value,
                     byte[] valueBytes,
                     org.apache.kafka.common.Cluster cluster)

Choose the partition for a record from partitions found on a limited number of brokers according to `subset.partitioner.partition.percentage`. NOTE: this partitioner will not respect the record key when choosing a partition for the record.

Specified by:

partition in interface org.apache.kafka.clients.producer.Partitioner

onNewBatch

public void onNewBatch(String topic,
                       org.apache.kafka.common.Cluster cluster,
                       int prevPartition)

We need to deal with topology changes and redefine which partitions to write to. Topology might change if

1. Number of partitions change
2. The preferred leaders of partitions change
3. The available partitions change

Specified by:

onNewBatch in interface org.apache.kafka.clients.producer.Partitioner

configure

public void configure(Map<String,?> configs)

Specified by:

configure in interface org.apache.kafka.common.Configurable

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.kafka.clients.producer.Partitioner