public final class BufferingPartitionDataBuilder extends Object
Used to consume incoming PushReplicationEvent
s for a given partition replica and
(eventually) to build a AppendRecordsRequestData.PartitionData
out of them. The partition data created this way
can then be sent with an AppendRecords request. When an incoming event can not be consumed,
the builder is expected to be used to build a AppendRecordsRequestData.PartitionData
to be sent, after which a
new builder should be created for the subsequent events for that partition replica.
In case of multiple PushReplicationEvent.Type.MEMORY_RECORDS
events, the buffered memory records can be
sent efficiently using a custom MultiBufferRecords
type. The records buffered for these
types of events can also be limited by the total memory consumed by them.
This class is not thread-safe. It expects that the same thread will be initializing it, sending push replication events to it, and building a partition data out of it.
Modifier and Type | Class and Description |
---|---|
static class |
BufferingPartitionDataBuilder.PartitionRecords
A
MultiBufferRecords that can send the buffers of multiple memory records in a
single org.apache.kafka.common.network.Send . |
Modifier and Type | Field and Description |
---|---|
static long |
UNSET_OFFSET_VALUE |
Constructor and Description |
---|
BufferingPartitionDataBuilder(org.apache.kafka.common.TopicIdPartition topicIdPartition,
PushSession initialSession,
long maxSizeInBytes)
Creates a new builder for the given partition with the given restriction for the maximum
amount of memory-consuming records that can be buffered in it.
|
Modifier and Type | Method and Description |
---|---|
boolean |
addPartitionUpdate(PushReplicationEvent<?> pushReplicationEvent,
PushSession mostRecentSession)
Processes any type of partition update event.
|
boolean |
addTransitionRecords(PushReplicationEvent<PushReplicationEvent.TransitionRecordsPayload> transitionEvent,
PushSession mostRecentSession)
Consumes a
PushReplicationEvent.Type.TRANSITION_RECORDS event for this partition replica. |
org.apache.kafka.common.message.AppendRecordsRequestData.PartitionData |
build()
Builds a
AppendRecordsRequestData.PartitionData out of the consumed push replication events for this
partition replica with the given leader epoch and replication session ID. |
List<org.apache.kafka.common.record.AbstractRecords> |
stopPushAndDiscardState(PushReplicationEvent<CompletableFuture<Void>> stopEvent)
Consumes a
PushReplicationEvent.Type.STOP_PUSH event for this partition replica. |
org.apache.kafka.common.TopicIdPartition |
topicIdPartition() |
public static final long UNSET_OFFSET_VALUE
public BufferingPartitionDataBuilder(org.apache.kafka.common.TopicIdPartition topicIdPartition, PushSession initialSession, long maxSizeInBytes)
public org.apache.kafka.common.TopicIdPartition topicIdPartition()
public boolean addTransitionRecords(PushReplicationEvent<PushReplicationEvent.TransitionRecordsPayload> transitionEvent, PushSession mostRecentSession)
PushReplicationEvent.Type.TRANSITION_RECORDS
event for this partition replica. At most one
such event can be consumed by a given builder instance, and subsequently
PushReplicationEvent.Type.MEMORY_RECORDS
events cannot be consumed by that instance (only offset-updating
events can be consumed before a PartitionData is built).public List<org.apache.kafka.common.record.AbstractRecords> stopPushAndDiscardState(PushReplicationEvent<CompletableFuture<Void>> stopEvent)
Consumes a PushReplicationEvent.Type.STOP_PUSH
event for this partition replica. At most one such
event can be consumed by a given builder instance, and when that happens all the buffered
records in it are evicted in order to be post-processed, and the offset data for the
partition updates is reset. Subsequently, no other events can be consumed and the partition
data that the builder creates indicates the stopping of the current push session.
The first sent stop push event can always be consumed.
public boolean addPartitionUpdate(PushReplicationEvent<?> pushReplicationEvent, PushSession mostRecentSession)
Processes any type of partition update event. If the corresponding update event cannot be consumed, that's indicated in the return value of the method.
The following processing is applied to the events:
PushReplicationEvent.Type.HWM_UPDATE
/PushReplicationEvent.Type.LSO_UPDATE
events update the currently stored
high watermark/log start offset if they carry a higher offset value. These events
can always be consumed.PushReplicationEvent.Type.MEMORY_RECORDS
PushReplicationEvent.Type.START_PUSH
(and as a corollary PushReplicationEvent.Type.FILE_RECORDS
) events. If the size of their memory
records doesn't bump the total bytes counter of the builder over its limit, they add
their memory records to the list of buffered records to be sent, update the counter, and
potentially update the high watermark of the builder. The first consumed event also
updates the append offset of the builder.
public org.apache.kafka.common.message.AppendRecordsRequestData.PartitionData build()
AppendRecordsRequestData.PartitionData
out of the consumed push replication events for this
partition replica with the given leader epoch and replication session ID. It also completes
the future for any consumed start event or stop event, assuming that from that point on,
the constructed partition data is eventually going to be sent through in an AppendRecords.