Class BufferingPartitionDataBuilder

java.lang.Object
io.confluent.kafka.replication.push.buffer.BufferingPartitionDataBuilder
public final class BufferingPartitionDataBuilder extends Object

Used to consume incoming PushReplicationEvents for a given partition replica and (eventually) to build a AppendRecordsRequestData.PartitionData out of them. The partition data created this way can then be sent with an AppendRecords request. When an incoming event can not be consumed, the builder is expected to be used to build a AppendRecordsRequestData.PartitionData to be sent, after which a new builder should be created for the subsequent events for that partition replica.

In case of multiple PushReplicationEvent.Type.MEMORY_RECORDS events, the buffered memory records can be sent efficiently using a custom MultiBufferRecords type. The records buffered for these types of events can also be limited by the total memory consumed by them.

This class is not thread-safe. It expects that the same thread will be initializing it, sending push replication events to it, and building a partition data out of it.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static final class 
    BufferingPartitionDataBuilder.MultiMemoryRecords
    A MultiBufferRecords that can send the buffers of multiple memory records in a single Send.

  • Method Summary

    Modifier and Type
    Method
    Description
    boolean
    addHighWatermarkUpdate(PushSession pushSession, long highWatermark)
    Updates the high watermark if the push session matches the initial push session and the new high watermark has a higher value
    boolean
    addLogStartOffsetUpdate(PushSession pushSession, long logStartOffset)
    Updates the log start offset if the push session matches the initial push session and the new log start offset has a higher value
    boolean
    addMemoryRecords(PushSession pushSession, org.apache.kafka.common.record.MemoryRecords memoryRecords, long appendOffset)
    Enqueues a new set of memory records to the request if the push session matches the initial push session and the size of the memory records doesn't bump the total bytes counter of the builder over its limit and potentially update the high watermark of the builder.
    List<org.apache.kafka.common.record.AbstractRecords>
    stopPushAndDiscardState(boolean sendEndSessionRequest)
    Stops a push session for this partition replica.

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • stopPushAndDiscardState

      public List<org.apache.kafka.common.record.AbstractRecords> stopPushAndDiscardState(boolean sendEndSessionRequest)

      Stops a push session for this partition replica.

      Whenever such an event is received, the builder will discard the data it has buffered, it will reset its internal state, and if sending an endSessionRequest is necessary it will mark that it has received such an event so that additional partition updates cannot be consumed. If sending an endSessionRequest isn't necessary, the caller is responsible for removing the partition data builder.

      If such an event has already been consumed before, the builder will take care to update its internal state so that it doesn't omit sending a request to the follower to notify it about the ending of the replication session if any of the consumed events had required that.

    • addHighWatermarkUpdate

      public boolean addHighWatermarkUpdate(PushSession pushSession, long highWatermark)
      Updates the high watermark if the push session matches the initial push session and the new high watermark has a higher value

    • addLogStartOffsetUpdate

      public boolean addLogStartOffsetUpdate(PushSession pushSession, long logStartOffset)
      Updates the log start offset if the push session matches the initial push session and the new log start offset has a higher value

    • addMemoryRecords

      public boolean addMemoryRecords(PushSession pushSession, org.apache.kafka.common.record.MemoryRecords memoryRecords, long appendOffset)
      Enqueues a new set of memory records to the request if the push session matches the initial push session and the size of the memory records doesn't bump the total bytes counter of the builder over its limit and potentially update the high watermark of the builder. The first records for a partition are always allowed, even if they would go over the configured partition limits. If that's not allowed, replicating these records would always require a downgrade to pull replication. The first set of memory records also updates the append offset of the builder.