public class FileTierPartitionState extends Object implements TierPartitionState, AutoCloseable
Modifier and Type | Class and Description |
---|---|
static interface |
FileTierPartitionState.MaterializationListener |
TierPartitionState.AppendResult, TierPartitionState.RecoveryOperation, TierPartitionState.RestoreResult
Modifier and Type | Field and Description |
---|---|
TierPartitionStateCleanupConfig |
cleanupConfig |
static long |
FILE_OFFSET |
static short |
SUPER_BLOCK_LENGTH_ADLER |
static short |
SUPER_BLOCK_LENGTH_NO_CHECKSUM |
Constructor and Description |
---|
FileTierPartitionState(File dir,
org.apache.kafka.storage.internals.log.LogDirFailureChannel logDirFailureChannel,
org.apache.kafka.common.TopicPartition topicPartition,
boolean tieringEnabled,
org.apache.kafka.server.util.Scheduler scheduler,
boolean checksumEnabled,
boolean compactFeatureFlag,
org.apache.kafka.common.utils.Time time,
TierPartitionStateCleanupConfig cleanupConfig,
boolean tierPartitionStateSnapshotFeatureFlag,
int brokerId) |
Modifier and Type | Method and Description |
---|---|
CompactStats |
accumulatedCompactStats()
Return the accumulated compaction stats recorded by the TierPartitionState.
|
TierPartitionState.AppendResult |
append(AbstractTierMetadata metadata,
OffsetAndEpoch offsetAndEpoch)
Appends abstract metadata to the tier partition.
|
void |
beginCatchup()
Begin catchup phase for tier partition state.
|
void |
beginDiscover()
Begin discover FTPS snapshot phase for tier partition state.
|
CheckedFileIO |
checkedFileIO() |
void |
checkInvariants()
Checks invariants for the tier state and throws an IllegalStateException if any are violated.
|
void |
close()
Close TierPartition, flushing to disk.
|
void |
closeHandlers() |
long |
committedEndOffset()
Return the end offset spanned by the TierPartitionState that has been committed to disk.
|
long |
compactDirtyStartOffset()
Return the compaction dirty start offset checkpoint recorded by the TierPartitionState.
|
Collection<kafka.log.TierLogSegment> |
compactedSegments() |
long |
dataEndOffset()
Return the offset of the highest offset where a record is available to be read.
|
void |
delete()
Delete this TierPartitionState from local storage.
|
File |
dir()
The directory where the TierPartition is stored on disk.
|
long |
endOffset()
Return the uncommitted end offset spanned by the TierPartitionState.
|
Collection<kafka.log.TierLogSegment> |
fencedSegments() |
boolean |
flush()
Flush data contained in this TierPartitionState to disk.
|
static Path |
flushedFilePath(String basePath,
Algorithm checksumAlgorithm) |
String |
flushedPath() |
kafka.common.FollowerRestorePoint |
followerRestorePoint(long localLogStartOffset)
Fetch and return the FollowerRestorePoint for the given localLogStartOffset.
|
TierPartitionState.RestoreResult |
forceRestoreState(TierPartitionForceRestore metadata,
ByteBuffer targetState,
TierPartitionStatus targetStatus,
OffsetAndEpoch offsetAndEpoch)
Performs a TierPartitionState restore, swapping the current state file
from buffer and replacing any internal state with the contents of the new TierPartitionState,
setting the TierPartitionStatus of the state to status.
|
boolean |
isErrorStatusReachedViaFenceEvent() |
boolean |
isTieringEnabled() |
static Optional<FileTierPartitionIterator> |
iterator(org.apache.kafka.common.TopicPartition topicPartition,
CheckedFileIO channel) |
UUID |
lastCommittedSnapshotId()
Return the UUID for snapshot written at the flushed file
|
CompactStats |
lastCompactStats()
Return the stats of last compaction recorded by the TierPartitionState.
|
OffsetAndEpoch |
lastLocalMaterializedSrcOffsetAndEpoch()
Return the last materialized source offset and epoch
|
UUID |
lastSnapshotId()
Return the last snapshot UUID of tier partition state metadata
|
long |
lastSnapshotTimestampMs()
Return the timestamp of when last snapshot of tier partition state metadata was taken
|
long |
materializationLag()
Return the current tier materialization lag.
|
CompletableFuture<Optional<kafka.log.TierLogSegment>> |
materializeUptoLeaderEpoch(int targetEpoch)
Listen for the flushed materialization state to reach or surpass the provided LeaderEpoch.
|
CompletableFuture<kafka.log.TierLogSegment> |
materializeUptoObjectIdAndRestoreEpoch(long upperBoundEndOffset,
UUID targetObjectId,
int targetRestoreEpoch)
Listen for the flushed materialization state to materialize a fully uploaded segment matching the provided targetObjectId.
|
CompletableFuture<kafka.log.TierLogSegment> |
materializeUptoOffset(long targetOffset)
Listen for the flushed materialization state to reach or surpass the provided targetOffset.
|
boolean |
maybeOpenChannelOnOffsetTieredException()
Called when a replica receives OFFSET_TIERED exception while replicating from leader.
|
boolean |
mayContainTieredData()
Indicates if a partition may have some tiered data, essentially whether tiered storage is currently enabled or was enabled
earlier for this partition.
|
Optional<kafka.log.TierLogSegment> |
metadata(long targetOffset)
Lookup the TierObjectMetadata which will contain data for a target offset.
|
static Path |
mutableFilePath(String basePath,
Algorithm checksumAlgorithm) |
int |
numSegments()
Scan the ObjectMetadata (segment) entries in this tier partition, and return the count.
|
int |
numSegments(long from,
long to)
Count of the number of segments lying within a range
|
void |
onCatchUpComplete()
Mark catchup completed for tier partition state.
|
void |
onDiscoverComplete()
Mark discover FTPS snapshot completed for tier partition state.
|
Optional<SegmentState> |
previousMetadataBeforeOffset(long targetStartOffset)
Get the in-memory metadata for the immediate previous segment (even beyond retention) to target startOffset
|
TierPartitionState.RestoreResult |
processRestoreEvents(AbstractTierMetadata event,
Optional<ByteBuffer> targetStateOpt,
TierPartitionStatus targetStatus,
OffsetAndEpoch offsetAndEpoch)
Processes one of the recovery events TierPartitionForceRestore or TierPartitionUnfreezeLogStartOffset.
|
TierPartitionState.RestoreResult |
processSnapshotMaterializationEvent(TierMetadataSnapshotUploadComplete metadata,
ByteBuffer targetState,
TierPartitionStatus targetStatus,
OffsetAndEpoch offsetAndEpoch)
Performs materialization of a FTPS snapshot file, swapping the current state file
from buffer and replacing any internal state with the contents of the snapshot,
setting the TierPartitionStatus of the state to status.
|
TierPartitionState.RestoreResult |
processUnfreezeLogStartOffset(TierPartitionUnfreezeLogStartOffset metadata,
TierPartitionStatus restoreStatus,
OffsetAndEpoch sourceOffsetAndEpoch)
Unfreeze log start offset at the MergedLog.
|
static Optional<Header> |
readHeader(CheckedFileIO channel) |
static Path |
recoverPath(String basePath,
Algorithm checksumAlgorithm) |
List<SegmentState> |
segmentInMemoryMetadataRange(long from,
long to)
List of all the segment (minimal) metadata whose startOffset is in the range (even if they are beyond retention).
|
Iterator<kafka.log.TierLogSegment> |
segments()
Get an iterator for all readable tiered segments.
|
Iterator<kafka.log.TierLogSegment> |
segments(long from,
long to)
Get an iterator for all readable tiered segments in a given range.
|
List<SegmentState> |
segmentsWithStatesNotEqualTo(Set<TierObjectMetadata.State> excludedStates) |
void |
setTieredPartitionRecoveryWorkflowCb(Consumer<TierPartitionState.RecoveryOperation> cb)
Sets the callback function used by tier partition state to communicate to the MergedLog layer during recovery of
a tiered partition.
|
void |
setTieringDisabled()
Reset the tier(ing) enabled flag at TierPartitionState.
|
boolean |
setTieringEnabled()
Called when tiering is enabled for this tier topic partition.
|
boolean |
setTopicId(UUID topicId)
Sets the TopicIdPartition for this TierPartitionState.
|
boolean |
shouldTriggerCleanup() |
Optional<Long> |
startOffset()
Determine start offset spanned by the TierPartitionState.
|
long |
stateFileSize()
return the tier partition state metadata file size
|
TierPartitionStatus |
status()
Return the current status of the TierPartitionState.
|
int |
tierEpoch()
Return the current tierEpoch.
|
Optional<TopicIdPartition> |
topicIdPartition()
Optional TopicIdPartition corresponding to this TierPartition
If one has not been set, returns empty
|
org.apache.kafka.common.TopicPartition |
topicPartition()
The topic-partition corresponding to this TierPartition.
|
String |
toString() |
long |
totalSize()
Sum the size of all segment spanned by this TierPartitionState.
|
CompletableFuture<Boolean> |
trackMetadataInitialization(int targetLeaderEpoch)
Listen for the flushed materialization state to materialize up to or past the provided leader epoch.
|
void |
updateDir(File dir)
Update the directory reference for the log and indices in this segment.
|
boolean |
validateChecksum()
Validates the integrity by computing the checksum and validating the checksum.
|
static void |
writeHeader(CheckedFileIO channel,
Header header) |
public static final long FILE_OFFSET
public static final short SUPER_BLOCK_LENGTH_ADLER
public static final short SUPER_BLOCK_LENGTH_NO_CHECKSUM
public final TierPartitionStateCleanupConfig cleanupConfig
public FileTierPartitionState(File dir, org.apache.kafka.storage.internals.log.LogDirFailureChannel logDirFailureChannel, org.apache.kafka.common.TopicPartition topicPartition, boolean tieringEnabled, org.apache.kafka.server.util.Scheduler scheduler, boolean checksumEnabled, boolean compactFeatureFlag, org.apache.kafka.common.utils.Time time, TierPartitionStateCleanupConfig cleanupConfig, boolean tierPartitionStateSnapshotFeatureFlag, int brokerId) throws IOException
IOException
public org.apache.kafka.common.TopicPartition topicPartition()
TierPartitionState
topicPartition
in interface TierPartitionState
public Optional<TopicIdPartition> topicIdPartition()
TierPartitionState
topicIdPartition
in interface TierPartitionState
public boolean setTopicId(UUID topicId) throws IOException
TierPartitionState
setTopicId
in interface TierPartitionState
topicId
- The topic id to assignIOException
public boolean isTieringEnabled()
isTieringEnabled
in interface TierPartitionState
public boolean setTieringEnabled() throws IOException
TierPartitionState
setTieringEnabled
in interface TierPartitionState
IOException
public void setTieringDisabled()
TierPartitionState
setTieringDisabled
in interface TierPartitionState
public Optional<Long> startOffset()
TierPartitionState
startOffset
in interface TierPartitionState
public long endOffset()
TierPartitionState
endOffset
in interface TierPartitionState
public long dataEndOffset()
TierPartitionState
dataEndOffset
in interface TierPartitionState
public long compactDirtyStartOffset()
TierPartitionState
compactDirtyStartOffset
in interface TierPartitionState
public CompactStats lastCompactStats()
TierPartitionState
lastCompactStats
in interface TierPartitionState
public CompactStats accumulatedCompactStats()
TierPartitionState
accumulatedCompactStats
in interface TierPartitionState
public long committedEndOffset()
TierPartitionState
committedEndOffset
in interface TierPartitionState
public long totalSize()
TierPartitionState
totalSize
in interface TierPartitionState
public long stateFileSize()
TierPartitionState
stateFileSize
in interface TierPartitionState
public boolean flush() throws IOException
TierPartitionState
flush
in interface TierPartitionState
IOException
- if the flush operation failedpublic boolean shouldTriggerCleanup()
public boolean validateChecksum() throws IOException, InstantiationException, IllegalAccessException
TierPartitionState
validateChecksum
in interface TierPartitionState
IOException
InstantiationException
IllegalAccessException
public List<SegmentState> segmentsWithStatesNotEqualTo(Set<TierObjectMetadata.State> excludedStates)
public int tierEpoch()
TierPartitionState
tierEpoch
in interface TierPartitionState
public File dir()
TierPartitionState
dir
in interface TierPartitionState
public void delete() throws IOException
TierPartitionState
delete
in interface TierPartitionState
IOException
public void updateDir(File dir) throws IOException
TierPartitionState
updateDir
in interface TierPartitionState
dir
- The new directoryIOException
public TierPartitionState.RestoreResult forceRestoreState(TierPartitionForceRestore metadata, ByteBuffer targetState, TierPartitionStatus targetStatus, OffsetAndEpoch offsetAndEpoch)
If the buffer was written using a different checksum algorithm, then we perform the migration to match the current broker checksum algorithm
metadata
- the TierPartitionForceRestore metadata including the coordinates and
metadata about the state to be restored.targetState
- ByteBuffer containing the contents of the TierPartitionState to restoretargetStatus
- the status that the TierPartitionState will be transitioned to upon
successful restorationoffsetAndEpoch
- Offset and epoch corresponding to this metadata entrypublic TierPartitionState.RestoreResult processSnapshotMaterializationEvent(TierMetadataSnapshotUploadComplete metadata, ByteBuffer targetState, TierPartitionStatus targetStatus, OffsetAndEpoch offsetAndEpoch)
processSnapshotMaterializationEvent
in interface TierPartitionState
metadata
- the TierMetadataSnapshotUploadComplete metadatatargetState
- ByteBuffer containing the contents of the TierPartitionState to restoretargetStatus
- the status that the TierPartitionState will be transitioned to upon
successful restorationoffsetAndEpoch
- Offset and epoch corresponding to this metadata entrypublic TierPartitionState.RestoreResult processRestoreEvents(AbstractTierMetadata event, Optional<ByteBuffer> targetStateOpt, TierPartitionStatus targetStatus, OffsetAndEpoch offsetAndEpoch)
TierPartitionState
processRestoreEvents
in interface TierPartitionState
event
- the metadata event to be processedtargetStateOpt
- ByteBuffer containing the contents of the TierPartitionState to restoretargetStatus
- the status that the TierPartitionState will be transitioned to upon
successful restorationoffsetAndEpoch
- Offset and epoch corresponding to this metadata entrypublic TierPartitionState.RestoreResult processUnfreezeLogStartOffset(TierPartitionUnfreezeLogStartOffset metadata, TierPartitionStatus restoreStatus, OffsetAndEpoch sourceOffsetAndEpoch) throws org.apache.kafka.common.errors.KafkaStorageException
metadata
- metadata event to unfreeze log start offsetrestoreStatus
- TierPartitionStatus to restore the topic partition to, once log start offset is unfrozen.
Valid values are TierPartitionStatus.ONLINE and TierPartitionStatus.CATCHUPsourceOffsetAndEpoch
- offset and epoch for the metadata eventorg.apache.kafka.common.errors.KafkaStorageException
public void closeHandlers() throws IOException
closeHandlers
in interface TierPartitionState
IOException
public TierPartitionStatus status()
TierPartitionState
status
in interface TierPartitionState
public long materializationLag()
TierPartitionState
materializationLag
in interface TierPartitionState
public void beginCatchup()
TierPartitionState
beginCatchup
in interface TierPartitionState
public void onCatchUpComplete()
TierPartitionState
onCatchUpComplete
in interface TierPartitionState
public void beginDiscover()
TierPartitionState
beginDiscover
in interface TierPartitionState
public void onDiscoverComplete()
TierPartitionState
onDiscoverComplete
in interface TierPartitionState
public int numSegments(long from, long to)
TierPartitionState
numSegments
in interface TierPartitionState
from
- Start of the range, include segment which contains "from" (inclusive)to
- End of the range, upper bound exclusive offset to include or the end of the log if "to" is past the endpublic int numSegments()
TierPartitionState
numSegments
in interface TierPartitionState
public CompletableFuture<kafka.log.TierLogSegment> materializeUptoOffset(long targetOffset) throws IOException
materializeUptoOffset
in interface TierPartitionState
targetOffset
- The minimum materialized offset required to complete this listener.IOException
- Propagated from the listener.public CompletableFuture<kafka.log.TierLogSegment> materializeUptoObjectIdAndRestoreEpoch(long upperBoundEndOffset, UUID targetObjectId, int targetRestoreEpoch) throws IOException
materializeUptoObjectIdAndRestoreEpoch
in interface TierPartitionState
upperBoundEndOffset
- The upper bound end offset of the targetObjectId. This is used to set an upper bound for which the targetObjectId (at the targetRestoreEpoch) will not be found above.
The upperBoundEndOffset is also used to provide approximate materialization progress for the materialization lag metric.targetObjectId
- The objectId of the target TierObjectMetadata. The listener will only complete if this objectId is found.targetRestoreEpoch
- The target TierPartitionState restore epoch. The listener will only complete if materialization has made it up to or surpassed this restore epoch.IOException
- Propagated from the listener.public CompletableFuture<Optional<kafka.log.TierLogSegment>> materializeUptoLeaderEpoch(int targetEpoch) throws IOException
materializeUptoLeaderEpoch
in interface TierPartitionState
targetEpoch
- The minimum LeaderEpoch required to complete this listener.IOException
- Propagated from the listener.public CompletableFuture<Boolean> trackMetadataInitialization(int targetLeaderEpoch) throws IOException
trackMetadataInitialization
in interface TierPartitionState
targetLeaderEpoch
- The minimum leader epoch required to complete this listener.IllegalStateException
- Thrown if there was an existing metadata initialization listener registered.IOException
- Propagated from the listener.public void close() throws IOException
TierPartitionState
close
in interface AutoCloseable
close
in interface TierPartitionState
IOException
public TierPartitionState.AppendResult append(AbstractTierMetadata metadata, OffsetAndEpoch offsetAndEpoch)
TierPartitionState
append
in interface TierPartitionState
metadata
- AbstractTierMetadata entry to be appended to the tier partition log.offsetAndEpoch
- Offset and epoch corresponding to this metadata entrypublic void checkInvariants()
public Iterator<kafka.log.TierLogSegment> segments()
TierPartitionState
segments
in interface TierPartitionState
public Iterator<kafka.log.TierLogSegment> segments(long from, long to)
TierPartitionState
segments
in interface TierPartitionState
from
- Start of the range, include segment which contains "from" (inclusive)to
- End of the range, upper bound exclusive offset to include or the end of the log if "to" is past the endpublic List<SegmentState> segmentInMemoryMetadataRange(long from, long to)
TierPartitionState
segmentInMemoryMetadataRange
in interface TierPartitionState
from
- Inclusive startOffset positionto
- Inclusive endOffset positionpublic CheckedFileIO checkedFileIO()
public Optional<SegmentState> previousMetadataBeforeOffset(long targetStartOffset)
TierPartitionState
previousMetadataBeforeOffset
in interface TierPartitionState
targetStartOffset
- target startOffsetpublic kafka.common.FollowerRestorePoint followerRestorePoint(long localLogStartOffset)
followerRestorePoint
in interface TierPartitionState
localLogStartOffset
- target startOffsetpublic Optional<kafka.log.TierLogSegment> metadata(long targetOffset) throws IOException
TierPartitionState
metadata
in interface TierPartitionState
targetOffset
- the target offset to lookup the overlapping or next metadata for.IOException
- if disk error encounteredpublic OffsetAndEpoch lastLocalMaterializedSrcOffsetAndEpoch()
TierPartitionState
lastLocalMaterializedSrcOffsetAndEpoch
in interface TierPartitionState
public long lastSnapshotTimestampMs()
TierPartitionState
lastSnapshotTimestampMs
in interface TierPartitionState
public UUID lastSnapshotId()
TierPartitionState
lastSnapshotId
in interface TierPartitionState
public UUID lastCommittedSnapshotId()
TierPartitionState
lastCommittedSnapshotId
in interface TierPartitionState
public String flushedPath()
public Collection<kafka.log.TierLogSegment> fencedSegments()
fencedSegments
in interface TierPartitionState
public Collection<kafka.log.TierLogSegment> compactedSegments()
compactedSegments
in interface TierPartitionState
public static Optional<Header> readHeader(CheckedFileIO channel) throws IOException
IOException
public static Optional<FileTierPartitionIterator> iterator(org.apache.kafka.common.TopicPartition topicPartition, CheckedFileIO channel) throws IOException
IOException
public boolean mayContainTieredData()
TierPartitionState
mayContainTieredData
in interface TierPartitionState
public boolean maybeOpenChannelOnOffsetTieredException() throws IOException
TierPartitionState
maybeOpenChannelOnOffsetTieredException
in interface TierPartitionState
IOException
public static void writeHeader(CheckedFileIO channel, Header header) throws IOException
IOException
public void setTieredPartitionRecoveryWorkflowCb(Consumer<TierPartitionState.RecoveryOperation> cb)
TierPartitionState
setTieredPartitionRecoveryWorkflowCb
in interface TierPartitionState
cb
- callback function that takes a TierPartitionState.RecoveryOperation as a parameterpublic boolean isErrorStatusReachedViaFenceEvent()