public class FileTierPartitionState extends Object implements TierPartitionState, AutoCloseable
Modifier and Type | Class and Description |
---|---|
static interface |
FileTierPartitionState.MaterializationListener |
static class |
FileTierPartitionState.TierInMemorySegmentMetadata
In-memory minimal cache of the essential tier segment metadata
|
TierPartitionState.AppendResult, TierPartitionState.RestoreResult
Constructor and Description |
---|
FileTierPartitionState(File dir,
kafka.server.LogDirFailureChannel logDirFailureChannel,
org.apache.kafka.common.TopicPartition topicPartition,
boolean tieringEnabled,
kafka.utils.Scheduler scheduler) |
Modifier and Type | Method and Description |
---|---|
TierPartitionState.AppendResult |
append(AbstractTierMetadata metadata,
OffsetAndEpoch offsetAndEpoch)
Appends abstract metadata to the tier partition.
|
void |
beginCatchup()
Begin catchup phase for tier partition state.
|
void |
close()
Close TierPartition, flushing to disk.
|
void |
closeHandlers() |
long |
committedEndOffset()
Return the end offset spanned by the TierPartitionState that has been committed to disk.
|
void |
delete()
Delete this TierPartitionState from local storage.
|
File |
dir()
The directory where the TierPartition is stored on disk.
|
long |
endOffset()
Return the uncommitted end offset spanned by the TierPartitionState.
|
Collection<TierObjectMetadata> |
fencedSegments() |
boolean |
flush()
Flush data contained in this TierPartitionState to disk.
|
String |
flushedPath() |
kafka.common.FollowerRestorePoint |
followerRestorePoint(long localLogStartOffset)
Fetch and return the FollowerRestorePoint for the given localLogStartOffset.
|
boolean |
isTieringEnabled() |
static Optional<FileTierPartitionIterator> |
iterator(org.apache.kafka.common.TopicPartition topicPartition,
FileChannel channel) |
OffsetAndEpoch |
lastLocalMaterializedSrcOffsetAndEpoch()
Return the last materialized source offset and epoch
|
long |
materializationLag()
Return the current tier materialization lag.
|
CompletableFuture<Optional<TierObjectMetadata>> |
materializeUptoLeaderEpoch(int targetEpoch)
Listen for the flushed materialization state to reach or surpass the provided LeaderEpoch.
|
CompletableFuture<TierObjectMetadata> |
materializeUptoObjectIdAndRestoreEpoch(long upperBoundEndOffset,
UUID targetObjectId,
int targetRestoreEpoch)
Listen for the flushed materialization state to materialize a fully uploaded segment matching the provided targetObjectId.
|
CompletableFuture<TierObjectMetadata> |
materializeUptoOffset(long targetOffset)
Listen for the flushed materialization state to reach or surpass the provided targetOffset.
|
boolean |
maybeOpenChannelOnOffsetTieredException()
Called when a replica receives OFFSET_TIERED exception while replicating from leader.
|
boolean |
mayContainTieredData()
Indicates if a partition may have some tiered data, essentially whether tiered storage is currently enabled or was enabled
earlier for this partition.
|
Optional<TierObjectMetadata> |
metadata(long targetOffset)
Lookup the TierObjectMetadata which will contain data for a target offset.
|
int |
numSegments()
Scan the ObjectMetadata (segment) entries in this tier partition, and return the count.
|
int |
numSegments(long from,
long to)
Count of the number of segments lying within a range
|
void |
onCatchUpComplete()
Mark catchup completed for tier partition state.
|
Optional<FileTierPartitionState.TierInMemorySegmentMetadata> |
previousMetadataBeforeOffset(long targetStartOffset)
Get the in-memory metadata for the immediate previous segment (even beyond retention) to target startOffset
|
static Optional<Header> |
readHeader(FileChannel channel) |
TierPartitionState.RestoreResult |
restoreState(TierPartitionForceRestore metadata,
ByteBuffer targetState,
TierPartitionStatus targetStatus,
OffsetAndEpoch offsetAndEpoch)
Performs a TierPartitionState restore, swapping the current state file
from buffer and replacing any internal state with the contents of the new TierPartitionState,
setting the TierPartitionStatus of the state to status.
|
List<FileTierPartitionState.TierInMemorySegmentMetadata> |
segmentInMemoryMetadataRange(long from,
long to)
List of all the segment (minimal) metadata whose startOffset is in the range (even if they are beyond retention).
|
Iterator<TierObjectMetadata> |
segments()
Get an iterator for all readable tiered segments.
|
Iterator<TierObjectMetadata> |
segments(long from,
long to)
Get an iterator for all readable tiered segments in a given range.
|
void |
setTieringDisabled()
Reset the tier(ing) enabled flag at TierPartitionState.
|
boolean |
setTieringEnabled()
Called when tiering is enabled for this tier topic partition.
|
boolean |
setTopicId(UUID topicId)
Sets the TopicIdPartition for this TierPartitionState.
|
Optional<Long> |
startOffset()
Determine start offset spanned by the TierPartitionState.
|
TierPartitionStatus |
status()
Return the current status of the TierPartitionState.
|
int |
tierEpoch()
Return the current tierEpoch.
|
Optional<TopicIdPartition> |
topicIdPartition()
Optional TopicIdPartition corresponding to this TierPartition
If one has not been set, returns empty
|
org.apache.kafka.common.TopicPartition |
topicPartition()
The topic-partition corresponding to this TierPartition.
|
String |
toString() |
long |
totalSize()
Sum the size of all segment spanned by this TierPartitionState.
|
CompletableFuture<Boolean> |
trackMetadataInitialization(int targetLeaderEpoch)
Listen for the flushed materialization state to materialize up to or past the provided leader epoch.
|
void |
updateDir(File dir)
Update the directory reference for the log and indices in this segment.
|
public FileTierPartitionState(File dir, kafka.server.LogDirFailureChannel logDirFailureChannel, org.apache.kafka.common.TopicPartition topicPartition, boolean tieringEnabled, kafka.utils.Scheduler scheduler) throws IOException
IOException
public org.apache.kafka.common.TopicPartition topicPartition()
TierPartitionState
topicPartition
in interface TierPartitionState
public Optional<TopicIdPartition> topicIdPartition()
TierPartitionState
topicIdPartition
in interface TierPartitionState
public boolean setTopicId(UUID topicId) throws IOException
TierPartitionState
setTopicId
in interface TierPartitionState
topicId
- The topic id to assignIOException
public boolean isTieringEnabled()
isTieringEnabled
in interface TierPartitionState
public boolean setTieringEnabled() throws IOException
TierPartitionState
setTieringEnabled
in interface TierPartitionState
IOException
public void setTieringDisabled()
TierPartitionState
setTieringDisabled
in interface TierPartitionState
public Optional<Long> startOffset()
TierPartitionState
startOffset
in interface TierPartitionState
public long endOffset()
TierPartitionState
endOffset
in interface TierPartitionState
public long committedEndOffset()
TierPartitionState
committedEndOffset
in interface TierPartitionState
public long totalSize()
TierPartitionState
totalSize
in interface TierPartitionState
public boolean flush() throws IOException
TierPartitionState
flush
in interface TierPartitionState
IOException
- if the flush operation failedpublic int tierEpoch()
TierPartitionState
tierEpoch
in interface TierPartitionState
public File dir()
TierPartitionState
dir
in interface TierPartitionState
public void delete() throws IOException
TierPartitionState
delete
in interface TierPartitionState
IOException
public void updateDir(File dir)
TierPartitionState
updateDir
in interface TierPartitionState
dir
- The new directorypublic TierPartitionState.RestoreResult restoreState(TierPartitionForceRestore metadata, ByteBuffer targetState, TierPartitionStatus targetStatus, OffsetAndEpoch offsetAndEpoch)
TierPartitionState
restoreState
in interface TierPartitionState
metadata
- the TierPartitionForceRestore metadata including the coordinates and
metadata about the state to be restored.targetState
- ByteBuffer containing the contents of the TierPartitionState to restoretargetStatus
- the status that the TierPartitionState will be transitioned to upon
successful restorationoffsetAndEpoch
- Offset and epoch corresponding to this metadata entrypublic void closeHandlers() throws IOException
closeHandlers
in interface TierPartitionState
IOException
public TierPartitionStatus status()
TierPartitionState
status
in interface TierPartitionState
public long materializationLag()
TierPartitionState
materializationLag
in interface TierPartitionState
public void beginCatchup()
TierPartitionState
beginCatchup
in interface TierPartitionState
public void onCatchUpComplete()
TierPartitionState
onCatchUpComplete
in interface TierPartitionState
public int numSegments(long from, long to)
TierPartitionState
numSegments
in interface TierPartitionState
from
- Start of the range, include segment which contains "from" (inclusive)to
- End of the range, upper bound exclusive offset to include or the end of the log if "to" is past the endpublic int numSegments()
TierPartitionState
numSegments
in interface TierPartitionState
public CompletableFuture<TierObjectMetadata> materializeUptoOffset(long targetOffset) throws IOException
materializeUptoOffset
in interface TierPartitionState
targetOffset
- The minimum materialized offset required to complete this listener.IOException
- Propagated from the listener.public CompletableFuture<TierObjectMetadata> materializeUptoObjectIdAndRestoreEpoch(long upperBoundEndOffset, UUID targetObjectId, int targetRestoreEpoch) throws IOException
materializeUptoObjectIdAndRestoreEpoch
in interface TierPartitionState
upperBoundEndOffset
- The upper bound end offset of the targetObjectId. This is used to set an upper bound for which the targetObjectId (at the targetRestoreEpoch) will not be found above.
The upperBoundEndOffset is also used to provide approximate materialization progress for the materialization lag metric.targetObjectId
- The objectId of the target TierObjectMetadata. The listener will only complete if this objectId is found.targetRestoreEpoch
- The target TierPartitionState restore epoch. The listener will only complete if materialization has made it up to or surpassed this restore epoch.IOException
- Propagated from the listener.public CompletableFuture<Optional<TierObjectMetadata>> materializeUptoLeaderEpoch(int targetEpoch) throws IOException
materializeUptoLeaderEpoch
in interface TierPartitionState
targetEpoch
- The minimum LeaderEpoch required to complete this listener.IOException
- Propagated from the listener.public CompletableFuture<Boolean> trackMetadataInitialization(int targetLeaderEpoch) throws IOException
trackMetadataInitialization
in interface TierPartitionState
targetLeaderEpoch
- The minimum leader epoch required to complete this listener.IllegalStateException
- Thrown if there was an existing metadata initialization listener registered.IOException
- Propagated from the listener.public void close() throws IOException
TierPartitionState
close
in interface AutoCloseable
close
in interface TierPartitionState
IOException
public TierPartitionState.AppendResult append(AbstractTierMetadata metadata, OffsetAndEpoch offsetAndEpoch)
TierPartitionState
append
in interface TierPartitionState
metadata
- AbstractTierMetadata entry to be appended to the tier partition log.offsetAndEpoch
- Offset and epoch corresponding to this metadata entrypublic Iterator<TierObjectMetadata> segments()
TierPartitionState
segments
in interface TierPartitionState
public Iterator<TierObjectMetadata> segments(long from, long to)
TierPartitionState
segments
in interface TierPartitionState
from
- Start of the range, include segment which contains "from" (inclusive)to
- End of the range, upper bound exclusive offset to include or the end of the log if "to" is past the endpublic List<FileTierPartitionState.TierInMemorySegmentMetadata> segmentInMemoryMetadataRange(long from, long to)
TierPartitionState
segmentInMemoryMetadataRange
in interface TierPartitionState
from
- Inclusive startOffset positionto
- Inclusive endOffset positionpublic Optional<FileTierPartitionState.TierInMemorySegmentMetadata> previousMetadataBeforeOffset(long targetStartOffset)
TierPartitionState
previousMetadataBeforeOffset
in interface TierPartitionState
targetStartOffset
- target startOffsetpublic kafka.common.FollowerRestorePoint followerRestorePoint(long localLogStartOffset)
followerRestorePoint
in interface TierPartitionState
localLogStartOffset
- target startOffsetpublic Optional<TierObjectMetadata> metadata(long targetOffset) throws IOException
TierPartitionState
metadata
in interface TierPartitionState
targetOffset
- the target offset to lookup the overlapping or next metadata for.IOException
- if disk error encounteredpublic OffsetAndEpoch lastLocalMaterializedSrcOffsetAndEpoch()
TierPartitionState
lastLocalMaterializedSrcOffsetAndEpoch
in interface TierPartitionState
public String flushedPath()
public Collection<TierObjectMetadata> fencedSegments()
fencedSegments
in interface TierPartitionState
public static Optional<Header> readHeader(FileChannel channel) throws IOException
IOException
public static Optional<FileTierPartitionIterator> iterator(org.apache.kafka.common.TopicPartition topicPartition, FileChannel channel) throws IOException
IOException
public boolean mayContainTieredData()
TierPartitionState
mayContainTieredData
in interface TierPartitionState
public boolean maybeOpenChannelOnOffsetTieredException() throws IOException
TierPartitionState
maybeOpenChannelOnOffsetTieredException
in interface TierPartitionState
IOException