@InterfaceStability.Evolving public interface RemoteStorageManager extends org.apache.kafka.common.Configurable, Closeable
Each upload or copy of a segment is initiated with RemoteLogSegmentMetadata
containing RemoteLogSegmentId
which is universally unique even for the same topic partition and offsets.
RemoteLogSegmentMetadata
is stored in RemoteLogMetadataManager
before and after copy/delete operations on
RemoteStorageManager
with the respective RemoteLogSegmentState
. RemoteLogMetadataManager
is
responsible for storing and fetching metadata about the remote log segments in a strongly consistent manner.
This allows RemoteStorageManager
to have eventual consistency on metadata (although the data is stored
in strongly consistent semantics).
Modifier and Type | Interface and Description |
---|---|
static class |
RemoteStorageManager.IndexType
Type of the index file.
|
Modifier and Type | Method and Description |
---|---|
void |
copyLogSegmentData(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
LogSegmentData logSegmentData)
Copies the given
LogSegmentData provided for the given remoteLogSegmentMetadata . |
void |
deleteLogSegmentData(RemoteLogSegmentMetadata remoteLogSegmentMetadata)
Deletes the resources associated with the given
remoteLogSegmentMetadata . |
InputStream |
fetchIndex(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
RemoteStorageManager.IndexType indexType)
Returns the index for the respective log segment of
RemoteLogSegmentMetadata . |
InputStream |
fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
int startPosition)
Returns the remote log segment data file/object as InputStream for the given
RemoteLogSegmentMetadata
starting from the given startPosition. |
InputStream |
fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
int startPosition,
int endPosition)
Returns the remote log segment data file/object as InputStream for the given
RemoteLogSegmentMetadata
starting from the given startPosition. |
void copyLogSegmentData(RemoteLogSegmentMetadata remoteLogSegmentMetadata, LogSegmentData logSegmentData) throws RemoteStorageException
LogSegmentData
provided for the given remoteLogSegmentMetadata
. This includes
log segment and its auxiliary indexes like offset index, time index, transaction index, leader epoch index, and
producer snapshot index.
Invoker of this API should always send a unique id as part of RemoteLogSegmentMetadata.remoteLogSegmentId()
even when it retries to invoke this method for the same log segment data.
remoteLogSegmentMetadata
- metadata about the remote log segment.logSegmentData
- data to be copied to tiered storage.RemoteStorageException
- if there are any errors in storing the data of the segment.InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata, int startPosition) throws RemoteStorageException
RemoteLogSegmentMetadata
starting from the given startPosition. The stream will end at the end of the remote log segment data file/object.remoteLogSegmentMetadata
- metadata about the remote log segment.startPosition
- start position of log segment to be read, inclusive.RemoteStorageException
- if there are any errors while fetching the desired segment.RemoteResourceNotFoundException
- when there are no resources associated with the given remoteLogSegmentMetadata.InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata, int startPosition, int endPosition) throws RemoteStorageException
RemoteLogSegmentMetadata
starting from the given startPosition. The stream will end at the smaller of endPosition and the end of the
remote log segment data file/object.remoteLogSegmentMetadata
- metadata about the remote log segment.startPosition
- start position of log segment to be read, inclusive.endPosition
- end position of log segment to be read, inclusive.RemoteStorageException
- if there are any errors while fetching the desired segment.RemoteResourceNotFoundException
- when there are no resources associated with the given remoteLogSegmentMetadata.InputStream fetchIndex(RemoteLogSegmentMetadata remoteLogSegmentMetadata, RemoteStorageManager.IndexType indexType) throws RemoteStorageException
RemoteLogSegmentMetadata
.remoteLogSegmentMetadata
- metadata about the remote log segment.indexType
- type of the index to be fetched for the segment.RemoteStorageException
- if there are any errors while fetching the index.RemoteResourceNotFoundException
- when there are no resources associated with the given remoteLogSegmentMetadata.void deleteLogSegmentData(RemoteLogSegmentMetadata remoteLogSegmentMetadata) throws RemoteStorageException
remoteLogSegmentMetadata
. Deletion is considered as
successful if this call returns successfully without any errors. It will throw RemoteStorageException
if
there are any errors in deleting the file.
remoteLogSegmentMetadata
- metadata about the remote log segment to be deleted.RemoteResourceNotFoundException
- if the requested resource is not foundRemoteStorageException
- if there are any storage related errors occurred.RemoteResourceNotFoundException
- when there are no resources associated with the given remoteLogSegmentMetadata.