public class PointInTimeTierPartitionStateBuilder extends Object
However, any failure of an individual getObject(FTPS) or append(FTPS, Event) is considered a localized failure that need not hinder execution of the remainder of the program.
Glossary of terms used in this class: - Tier Partition Snapshots (TPS), an abbreviation of File Tier Partition State (FTPS) Snapshots, refers to the periodic snapshots taken of the tier state of each tiered topic-partition in a Kafka cluster. These files are also known as Tier Metadata Snapshots, but this term is not used here because "metadata" in this class typically refers to Tier Object Metadata. - Tier Topic Snapshots (TTS), an abbreviation Tier Topic Partition Snapshots, refers to the continuous snapshots taken of (a subset of) the Kafka cluster's tier topic. - Tier Object Metadata (or just Object Metadata) refers to the subclasses of TierObjectStore.ObjectStoreMetadata; this class is used for abstract representing files in tier store; they are the parameter passed to getObject for downloading a file from a tier storage provider.
Modifier and Type | Class and Description |
---|---|
static class |
PointInTimeTierPartitionStateBuilder.FtpsSnapshotsMetadata
Result tuple for fetchTierPartitionStateMetadata
|
Modifier and Type | Field and Description |
---|---|
protected static String |
DEFAULT_KEY_PREFIX |
protected static RetryPolicy |
DEFAULT_RETRY_POLICY |
protected static String |
TIER_PARTITION_SNAPSHOT_PREFIX |
protected static String |
TIER_STATE_CHECKSUM_FILENAME |
protected static String |
TIER_STATE_FILENAME |
protected static String |
TIER_TOPIC_NAME |
Modifier | Constructor and Description |
---|---|
protected |
PointInTimeTierPartitionStateBuilder(TierObjectStore store,
RetryPolicy retryPolicy,
ThreadPoolExecutor threadPool,
String keyPrefix,
RestoreMetricsManager metricsManager)
Private constructor used by tests to instantiate keyPrefix, parallelism level, and working directory with custom
values.
|
|
PointInTimeTierPartitionStateBuilder(TierObjectStore store,
ThreadPoolExecutor threadPool,
RestoreMetricsManager metricsManager)
Different object stores have different performance characteristics, so the number of threads should be customized
according to the particular object store in use.
|
Modifier and Type | Method and Description |
---|---|
Map<TopicIdPartition,FtpsStateForRestore> |
buildFtpsFromSnapshot(Map<org.apache.kafka.common.TopicPartition,PartitionRestoreContext> partitionRestoreContextMap)
For each partition, starting with FTPS snapshots close to the given revertSinceTimestamp, builds a new
Tier Partition State for each given topic-partition by replaying the events in Tier Topic Snapshots.
|
static Map<Integer,Long> |
calculateTierTopicLastMaterializedOffsets(Map<TopicIdPartition,FtpsStateForRestore> stateToRestoreMap)
Based on to-be-restored user partitions to find where to start materializing from tier topic.
|
SnapshotObjectStoreUtils |
getSnapshotUtils() |
protected Map<TopicIdPartition,FtpsStateForRestore> |
initStateForRestoreMap(Map<TopicIdPartition,Future<Path>> tierPartitionSnapshotFileFutures,
Path tierPartitionStatesDir,
Map<org.apache.kafka.common.TopicPartition,PartitionRestoreContext> partitionRestoreContextMap)
Constructs results map for the stitching process by copying snapshots inside new FTPS parent directories.
|
protected static final String TIER_TOPIC_NAME
protected static final String TIER_PARTITION_SNAPSHOT_PREFIX
protected static final String DEFAULT_KEY_PREFIX
protected static final String TIER_STATE_FILENAME
protected static final String TIER_STATE_CHECKSUM_FILENAME
protected static final RetryPolicy DEFAULT_RETRY_POLICY
public PointInTimeTierPartitionStateBuilder(TierObjectStore store, ThreadPoolExecutor threadPool, RestoreMetricsManager metricsManager)
store
- TierObjectStore from which to retrieve FTPS and TTPS filesthreadPool
- Thread poolprotected PointInTimeTierPartitionStateBuilder(TierObjectStore store, RetryPolicy retryPolicy, ThreadPoolExecutor threadPool, String keyPrefix, RestoreMetricsManager metricsManager)
keyPrefix
is for running tests in an isolated directory, e.g. "foo" will result in fetch(myFile)
calls being made to "foo/myFile".store
- TierObjectStore from which to retrieve FTPS and TTPS filesretryPolicy
- RetryPolicy for object store accessthreadPool
- Thread poolkeyPrefix
- A private tier store prefixpublic SnapshotObjectStoreUtils getSnapshotUtils()
protected Map<TopicIdPartition,FtpsStateForRestore> initStateForRestoreMap(Map<TopicIdPartition,Future<Path>> tierPartitionSnapshotFileFutures, Path tierPartitionStatesDir, Map<org.apache.kafka.common.TopicPartition,PartitionRestoreContext> partitionRestoreContextMap) throws IOException
tierPartitionSnapshotFileFutures
- The local paths to which Tier Partition Snapshots are being downloadedtierPartitionStatesDir
- The parent directory of all new Tier Partition StatesIOException
- when unable to create files/directoriespublic Map<TopicIdPartition,FtpsStateForRestore> buildFtpsFromSnapshot(Map<org.apache.kafka.common.TopicPartition,PartitionRestoreContext> partitionRestoreContextMap) throws IOException, InterruptedException
partitionRestoreContextMap
- partitionRestoreContexts for which new Tier Partition States need to be builtFtpsStateForRestore
. An empty collection indicates
that the operation failed to find the needed FTPS or TTPS files in object store, likely due to the given
startTimestamp
being too low. Otherwise, each FtpsStateForRestore
contains info for
(FileTierPartitionState
, segmentsToRestore
, segmentsToDelete
) representing the newly
generated FTPS snapshot and the UUIDs and SegmentState data of segments from that FTPS file that the caller needs
to restore/delete.IOException
- on error from file reading/writingTierObjectStoreFatalException
- on Object Store fatal error from underlying call to any listObject, a call
to getObject(TTPS), or an event deserialization errorInterruptedException
public static Map<Integer,Long> calculateTierTopicLastMaterializedOffsets(Map<TopicIdPartition,FtpsStateForRestore> stateToRestoreMap)
stateToRestoreMap
- to-be-restored user partition's state mapTierTopicConsumerForRestore
should consume from to pull the live TierTopic events.