public class KafkaCruiseControl extends Object
Modifier and Type | Class and Description |
---|---|
static class |
KafkaCruiseControl.CcStartupMode |
Modifier and Type | Field and Description |
---|---|
protected KafkaCruiseControlContext |
kafkaCruiseControlContext |
Constructor and Description |
---|
KafkaCruiseControl(Integer brokerId,
KafkaCruiseControlConfig config,
DataBalancerMetricsRegistry metricRegistry,
BlockingSendClient.Builder blockingSendClientBuilder,
KafkaCruiseControl.CcStartupMode startupMode)
Construct the Cruise Control
|
Modifier and Type | Method and Description |
---|---|
OptimizerResult |
addBrokers(MultiBrokerAdditionOperation additionCallback,
BalanceOpExecutionCompletionCallback completionCallback,
String uid)
Add brokers.
|
static OptimizerResult |
computeDrainBrokersPlan(Set<Integer> removedBrokers,
List<String> overriddenGoals,
OperationProgress operationProgress,
PlanComputationOptions options,
KafkaCruiseControlContext context)
Computes a plan to drain all of the partition replicas off of #
removedBrokers |
KafkaCruiseControlContext |
context() |
static Future<?> |
executeRemoval(Set<ExecutionProposal> proposals,
Set<Integer> removedBrokers,
String uuid,
BalanceOpExecutionCompletionCallback completionCallback,
KafkaCruiseControlContext context)
Execute the given balancing proposals for remove operations.
|
ExecutorState.State |
executionState() |
boolean |
executorIsReserved() |
OptimizerResult |
fixBrokerFailures(Set<Integer> removedBrokers,
List<String> goals,
String uuid,
PlanComputationOptions opts)
Drain brokers #
removedBrokers of all of their partition replicas,
moving them to other brokers in the cluster. |
AnomalyDetector |
getAnomalyDetector() |
static OptimizerResult |
getProposals(ClusterModel clusterModel,
List<Goal> goalsByPriority,
boolean allowCapacityEstimation,
boolean excludeRecentlyDemotedBrokers,
boolean excludeRecentlyRemovedBrokers,
boolean isTriggeredByGoalViolation,
Set<Integer> requestedDestinationBrokerIds,
KafkaCruiseControlContext context) |
OptimizerResult |
getProposals(List<String> goals,
ModelCompletenessRequirements requirements,
OperationProgress operationProgress,
boolean allowCapacityEstimation,
boolean excludeRecentlyDemotedBrokers,
boolean excludeRecentlyRemovedBrokers,
boolean isTriggeredByGoalViolation,
Set<Integer> requestedDestinationBrokerIds,
boolean isRebalanceDiskMode)
Optimize a cluster workload model.
|
static List<Goal> |
goalsByPriority(List<String> goals,
KafkaCruiseControlContext context)
Get a goals by priority based on the goal list.
|
boolean |
meetCompletenessRequirements(List<String> goals)
Check if the completeness requirements are met for the given goals.
|
static ModelCompletenessRequirements |
modelCompletenessRequirements(Collection<Goal> overrides,
KafkaCruiseControlContext context) |
void |
notifyBrokerChange(Set<Integer> changedBrokers,
BrokerChangeEvent changeEvent)
Receive a notification about a set of brokers
changedBrokers that have changed according to some BrokerChangeEvent event. |
RebalanceResult |
rebalance(List<String> goals,
boolean dryRun,
ModelCompletenessRequirements requirements,
OperationProgress operationProgress,
boolean allowCapacityEstimation,
String uuid,
boolean excludeRecentlyDemotedBrokers,
boolean excludeRecentlyRemovedBrokers,
boolean isTriggeredByGoalViolation,
Collection<String> goalsToImprove,
Set<Integer> requestedDestinationBrokerIds,
boolean isRebalanceDiskMode)
Rebalance the cluster
|
static Set<Integer> |
recentlyDemotedBrokers(KafkaCruiseControlContext context)
Get the set of broker IDs that have recently been demoted (had all leadership responsibilities removed)
|
static Set<Integer> |
recentlyRemovedBrokers(KafkaCruiseControlContext context)
Get the set of broker IDs that have recently been removed from the cluster via a removeBrokers call.
|
BrokerRemovalFuture |
removeBrokers(Map<Integer,Optional<Long>> brokersToRemoveAndEpochs,
boolean shouldShutdown,
BalanceOpExecutionCompletionCallback executionCompletionCallback,
BrokerRemovalCallback progressCallback,
String uuid)
Broker removal consists of 4 steps:
1.
|
static void |
sanityCheckBrokerPresence(Set<Integer> brokerIds,
KafkaCruiseControlContext context)
Sanity check whether the provided brokers exist in cluster or not.
|
static void |
sanityCheckCapacityEstimation(boolean allowCapacityEstimation,
Map<Integer,String> capacityEstimationInfoByBrokerId)
Check whether the given capacity estimation info indicates estimations for any broker when capacity estimation is
not permitted.
|
void |
setEvenClusterLoadStateManager(boolean selfHealingEnabled) |
void |
setGoalViolationSelfHealing(boolean selfHealingEnabled)
Enable or disable self healing for the GOAL_VIOLATION anomaly type.
|
void |
shutdown() |
void |
startUp(ApiStatePersistenceStore persistenceStore)
Start up the Cruise Control.
|
void |
updateConfigPermanently(String configKey,
Object configValue)
Update the KafkaCruiseControlConfig "key" to "value" permanently.
|
void |
updateThrottle(long newThrottle)
Update the throttle used for an ongoing execution.
|
void |
userTriggeredStopExecution()
Request the executor to stop any ongoing execution.
|
protected KafkaCruiseControlContext kafkaCruiseControlContext
public KafkaCruiseControl(Integer brokerId, KafkaCruiseControlConfig config, DataBalancerMetricsRegistry metricRegistry, BlockingSendClient.Builder blockingSendClientBuilder, KafkaCruiseControl.CcStartupMode startupMode)
config
- the configuration of Cruise Control.startupMode
- -- how CruiseControl was startedpublic void startUp(ApiStatePersistenceStore persistenceStore) throws ExecutionException, InterruptedException
public void shutdown()
public AnomalyDetector getAnomalyDetector()
public OptimizerResult fixBrokerFailures(Set<Integer> removedBrokers, List<String> goals, String uuid, PlanComputationOptions opts) throws Exception
removedBrokers
of all of their partition replicas,
moving them to other brokers in the cluster. Drain brokers should not be used
for live brokers, since execution will ignore the broker's static throttleremovedBrokers
- The brokers to decommission.goals
- The goal names (i.e. each matching Goal.name()
) to be met when decommissioning the brokers.
When empty all goals will be used.uuid
- UUID of the execution.opts
- the options to compute the plan withException
- When any exception occurred during the broker removal process.public BrokerRemovalFuture removeBrokers(Map<Integer,Optional<Long>> brokersToRemoveAndEpochs, boolean shouldShutdown, @Nonnull BalanceOpExecutionCompletionCallback executionCompletionCallback, @Nonnull BrokerRemovalCallback progressCallback, String uuid)
brokersToRemoveAndEpochs
- - a map consisting of the broker ids to remove as keys and an
optional value of their broker epochsshouldShutdown
- - whether the removal operation should include the shutdown stepprogressCallback
- - a callback utilized for tracking the progress of the remove broker callexecutionCompletionCallback
- -- a callback utilized to notify the operation scheduler when this has
finished.uuid
- - the unique ID of this operationpublic static OptimizerResult computeDrainBrokersPlan(Set<Integer> removedBrokers, List<String> overriddenGoals, OperationProgress operationProgress, PlanComputationOptions options, KafkaCruiseControlContext context) throws Exception
removedBrokers
Exception
public static void sanityCheckCapacityEstimation(boolean allowCapacityEstimation, Map<Integer,String> capacityEstimationInfoByBrokerId)
allowCapacityEstimation
- Allow capacity estimation in cluster model if the requested broker capacity is unavailable.capacityEstimationInfoByBrokerId
- Capacity estimation info by broker id for which there has been an estimation.public RebalanceResult rebalance(List<String> goals, boolean dryRun, ModelCompletenessRequirements requirements, OperationProgress operationProgress, boolean allowCapacityEstimation, String uuid, boolean excludeRecentlyDemotedBrokers, boolean excludeRecentlyRemovedBrokers, boolean isTriggeredByGoalViolation, Collection<String> goalsToImprove, Set<Integer> requestedDestinationBrokerIds, boolean isRebalanceDiskMode) throws KafkaCruiseControlException
goals
- The goal names (i.e. each matching Goal.name()
) to be met during the rebalance.
When empty all goals will be used.dryRun
- Whether it is a dry run or not.requirements
- The cluster model completeness requirements.operationProgress
- The progress of the job to report.allowCapacityEstimation
- Allow capacity estimation in cluster model if the requested broker capacity is unavailable.uuid
- UUID of the execution.excludeRecentlyDemotedBrokers
- Exclude recently demoted brokers from proposal generation for leadership transfer.excludeRecentlyRemovedBrokers
- Exclude recently removed brokers from proposal generation for replica transfer.isTriggeredByGoalViolation
- True if rebalance is triggered by goal violation, false otherwise.goalsToImprove
- -- a set of goals at least some of which must be fixable before executing the rebalance.requestedDestinationBrokerIds
- Explicitly requested destination broker Ids to limit the replica movement to
these brokers (if empty, no explicit filter is enforced -- cannot be null).isRebalanceDiskMode
- Whether rebalance between brokers or disks within the brokers.KafkaCruiseControlException
- When the rebalance encounter errors.public OptimizerResult addBrokers(MultiBrokerAdditionOperation additionCallback, @Nonnull BalanceOpExecutionCompletionCallback completionCallback, String uid) throws Exception
additionCallback
- - the callback for the operationcompletionCallback
- routine to be invoked on completion of the proposal execution. Accepts success/failure and Exception of the result.uid
- Unique ID of the execution.InterruptedException
- if the thread was interrupted.KafkaCruiseControlException
- When any exception other than InterruptedException occurred during the broker addition.Exception
public void notifyBrokerChange(Set<Integer> changedBrokers, BrokerChangeEvent changeEvent)
changedBrokers
that have changed according to some BrokerChangeEvent
event.public OptimizerResult getProposals(List<String> goals, ModelCompletenessRequirements requirements, OperationProgress operationProgress, boolean allowCapacityEstimation, boolean excludeRecentlyDemotedBrokers, boolean excludeRecentlyRemovedBrokers, boolean isTriggeredByGoalViolation, Set<Integer> requestedDestinationBrokerIds, boolean isRebalanceDiskMode) throws KafkaCruiseControlException
goals
- A list of goal names (i.e. each matching Goal.name()
) to optimize. When empty all goals will be used.requirements
- The model completeness requirements to enforce when generating the proposals.operationProgress
- The progress of the job to report.allowCapacityEstimation
- Allow capacity estimation in cluster model if the requested broker capacity is unavailable.excludeRecentlyDemotedBrokers
- Exclude recently demoted brokers from proposal generation for leadership transfer.excludeRecentlyRemovedBrokers
- Exclude recently removed brokers from proposal generation for replica transfer.isTriggeredByGoalViolation
- True if proposals is triggered by goal violation, false otherwise.requestedDestinationBrokerIds
- Explicitly requested destination broker Ids to limit the replica movement to
these brokers (if empty, no explicit filter is enforced -- cannot be null).isRebalanceDiskMode
- True to generate proposal to rebalance between disks within the brokers, false otherwise.KafkaCruiseControlException
- If anything goes wrong in optimization proposal calculation.public static OptimizerResult getProposals(ClusterModel clusterModel, List<Goal> goalsByPriority, boolean allowCapacityEstimation, boolean excludeRecentlyDemotedBrokers, boolean excludeRecentlyRemovedBrokers, boolean isTriggeredByGoalViolation, Set<Integer> requestedDestinationBrokerIds, KafkaCruiseControlContext context) throws KafkaCruiseControlException
KafkaCruiseControlException
public KafkaCruiseControlContext context()
public static Set<Integer> recentlyDemotedBrokers(KafkaCruiseControlContext context)
public static Set<Integer> recentlyRemovedBrokers(KafkaCruiseControlContext context)
public static Future<?> executeRemoval(Set<ExecutionProposal> proposals, Set<Integer> removedBrokers, String uuid, BalanceOpExecutionCompletionCallback completionCallback, KafkaCruiseControlContext context)
proposals
- the given balancing proposalsremovedBrokers
- Brokers to be removed, null if no brokers has been removed.uuid
- UUID of the execution.completionCallback
- the callback to be invoked when the proposal execution fails/succeedspublic void userTriggeredStopExecution()
public ExecutorState.State executionState()
public boolean executorIsReserved()
public static ModelCompletenessRequirements modelCompletenessRequirements(Collection<Goal> overrides, KafkaCruiseControlContext context)
public boolean meetCompletenessRequirements(List<String> goals)
goals
- A list of goals to check completeness for.public static List<Goal> goalsByPriority(List<String> goals, KafkaCruiseControlContext context)
goals
- A list of goals.public static void sanityCheckBrokerPresence(Set<Integer> brokerIds, KafkaCruiseControlContext context)
brokerIds
- A set of broker ids.public void updateThrottle(long newThrottle)
newThrottle
- The new value to be used for throttlingpublic void setGoalViolationSelfHealing(boolean selfHealingEnabled)
selfHealingEnabled
- #True
if self healing should be enabled for the GOAL_VIOLATION anomaly type, #false
otherwise.public void updateConfigPermanently(String configKey, Object configValue)
configKey
- -- name of the config, e.g. KafkaCruiseControlConfig.TOPICS_EXCLUDED_FROM_PARTITION_MOVEMENT_CONFIGconfigValue
- -- the new value that the config should getpublic void setEvenClusterLoadStateManager(boolean selfHealingEnabled)