The broker capacity config resolver implementation based on files. The format of the file is JSON. Depending on
whether the JBOD configuration is used or not, this JSON file may specify disk capacity per logDir.
Example capacity file without JBOD:
{
"brokerCapacities":[
{
"brokerId": "-1",
"capacity": {
"DISK": "1000000",
"CPU": "100",
"NW_IN": "100000",
"NW_OUT": "100000"
}
},
{
"brokerId": "0",
"capacity": {
"DISK": "1000000",
"CPU": "100",
"NW_IN": "100000",
"NW_OUT": "100000"
}
},
{
"brokerId": "1",
"capacity": {
"DISK": "1000000",
"CPU": "100",
"NW_IN": "100000",
"NW_OUT": "100000"
}
}
]
}
Example capacity file with JBOD (provided logDirs must be absolute paths):
{
"brokerCapacities":[
{
"brokerId": "-1",
"capacity": {
"DISK": {"/tmp/kafka-logs-1": "400000", "/tmp/kafka-logs-2": "200000", "/tmp/kafka-logs-3": "200000",
"/tmp/kafka-logs-4": "200000", "/tmp/kafka-logs-5": "200000", "/tmp/kafka-logs-6": "200000"},
"CPU": "100",
"NW_IN": "100000",
"NW_OUT": "100000"
}
},
{
"brokerId": "0",
"capacity": {
"DISK": {"/tmp/kafka-logs-1": "350000", "/tmp/kafka-logs-2": "550000"},
"CPU": "100",
"NW_IN": "100000",
"NW_OUT": "100000"
}
},
{
"brokerId": "1",
"capacity": {
"DISK": {"/tmp/kafka-logs": "2000000"},
"CPU": "100",
"NW_IN": "100000",
"NW_OUT": "100000"
}
}
]
}
Example capacity file with number of cores (i.e. see the use of
NUM_CORES_CONFIG
config):
{
"brokerCapacities":[
{
"brokerId": "-1",
"capacity": {
"DISK": {"/tmp/kafka-logs-1": "100000", "/tmp/kafka-logs-2": "100000", "/tmp/kafka-logs-3": "50000",
"/tmp/kafka-logs-4": "50000", "/tmp/kafka-logs-5": "150000", "/tmp/kafka-logs-6": "50000"},
"CPU": {"num.cores": "16"},
"NW_IN": "10000",
"NW_OUT": "10000"
},
"doc": "The default capacity for a broker with multiple logDirs each on a separate heterogeneous disk and num.cores CPU cores."
},
{
"brokerId": "0",
"capacity": {
"DISK": {"/tmp/kafka-logs": "500000"},
"CPU": {"num.cores": "32"},
"NW_IN": "50000",
"NW_OUT": "50000"
},
"doc": "This overrides the capacity for broker 0. This broker is not a JBOD broker, but has num.cores CPU cores."
},
{
"brokerId": "1",
"capacity": {
"DISK": {"/tmp/kafka-logs-1": "250000", "/tmp/kafka-logs-2": "250000"},
"CPU": {"num.cores": "24"},
"NW_IN": "50000",
"NW_OUT": "50000"
},
"doc": "This overrides the capacity for broker 1. This broker is a JBOD broker and has num.cores CPU cores."
}
]
}
The broker id -1 defines the default broker capacity estimate. A broker capacity is overridden if there is a capacity
defined for a particular broker id. In case a broker capacity is missing, the default estimate for a broker capacity
will be used.
The units of the definition are:
- DISK - MB
- CPU - either (1) Percentage (0 - 100) or (2) number of cores (see the use of
NUM_CORES_CONFIG
config)
- NW_IN - KB/s
- NW_OUT - KB/s