E
- The entity class. Note that the entity will be used as a key to HashMaps, so it must have
a valid Object.hashCode()
and Object.equals(Object)
implementation.public class MetricSampleAggregator<E extends Entity> extends LongGenerationed
MetricSamples
for entities
. It uses
a cyclic buffer to keep track of the most recent N (configured) windows. The oldest windows are evicted when
the buffer is full.
Windows are given an absolute ID based on their timestamp since the epoch. As samples come in, they are assigned to the window ID for the configured timestamp.
The MetricSampleAggregator
aggregates the metrics of each entities in each window. The aggregation
can be viewed as two dimensions: per entity and per window.
From per entity's perspective, each entity would have sufficient metrics from all the windows when everything
works fine. However, it is also possible that some metrics may be missing from the MetricSampleAggregator
in one or more windows. If that happens, some Extrapolation
will be used to fill in the missing data.
If none of the Extrapolation
works. We claim the entity is invalid in this window. With above,
a given entity may have the following three states in any given window from the MetricSampleAggregator
's
perspective:
Furthermore, each entity belongs to an aggregation entity group. The aggregation entity group is only used
for metric aggregation purpose. Users can specify the AggregationOptions.Granularity
of the metric aggregation. The
granularity could be one of the following:
AggregationOptions.Granularity.ENTITY
: The validity of the entities in the same aggregation group
are considered independently, i.e. an invalid entity in an aggregation group does not invalidate the other
entities in the same aggregation entity group.
AggregationOptions.Granularity.ENTITY_GROUP
: The validity of the entities in the same aggregation entity group are
considered as an entirety. i.e. a single invalid entity in the aggregation entity group invalidates all
the entities in the same aggregation entity group.
From per window's perspective, for each window, there is a given set of valid entities and entity
groups as described above. The validity of a window depends on the requirements specified in the
AggregationOptions
during the aggregation. More specifically whether the entity coverage (valid entity
ratio) and entity group coverage (valid entity group ratio) meet the requirements.
This class is thread safe.
Constructor and Description |
---|
MetricSampleAggregator(int numWindows,
long windowMs,
byte minSamplesPerWindow,
int completenessCacheSize,
MetricDef metricDef)
Construct the metric sample aggregator.
|
Modifier and Type | Method and Description |
---|---|
boolean |
addSample(MetricSample<E> sample)
Add a sample to the metric aggregator.
|
MetricSampleAggregationResult<E> |
aggregate(long from,
long to,
AggregationOptions<E> options)
Overload of
aggregate(long, long, AggregationOptions, Set)
passes empty set of failed brokers. |
MetricSampleAggregationResult<E> |
aggregate(long from,
long to,
AggregationOptions<E> options,
Set<Integer> failedBrokers)
Aggregate the metric samples in the given period into a
MetricSampleAggregationResult based on the
specified AggregationOptions . |
List<Long> |
allWindows() |
List<Long> |
availableWindows()
Get a list of available windows in the MetricSampleAggregator.
|
MetricSampleCompleteness<E> |
completeness(long from,
long to,
AggregationOptions<E> options)
Override of
completeness(long, long, AggregationOptions, Set)
passes empty set of failed brokers to original method. |
MetricSampleCompleteness<E> |
completeness(long from,
long to,
AggregationOptions<E> options,
Set<Integer> failedBrokerIds)
Get the
MetricSampleCompleteness of the MetricSampleAggregator with the given AggregationOptions
for a given period of time. |
boolean |
maybeInvalidateWindowsBeforeTime(long from)
Invalidate metric windows before a given timestamp.
|
int |
numAvailableWindows()
Get the number of available windows in the MetricSampleAggregator.
|
int |
numAvailableWindows(long from,
long to)
Get the number of available windows in the given time range, excluding the current active window.
|
int |
numSamples()
Get the total number of samples that is currently aggregated by the MetricSampleAggregator.
|
boolean |
onSamplingFinish(long endMs)
When the metric sampling run finishes, this method gets called.
|
Map<E,ValuesAndExtrapolations> |
peekCurrentWindow()
Peek the information for all the available entities of the current window.
|
void |
retainEntityGroup(Set<String> entityGroups)
Keep the given set of entity groups in the MetricSampleAggregator and remove the reset of the entity groups.
|
generation
public MetricSampleAggregator(int numWindows, long windowMs, byte minSamplesPerWindow, int completenessCacheSize, MetricDef metricDef)
numWindows
- the number of windows needed.windowMs
- the size of each window in millisecondsminSamplesPerWindow
- minimum samples per window.completenessCacheSize
- the completeness cache size, i.e. the number of recent completeness query result to
cache.metricDef
- metric definitions.public boolean addSample(MetricSample<E> sample)
sample
- The metric sample to add.public MetricSampleAggregationResult<E> aggregate(long from, long to, AggregationOptions<E> options, Set<Integer> failedBrokers) throws NotEnoughValidWindowsException
MetricSampleAggregationResult
based on the
specified AggregationOptions
.
The aggregation result contains all the entities in AggregationOptions.interestedEntities()
.
For the entities that are completely missing, an empty result is added with all the value set to 0.0 and
all the window indices marked as Extrapolation.NO_VALID_EXTRAPOLATION
.
from
- the starting timestamp of the aggregation period in milliseconds.to
- the end timestamp of the aggregation period in milliseconds.options
- the AggregationOptions
used to perform the aggregation.failedBrokers
- Set of failed(Dead) Brokers.MetricSampleAggregationResult
based on the given AggregationOptions.NotEnoughValidWindowsException
public MetricSampleAggregationResult<E> aggregate(long from, long to, AggregationOptions<E> options) throws NotEnoughValidWindowsException
aggregate(long, long, AggregationOptions, Set)
passes empty set of failed brokers.NotEnoughValidWindowsException
public Map<E,ValuesAndExtrapolations> peekCurrentWindow()
public MetricSampleCompleteness<E> completeness(long from, long to, AggregationOptions<E> options, Set<Integer> failedBrokerIds)
MetricSampleCompleteness
of the MetricSampleAggregator with the given AggregationOptions
for a given period of time. The current active window is excluded.from
- starting time of the period to check.to
- ending time of the period to check.options
- the AggregationOptions
to use for the completeness check.failedBrokerIds
- MetricSampleCompleteness
of the MetricSampleAggregator.public MetricSampleCompleteness<E> completeness(long from, long to, AggregationOptions<E> options)
completeness(long, long, AggregationOptions, Set)
passes empty set of failed brokers to original method.public List<Long> availableWindows()
public int numAvailableWindows()
public int numAvailableWindows(long from, long to)
from
- the starting time of the time range. (inclusive)to
- the end time of the time range. (inclusive)public List<Long> allWindows()
public boolean maybeInvalidateWindowsBeforeTime(long from)
from
- timestamp in epoch millisecondspublic int numSamples()
public void retainEntityGroup(Set<String> entityGroups)
entityGroups
- the entity groups to retain.public boolean onSamplingFinish(long endMs)
endMs
- the end time of the sampling run.