Package org.apache.flink.table.runtime.operators.window.tvf.common

Class WindowAggOperator<K,​W>

  • All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.flink.table.data.RowData,​org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.Triggerable<K,​W>
    @Internal
public final class WindowAggOperator<K,​W>
extends TableStreamOperator<org.apache.flink.table.data.RowData>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.flink.table.data.RowData,​org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.Triggerable<K,​W>, org.apache.flink.streaming.api.operators.KeyContext
    We divide windows into 2 categories: Aligned Windows and Unaligned Windows.

    Concept of Aligned Window and Unaligned Window

    Aligned Windows are windows have predetermined window boundaries and windows can be divided into finite number of non-overlapping chunks. The boundary of an aligned window is determined independently from the time characteristic of the data stream, or messages it receives. For example, hopping (sliding) window is an aligned window as the window boundaries are predetermined based on the window size and slide. Aligned windows include tumbling, hopping, cumulative windows.

    Unaligned Windows are windows determined dynamically based on elements. For example, session window is an unaligned window as the window boundaries are determined based on the messages timestamps and their correlations. Currently, unaligned windows include session window only.

    Because aligned windows can be divided into finite number of non-overlapping chunks (a.k.a. slices), which can apply efficient processing to share intermediate results. 

     Window
 |
 +-- Aligned Window (Slicing Window)
 |    |
 |    +-- Tumble (Slice Unshared Window)
 |    |
 |    +-- Hop (Slice Shared Window)
 |    |
 |    +-- Cumulate (Slice Shared Window)
 |
 +-- Unaligned Window (Unslice Window)
      |
      +-- Session

    Note: currently, WindowAggOperator doesn't support early-fire and late-arrival. Thus late elements (elements belong to emitted windows) will be simply dropped.

    See more in SlicingWindowProcessor and UnslicingWindowProcessor.

    See Also:
    Serialized Form

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected org.apache.flink.streaming.api.operators.TimestampedCollector<org.apache.flink.table.data.RowData> collector
      This is used for emitting elements with a given timestamp.
      protected org.apache.flink.streaming.api.operators.InternalTimerService<W> internalTimerService
      The service to register timers.
      protected long lastTriggeredProcessingTime
      The tracked processing time triggered last time.
      protected org.apache.flink.metrics.Meter lateRecordsDroppedRate  
      protected org.apache.flink.metrics.Counter numLateRecordsDropped  
      protected org.apache.flink.metrics.Gauge<Long> watermarkLatency  
      protected org.apache.flink.api.common.state.ListState<Long> watermarkState
      The operator state to store watermark.
      protected WindowProcessor<W> windowProcessor
      The concrete window operator implementation.

      • Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        chainingStrategy, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      org.apache.flink.metrics.Counter getNumLateRecordsDropped()  
      org.apache.flink.metrics.Gauge<Long> getWatermarkLatency()  
      void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)  
      void onEventTime​(org.apache.flink.streaming.api.operators.InternalTimer<K,​W> timer)  
      void onProcessingTime​(org.apache.flink.streaming.api.operators.InternalTimer<K,​W> timer)  
      void open()  
      void prepareSnapshotPreBarrier​(long checkpointId)  
      void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.flink.table.data.RowData> element)  
      void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)  
      void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)  

      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        finish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState

      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted, notifyCheckpointComplete

      • Methods inherited from interface org.apache.flink.streaming.api.operators.Input

        processLatencyMarker, processRecordAttributes, processWatermarkStatus

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

        getCurrentKey, setCurrentKey

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

        hasKeyContext

      • Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

        setKeyContextElement

      • Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

        finish, getMetricGroup, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState

    • Field Detail

      • windowProcessor

        protected final WindowProcessor<W> windowProcessor
        The concrete window operator implementation.

      • collector

        protected transient org.apache.flink.streaming.api.operators.TimestampedCollector<org.apache.flink.table.data.RowData> collector
        This is used for emitting elements with a given timestamp.

      • internalTimerService

        protected transient org.apache.flink.streaming.api.operators.InternalTimerService<W> internalTimerService
        The service to register timers.

      • lastTriggeredProcessingTime

        protected transient long lastTriggeredProcessingTime
        The tracked processing time triggered last time.

      • watermarkState

        protected transient org.apache.flink.api.common.state.ListState<Long> watermarkState
        The operator state to store watermark.

      • numLateRecordsDropped

        protected transient org.apache.flink.metrics.Counter numLateRecordsDropped

      • lateRecordsDroppedRate

        protected transient org.apache.flink.metrics.Meter lateRecordsDroppedRate

      • watermarkLatency

        protected transient org.apache.flink.metrics.Gauge<Long> watermarkLatency

    • Constructor Detail

      • WindowAggOperator

        public WindowAggOperator​(WindowProcessor<W> windowProcessor)

    • Method Detail

      • open

        public void open()
          throws Exception
        Specified by:
        open in interface org.apache.flink.streaming.api.operators.StreamOperator<K>
        Overrides:
        open in class TableStreamOperator<org.apache.flink.table.data.RowData>
        Throws:
        Exception

      • initializeState

        public void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception
        Specified by:
        initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
        Throws:
        Exception

      • snapshotState

        public void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception
        Specified by:
        snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
        Throws:
        Exception

      • close

        public void close()
           throws Exception
        Specified by:
        close in interface org.apache.flink.streaming.api.operators.StreamOperator<K>
        Overrides:
        close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
        Throws:
        Exception

      • processElement

        public void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.flink.table.data.RowData> element)
                    throws Exception
        Specified by:
        processElement in interface org.apache.flink.streaming.api.operators.Input<K>
        Throws:
        Exception

      • processWatermark

        public void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws Exception
        Specified by:
        processWatermark in interface org.apache.flink.streaming.api.operators.Input<K>
        Overrides:
        processWatermark in class TableStreamOperator<org.apache.flink.table.data.RowData>
        Throws:
        Exception

      • onEventTime

        public void onEventTime​(org.apache.flink.streaming.api.operators.InternalTimer<K,​W> timer)
                 throws Exception
        Specified by:
        onEventTime in interface org.apache.flink.streaming.api.operators.Triggerable<K,​W>
        Throws:
        Exception

      • onProcessingTime

        public void onProcessingTime​(org.apache.flink.streaming.api.operators.InternalTimer<K,​W> timer)
                      throws Exception
        Specified by:
        onProcessingTime in interface org.apache.flink.streaming.api.operators.Triggerable<K,​W>
        Throws:
        Exception

      • prepareSnapshotPreBarrier

        public void prepareSnapshotPreBarrier​(long checkpointId)
                               throws Exception
        Specified by:
        prepareSnapshotPreBarrier in interface org.apache.flink.streaming.api.operators.StreamOperator<K>
        Overrides:
        prepareSnapshotPreBarrier in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
        Throws:
        Exception

      • getNumLateRecordsDropped

        @VisibleForTesting
public org.apache.flink.metrics.Counter getNumLateRecordsDropped()

      • getWatermarkLatency

        @VisibleForTesting
public org.apache.flink.metrics.Gauge<Long> getWatermarkLatency()