Package org.apache.flink.streaming.api.operators.python

Class AbstractPythonFunctionOperator<OUT>

  • All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.SetupableStreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<OUT>
    Direct Known Subclasses:
    AbstractEmbeddedPythonFunctionOperator, AbstractExternalPythonFunctionOperator, DelegateOperatorTransformation.DelegateOperator
    @Internal
public abstract class AbstractPythonFunctionOperator<OUT>
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
    Base class for all stream operators to execute Python functions.
    See Also:
    Serialized Form

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected Runnable bundleFinishedCallback
      Callback to be executed after the current bundle was finished.
      protected org.apache.flink.configuration.Configuration config  
      protected int elementCount
      Number of processed elements in the current bundle.
      protected long lastFinishBundleTime
      Time that the last bundle was finished.
      protected int maxBundleSize
      Max number of elements to include in a bundle.
      protected boolean systemEnvEnabled  

      • Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        chainingStrategy, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

    • Method Summary

      All Methods Instance Methods Abstract Methods Concrete Methods 
      Modifier and Type Method Description
      protected void checkInvokeFinishBundleByCount()
      Checks whether to invoke finishBundle by elements count.
      void close()  
      protected abstract PythonEnvironmentManager createPythonEnvironmentManager()  
      protected void drainUnregisteredTimers()  
      void finish()  
      org.apache.flink.configuration.Configuration getConfiguration()
      Returns the Configuration.
      protected FlinkMetricContainer getFlinkMetricContainer()  
      protected abstract void invokeFinishBundle()  
      boolean isBundleFinished()
      Returns whether the bundle is finished.
      void open()  
      void prepareSnapshotPreBarrier​(long checkpointId)  
      void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)  
      void setCurrentKey​(Object key)  

      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useSplittableTimers

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

        hasKeyContext

      • Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

        getOperatorAttributes

    • Field Detail

      • config

        protected final org.apache.flink.configuration.Configuration config

      • systemEnvEnabled

        protected transient boolean systemEnvEnabled

      • maxBundleSize

        protected transient int maxBundleSize
        Max number of elements to include in a bundle.

      • elementCount

        protected transient int elementCount
        Number of processed elements in the current bundle.

      • lastFinishBundleTime

        protected transient long lastFinishBundleTime
        Time that the last bundle was finished.

      • bundleFinishedCallback

        protected transient Runnable bundleFinishedCallback
        Callback to be executed after the current bundle was finished.

    • Constructor Detail

      • AbstractPythonFunctionOperator

        public AbstractPythonFunctionOperator​(org.apache.flink.configuration.Configuration config)

    • Method Detail

      • open

        public void open()
          throws Exception
        Specified by:
        open in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>
        Overrides:
        open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
        Throws:
        Exception

      • finish

        public void finish()
            throws Exception
        Specified by:
        finish in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>
        Overrides:
        finish in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
        Throws:
        Exception

      • close

        public void close()
           throws Exception
        Specified by:
        close in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>
        Overrides:
        close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
        Throws:
        Exception

      • prepareSnapshotPreBarrier

        public void prepareSnapshotPreBarrier​(long checkpointId)
                               throws Exception
        Specified by:
        prepareSnapshotPreBarrier in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>
        Overrides:
        prepareSnapshotPreBarrier in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
        Throws:
        Exception

      • processWatermark

        public void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws Exception
        Overrides:
        processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
        Throws:
        Exception

      • setCurrentKey

        public void setCurrentKey​(Object key)
        Specified by:
        setCurrentKey in interface org.apache.flink.streaming.api.operators.KeyContext
        Overrides:
        setCurrentKey in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

      • isBundleFinished

        public boolean isBundleFinished()
        Returns whether the bundle is finished.

      • getConfiguration

        public org.apache.flink.configuration.Configuration getConfiguration()
        Returns the Configuration.

      • invokeFinishBundle

        protected abstract void invokeFinishBundle()
                                    throws Exception
        Throws:
        Exception

      • drainUnregisteredTimers

        protected void drainUnregisteredTimers()

      • checkInvokeFinishBundleByCount

        protected void checkInvokeFinishBundleByCount()
                                       throws Exception
        Checks whether to invoke finishBundle by elements count. Called in processElement.
        Throws:
        Exception