Class AbstractPythonScalarFunctionOperator

  • All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.flink.table.data.RowData,​org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<org.apache.flink.table.data.RowData>
    Direct Known Subclasses:
    ArrowPythonScalarFunctionOperator, PythonScalarFunctionOperator

    @Internal
    public abstract class AbstractPythonScalarFunctionOperator
    extends AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,​org.apache.flink.table.data.RowData,​org.apache.flink.table.data.RowData>
    Base class for all stream operators to execute Python ScalarFunctions. It executes the Python ScalarFunctions in separate Python execution environment.

    The inputs are assumed as the following format: {{{ +------------------+--------------+ | forwarded fields | extra fields | +------------------+--------------+ }}}.

    The Python UDFs may take input columns directly from the input row or the execution result of Java UDFs: 1) The input columns from the input row can be referred from the 'forwarded fields'; 2) The Java UDFs will be computed and the execution results can be referred from the 'extra fields'.

    The outputs will be as the following format: {{{ +------------------+-------------------------+ | forwarded fields | scalar function results | +------------------+-------------------------+ }}}.

    See Also:
    Serialized Form
    • Nested Class Summary

      • Nested classes/interfaces inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        org.apache.flink.streaming.api.operators.AbstractStreamOperator.OutputAdjustment<OUT extends Object>
    • Constructor Summary

      Constructors 
      Constructor Description
      AbstractPythonScalarFunctionOperator​(org.apache.flink.configuration.Configuration config, org.apache.flink.table.functions.python.PythonFunctionInfo[] scalarFunctions, org.apache.flink.table.types.logical.RowType inputType, org.apache.flink.table.types.logical.RowType udfInputType, org.apache.flink.table.types.logical.RowType udfOutputType, org.apache.flink.table.runtime.generated.GeneratedProjection udfInputGeneratedProjection, org.apache.flink.table.runtime.generated.GeneratedProjection forwardedFieldGeneratedProjection)  
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void bufferInput​(org.apache.flink.table.data.RowData input)
      Buffers the specified input, it will be used to construct the operator result together with the user-defined function execution result.
      FlinkFnApi.UserDefinedFunctions createUserDefinedFunctionsProto()
      Gets the proto representation of the Python user-defined functions to be executed.
      org.apache.flink.table.data.RowData getFunctionInput​(org.apache.flink.table.data.RowData element)  
      String getFunctionUrn()  
      org.apache.flink.table.functions.python.PythonEnv getPythonEnv()
      Returns the PythonEnv used to create PythonEnvironmentManager..
      void open()  
      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        beforeInitializeStateHandler, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isAsyncKeyOrderedProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useInterruptibleTimers
      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted, notifyCheckpointComplete
      • Methods inherited from interface org.apache.flink.streaming.api.operators.Input

        processLatencyMarker, processRecordAttributes, processWatermark, processWatermark, processWatermarkStatus
      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

        getCurrentKey, setCurrentKey
      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

        hasKeyContext
      • Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

        setKeyContextElement
      • Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

        close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
    • Field Detail

      • scalarFunctions

        protected final org.apache.flink.table.functions.python.PythonFunctionInfo[] scalarFunctions
        The Python ScalarFunctions to be executed.
      • reuseJoinedRow

        protected transient org.apache.flink.table.data.utils.JoinedRowData reuseJoinedRow
        The JoinedRowData reused holding the execution result.
    • Constructor Detail

      • AbstractPythonScalarFunctionOperator

        public AbstractPythonScalarFunctionOperator​(org.apache.flink.configuration.Configuration config,
                                                    org.apache.flink.table.functions.python.PythonFunctionInfo[] scalarFunctions,
                                                    org.apache.flink.table.types.logical.RowType inputType,
                                                    org.apache.flink.table.types.logical.RowType udfInputType,
                                                    org.apache.flink.table.types.logical.RowType udfOutputType,
                                                    org.apache.flink.table.runtime.generated.GeneratedProjection udfInputGeneratedProjection,
                                                    org.apache.flink.table.runtime.generated.GeneratedProjection forwardedFieldGeneratedProjection)
    • Method Detail

      • open

        public void open()
                  throws Exception
        Specified by:
        open in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>
        Overrides:
        open in class AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,​org.apache.flink.table.data.RowData,​org.apache.flink.table.data.RowData>
        Throws:
        Exception
      • bufferInput

        public void bufferInput​(org.apache.flink.table.data.RowData input)
        Description copied from class: AbstractStatelessFunctionOperator
        Buffers the specified input, it will be used to construct the operator result together with the user-defined function execution result.
        Specified by:
        bufferInput in class AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,​org.apache.flink.table.data.RowData,​org.apache.flink.table.data.RowData>
      • getFunctionInput

        public org.apache.flink.table.data.RowData getFunctionInput​(org.apache.flink.table.data.RowData element)
        Specified by:
        getFunctionInput in class AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,​org.apache.flink.table.data.RowData,​org.apache.flink.table.data.RowData>