Package org.apache.flink.table.runtime.operators.python.aggregate

Class AbstractPythonStreamAggregateOperator

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator<OUT>

org.apache.flink.table.runtime.operators.python.AbstractOneInputPythonFunctionOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>

org.apache.flink.table.runtime.operators.python.aggregate.AbstractPythonStreamAggregateOperator

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<org.apache.flink.table.data.RowData>

Direct Known Subclasses:

AbstractPythonStreamGroupAggregateOperator, PythonStreamGroupWindowAggregateOperator

@Internal
public abstract class AbstractPythonStreamAggregateOperator
extends AbstractOneInputPythonFunctionOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>

Base class for AbstractPythonStreamGroupAggregateOperator and PythonStreamGroupWindowAggregateOperator.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
protected org.apache.flink.core.memory.ByteArrayInputStreamWithPos	bais	Reusable InputStream used to holding the execution results to be deserialized.
protected org.apache.flink.core.memory.DataInputViewStreamWrapper	baisWrapper	InputStream Wrapper.
protected org.apache.flink.core.memory.ByteArrayOutputStreamWithPos	baos	Reusable OutputStream used to holding the serialized input elements.
protected org.apache.flink.core.memory.DataOutputViewStreamWrapper	baosWrapper	OutputStream Wrapper.
protected org.apache.flink.table.types.logical.RowType	inputType	The input logical type.
protected org.apache.flink.table.types.logical.RowType	outputType	The output logical type.
protected StreamRecordRowDataWrappingCollector	rowDataWrapper	The collector used to collect records.
protected org.apache.flink.table.types.logical.RowType	userDefinedFunctionInputType	The user-defined function input logical type.
protected org.apache.flink.table.types.logical.RowType	userDefinedFunctionOutputType	The user-defined function output logical type.

Fields inherited from class org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator

pythonFunctionRunner

Fields inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

bundleFinishedCallback, config, elementCount, lastFinishBundleTime, maxBundleSize, systemEnvEnabled

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

Constructor Summary

Constructors

Constructor

Description

AbstractPythonStreamAggregateOperator(org.apache.flink.configuration.Configuration config, org.apache.flink.table.types.logical.RowType inputType, org.apache.flink.table.types.logical.RowType outputType, org.apache.flink.table.functions.python.PythonAggregateFunctionInfo[] aggregateFunctions, org.apache.flink.table.runtime.dataview.DataViewSpec[][] dataViewSpecs, int[] grouping, int indexOfCountStar, boolean generateUpdateBefore)

Method Summary

Modifier and Type	Method	Description
FlinkFnApi.CoderInfoDescriptor	createInputCoderInfoDescriptor(org.apache.flink.table.types.logical.RowType runnerInputType)
FlinkFnApi.CoderInfoDescriptor	createOutputCoderInfoDescriptor(org.apache.flink.table.types.logical.RowType runnerOutType)
PythonFunctionRunner	createPythonFunctionRunner()	Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.
abstract org.apache.flink.table.types.logical.RowType	createUserDefinedFunctionInputType()
abstract org.apache.flink.table.types.logical.RowType	createUserDefinedFunctionOutputType()
Object	getCurrentKey()
abstract String	getFunctionUrn()
protected org.apache.flink.table.types.logical.RowType	getKeyType()
org.apache.flink.table.functions.python.PythonEnv	getPythonEnv()	Returns the PythonEnv used to create PythonEnvironmentManager..
protected FlinkFnApi.UserDefinedAggregateFunctions	getUserDefinedFunctionsProto()	Gets the proto representation of the Python user-defined aggregate functions to be executed.
void	open()
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.flink.table.data.RowData> element)
abstract void	processElementInternal(org.apache.flink.table.data.RowData value)
void	setCurrentKey(Object key)	As the beam state gRPC service will access the KeyedStateBackend in parallel with this operator, we must override this method to prevent changing the current key of the KeyedStateBackend while the beam service is handling requests.

Methods inherited from class org.apache.flink.table.runtime.operators.python.AbstractOneInputPythonFunctionOperator

endInput

Methods inherited from class org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator

close, createPythonEnvironmentManager, drainUnregisteredTimers, emitResult, emitResults, invokeFinishBundle

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

checkInvokeFinishBundleByCount, finish, getConfiguration, getFlinkMetricContainer, isBundleFinished, prepareSnapshotPreBarrier, processWatermark

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useSplittableTimers

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processRecordAttributes, processWatermark, processWatermarkStatus

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Field Detail

inputType

protected final org.apache.flink.table.types.logical.RowType inputType

The input logical type.

outputType

protected final org.apache.flink.table.types.logical.RowType outputType

The output logical type.

userDefinedFunctionInputType

protected transient org.apache.flink.table.types.logical.RowType userDefinedFunctionInputType

The user-defined function input logical type.

userDefinedFunctionOutputType

protected transient org.apache.flink.table.types.logical.RowType userDefinedFunctionOutputType

The user-defined function output logical type.

bais

protected transient org.apache.flink.core.memory.ByteArrayInputStreamWithPos bais

Reusable InputStream used to holding the execution results to be deserialized.

baisWrapper

protected transient org.apache.flink.core.memory.DataInputViewStreamWrapper baisWrapper

InputStream Wrapper.

baos

protected transient org.apache.flink.core.memory.ByteArrayOutputStreamWithPos baos

Reusable OutputStream used to holding the serialized input elements.

baosWrapper

protected transient org.apache.flink.core.memory.DataOutputViewStreamWrapper baosWrapper

OutputStream Wrapper.

rowDataWrapper

protected transient StreamRecordRowDataWrappingCollector rowDataWrapper

The collector used to collect records.

Constructor Detail

AbstractPythonStreamAggregateOperator

public AbstractPythonStreamAggregateOperator(org.apache.flink.configuration.Configuration config,
                                             org.apache.flink.table.types.logical.RowType inputType,
                                             org.apache.flink.table.types.logical.RowType outputType,
                                             org.apache.flink.table.functions.python.PythonAggregateFunctionInfo[] aggregateFunctions,
                                             org.apache.flink.table.runtime.dataview.DataViewSpec[][] dataViewSpecs,
                                             int[] grouping,
                                             int indexOfCountStar,
                                             boolean generateUpdateBefore)

Method Detail

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>

Overrides:

open in class AbstractExternalPythonFunctionOperator<org.apache.flink.table.data.RowData>

Throws:

Exception

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.flink.table.data.RowData> element)
                    throws Exception

Throws:

Exception

createPythonFunctionRunner

public PythonFunctionRunner createPythonFunctionRunner()
                                                throws Exception

Description copied from class: AbstractExternalPythonFunctionOperator

Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.

Specified by:

createPythonFunctionRunner in class AbstractExternalPythonFunctionOperator<org.apache.flink.table.data.RowData>

Throws:

Exception

setCurrentKey

public void setCurrentKey(Object key)

As the beam state gRPC service will access the KeyedStateBackend in parallel with this operator, we must override this method to prevent changing the current key of the KeyedStateBackend while the beam service is handling requests.

Specified by:

setCurrentKey in interface org.apache.flink.streaming.api.operators.KeyContext

Overrides:

setCurrentKey in class AbstractPythonFunctionOperator<org.apache.flink.table.data.RowData>

getCurrentKey

public Object getCurrentKey()

Specified by:

getCurrentKey in interface org.apache.flink.streaming.api.operators.KeyContext

Overrides:

getCurrentKey in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>

getPythonEnv

public org.apache.flink.table.functions.python.PythonEnv getPythonEnv()

Description copied from class: AbstractExternalPythonFunctionOperator

Returns the PythonEnv used to create PythonEnvironmentManager..

Specified by:

getPythonEnv in class AbstractExternalPythonFunctionOperator<org.apache.flink.table.data.RowData>

getKeyType

protected org.apache.flink.table.types.logical.RowType getKeyType()

getUserDefinedFunctionsProto

protected FlinkFnApi.UserDefinedAggregateFunctions getUserDefinedFunctionsProto()

Gets the proto representation of the Python user-defined aggregate functions to be executed.

getFunctionUrn

public abstract String getFunctionUrn()

processElementInternal

public abstract void processElementInternal(org.apache.flink.table.data.RowData value)
                                     throws Exception

Throws:

Exception

createUserDefinedFunctionInputType

public abstract org.apache.flink.table.types.logical.RowType createUserDefinedFunctionInputType()

createUserDefinedFunctionOutputType

public abstract org.apache.flink.table.types.logical.RowType createUserDefinedFunctionOutputType()

createInputCoderInfoDescriptor

public FlinkFnApi.CoderInfoDescriptor createInputCoderInfoDescriptor(org.apache.flink.table.types.logical.RowType runnerInputType)

createOutputCoderInfoDescriptor

public FlinkFnApi.CoderInfoDescriptor createOutputCoderInfoDescriptor(org.apache.flink.table.types.logical.RowType runnerOutType)