Package org.apache.flink.api.common.operators

Class GenericDataSourceBase<OUT,T extends InputFormat<OUT,?>>

java.lang.Object

org.apache.flink.api.common.operators.Operator<OUT>

org.apache.flink.api.common.operators.GenericDataSourceBase<OUT,T>

Type Parameters:

OUT - The output type of the data source

T - The type of input format invoked by instances of this data source.

All Implemented Interfaces:

Visitable<Operator<?>>

@Internal
public class GenericDataSourceBase<OUT,T extends InputFormat<OUT,?>>
extends Operator<OUT>

Abstract superclass for data sources in a Pact plan.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	GenericDataSourceBase.SplitDataProperties<T>

Field Summary

Fields

Modifier and Type	Field	Description
protected UserCodeWrapper<? extends T>	formatWrapper
protected String	statisticsKey

Fields inherited from class org.apache.flink.api.common.operators.Operator

compilerHints, name, operatorInfo, parameters

Constructor Summary

Constructors

Constructor	Description
GenericDataSourceBase(Class<? extends T> format, OperatorInformation<OUT> operatorInfo)	Creates a new instance for the given file using the given input format, using the default name.
GenericDataSourceBase(Class<? extends T> format, OperatorInformation<OUT> operatorInfo, String name)	Creates a new instance for the given file using the given input format.
GenericDataSourceBase(T format, OperatorInformation<OUT> operatorInfo)	Creates a new instance for the given file using the given input format, using the default name.
GenericDataSourceBase(T format, OperatorInformation<OUT> operatorInfo, String name)	Creates a new instance for the given file using the given input format.

Method Summary

Modifier and Type	Method	Description
void	accept(Visitor<Operator<?>> visitor)	Accepts the visitor and applies it this instance.
protected List<OUT>	executeOnCollections(RuntimeContext ctx, ExecutionConfig executionConfig)
UserCodeWrapper<? extends T>	getFormatWrapper()	Gets the class describing the input format.
GenericDataSourceBase.SplitDataProperties<OUT>	getSplitDataProperties()	Returns the data properties of this data source's splits.
String	getStatisticsKey()	Gets the key under which statistics about this data source may be obtained from the statistics cache.
UserCodeWrapper<? extends T>	getUserCodeWrapper()	Gets the class describing the input format.
void	setSplitDataProperties(GenericDataSourceBase.SplitDataProperties<OUT> splitDataProperties)	Sets properties of input splits for this data source.
void	setStatisticsKey(String statisticsKey)	Sets the key under which statistics about this data source may be obtained from the statistics cache.
String	toString()

Methods inherited from class org.apache.flink.api.common.operators.Operator

createUnionCascade, createUnionCascade, createUnionCascade, getCompilerHints, getMinResources, getName, getOperatorInfo, getParallelism, getParameters, getPreferredResources, setName, setParallelism, setParameter, setParameter, setParameter, setResources

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

formatWrapper

protected final UserCodeWrapper<? extends T extends InputFormat<OUT,?>> formatWrapper

statisticsKey

protected String statisticsKey

Constructor Detail

GenericDataSourceBase

public GenericDataSourceBase(T format,
                             OperatorInformation<OUT> operatorInfo,
                             String name)

Creates a new instance for the given file using the given input format.

Parameters:

format - The InputFormat implementation used to read the data.

operatorInfo - The type information for the operator.

name - The given name for the Pact, used in plans, logs and progress messages.

GenericDataSourceBase

public GenericDataSourceBase(T format,
                             OperatorInformation<OUT> operatorInfo)

Creates a new instance for the given file using the given input format, using the default name.

Parameters:

format - The InputFormat implementation used to read the data.

operatorInfo - The type information for the operator.

GenericDataSourceBase

public GenericDataSourceBase(Class<? extends T> format,
                             OperatorInformation<OUT> operatorInfo,
                             String name)

Creates a new instance for the given file using the given input format.

Parameters:

format - The InputFormat implementation used to read the data.

operatorInfo - The type information for the operator.

name - The given name for the Pact, used in plans, logs and progress messages.

GenericDataSourceBase

public GenericDataSourceBase(Class<? extends T> format,
                             OperatorInformation<OUT> operatorInfo)

Creates a new instance for the given file using the given input format, using the default name.

Parameters:

format - The InputFormat implementation used to read the data.

operatorInfo - The type information for the operator.

Method Detail

getFormatWrapper

public UserCodeWrapper<? extends T> getFormatWrapper()

Gets the class describing the input format.

Returns:

The class describing the input format.

getUserCodeWrapper

public UserCodeWrapper<? extends T> getUserCodeWrapper()

Gets the class describing the input format.

This method is basically identical to getFormatWrapper().

Overrides:

getUserCodeWrapper in class Operator<OUT>

Returns:

The class describing the input format.

See Also:

Operator.getUserCodeWrapper()

getStatisticsKey

public String getStatisticsKey()

Gets the key under which statistics about this data source may be obtained from the statistics cache.

Returns:

The statistics cache key.

setStatisticsKey

public void setStatisticsKey(String statisticsKey)

Sets the key under which statistics about this data source may be obtained from the statistics cache. Useful for testing purposes, when providing mock statistics.

Parameters:

statisticsKey - The key for the statistics object.

setSplitDataProperties

public void setSplitDataProperties(GenericDataSourceBase.SplitDataProperties<OUT> splitDataProperties)

Sets properties of input splits for this data source. Split properties can help to generate more efficient execution plans.
IMPORTANT: Providing wrong split data properties can cause wrong results!

Parameters:

splitDataProperties - The data properties of this data source's splits.

getSplitDataProperties

public GenericDataSourceBase.SplitDataProperties<OUT> getSplitDataProperties()

Returns the data properties of this data source's splits.

Returns:

The data properties of this data source's splits or null if no properties have been set.

accept

public void accept(Visitor<Operator<?>> visitor)

Accepts the visitor and applies it this instance. Since the data sources have no inputs, no recursive descend happens. The visitors pre-visit method is called and, if returning true, the post-visit method is called.

Parameters:

visitor - The visitor.

See Also:

Visitable.accept(org.apache.flink.util.Visitor)

executeOnCollections

protected List<OUT> executeOnCollections(RuntimeContext ctx,
                                         ExecutionConfig executionConfig)
                                  throws Exception

Throws:

Exception

toString

public String toString()

Overrides:

toString in class Operator<OUT>