Package org.apache.flink.formats.parquet

Class ParquetColumnarRowInputFormat<SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

java.lang.Object

org.apache.flink.formats.parquet.ParquetVectorizedInputFormat<org.apache.flink.table.data.RowData,SplitT>

org.apache.flink.formats.parquet.ParquetColumnarRowInputFormat<SplitT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.java.typeutils.ResultTypeQueryable<org.apache.flink.table.data.RowData>, org.apache.flink.connector.file.src.reader.BulkFormat<org.apache.flink.table.data.RowData,SplitT>, org.apache.flink.table.connector.format.FileBasedStatisticsReportableInputFormat

public class ParquetColumnarRowInputFormat<SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>
extends ParquetVectorizedInputFormat<org.apache.flink.table.data.RowData,SplitT>
implements org.apache.flink.table.connector.format.FileBasedStatisticsReportableInputFormat

A ParquetVectorizedInputFormat to provide RowData iterator. Using ColumnarRowData to provide a row view of column batch.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.formats.parquet.ParquetVectorizedInputFormat

ParquetVectorizedInputFormat.ParquetReaderBatch<T>

Nested classes/interfaces inherited from interface org.apache.flink.connector.file.src.reader.BulkFormat

org.apache.flink.connector.file.src.reader.BulkFormat.Reader<T extends Object>, org.apache.flink.connector.file.src.reader.BulkFormat.RecordIterator<T extends Object>

Field Summary

Fields inherited from class org.apache.flink.formats.parquet.ParquetVectorizedInputFormat

hadoopConfig, isUtcTimestamp

Constructor Summary

Constructors

Constructor	Description
ParquetColumnarRowInputFormat(org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.flink.table.types.logical.RowType projectedType, org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> producedTypeInfo, int batchSize, boolean isUtcTimestamp, boolean isCaseSensitive)	Constructor to create parquet format without extra fields.

Method Summary

Modifier and Type	Method	Description
static <SplitT extends org.apache.flink.connector.file.src.FileSourceSplit> ParquetColumnarRowInputFormat<SplitT>	createPartitionedFormat(org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.flink.table.types.logical.RowType producedRowType, org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> producedTypeInfo, List<String> partitionKeys, org.apache.flink.connector.file.table.PartitionFieldExtractor<SplitT> extractor, int batchSize, boolean isUtcTimestamp, boolean isCaseSensitive)	Create a partitioned ParquetColumnarRowInputFormat, the partition columns can be generated by Path.
protected ParquetVectorizedInputFormat.ParquetReaderBatch<org.apache.flink.table.data.RowData>	createReaderBatch(org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector[] writableVectors, org.apache.flink.table.data.columnar.vector.VectorizedColumnBatch columnarBatch, org.apache.flink.connector.file.src.util.Pool.Recycler<ParquetVectorizedInputFormat.ParquetReaderBatch<org.apache.flink.table.data.RowData>> recycler)
org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData>	getProducedType()
protected int	numBatchesToCirculate(org.apache.flink.configuration.Configuration config)
org.apache.flink.table.plan.stats.TableStats	reportStatistics(List<org.apache.flink.core.fs.Path> files, org.apache.flink.table.types.DataType producedDataType)

Methods inherited from class org.apache.flink.formats.parquet.ParquetVectorizedInputFormat

createReader, isSplittable, restoreReader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParquetColumnarRowInputFormat

public ParquetColumnarRowInputFormat(org.apache.hadoop.conf.Configuration hadoopConfig,
                                     org.apache.flink.table.types.logical.RowType projectedType,
                                     org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> producedTypeInfo,
                                     int batchSize,
                                     boolean isUtcTimestamp,
                                     boolean isCaseSensitive)

Constructor to create parquet format without extra fields.

Method Detail

numBatchesToCirculate

protected int numBatchesToCirculate(org.apache.flink.configuration.Configuration config)

Overrides:

numBatchesToCirculate in class ParquetVectorizedInputFormat<org.apache.flink.table.data.RowData,SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

createReaderBatch

protected ParquetVectorizedInputFormat.ParquetReaderBatch<org.apache.flink.table.data.RowData> createReaderBatch(org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector[] writableVectors,
                                                                                                                 org.apache.flink.table.data.columnar.vector.VectorizedColumnBatch columnarBatch,
                                                                                                                 org.apache.flink.connector.file.src.util.Pool.Recycler<ParquetVectorizedInputFormat.ParquetReaderBatch<org.apache.flink.table.data.RowData>> recycler)

Specified by:

createReaderBatch in class ParquetVectorizedInputFormat<org.apache.flink.table.data.RowData,SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

Parameters:

writableVectors - vectors to be write

columnarBatch - vectors to be read

recycler - batch recycler

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.connector.file.src.reader.BulkFormat<org.apache.flink.table.data.RowData,SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

reportStatistics

public org.apache.flink.table.plan.stats.TableStats reportStatistics(List<org.apache.flink.core.fs.Path> files,
                                                                     org.apache.flink.table.types.DataType producedDataType)

Specified by:

reportStatistics in interface org.apache.flink.table.connector.format.FileBasedStatisticsReportableInputFormat

createPartitionedFormat

public static <SplitT extends org.apache.flink.connector.file.src.FileSourceSplit> ParquetColumnarRowInputFormat<SplitT> createPartitionedFormat(org.apache.hadoop.conf.Configuration hadoopConfig,
                                                                                                                                                 org.apache.flink.table.types.logical.RowType producedRowType,
                                                                                                                                                 org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> producedTypeInfo,
                                                                                                                                                 List<String> partitionKeys,
                                                                                                                                                 org.apache.flink.connector.file.table.PartitionFieldExtractor<SplitT> extractor,
                                                                                                                                                 int batchSize,
                                                                                                                                                 boolean isUtcTimestamp,
                                                                                                                                                 boolean isCaseSensitive)

Create a partitioned ParquetColumnarRowInputFormat, the partition columns can be generated by Path.