Package org.apache.flink.runtime.io.network.partition.hybrid

Class HsSubpartitionFileReaderImpl

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.HsSubpartitionFileReaderImpl

All Implemented Interfaces:

Comparable<HsSubpartitionFileReader>, HsDataView, HsSubpartitionFileReader

public class HsSubpartitionFileReaderImpl
extends Object
implements HsSubpartitionFileReader

Default implementation of HsSubpartitionFileReader.

Note: This class is not thread safe.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	HsSubpartitionFileReaderImpl.BufferIndexOrError	Indicates a buffer with index or an error.
static class	HsSubpartitionFileReaderImpl.Factory	Factory of HsSubpartitionFileReader.

Constructor Summary

Constructors

Constructor	Description
HsSubpartitionFileReaderImpl(int subpartitionId, HsConsumerId consumerId, FileChannel dataFileChannel, HsSubpartitionConsumerInternalOperations operations, HsFileDataIndex dataIndex, int maxBufferReadAhead, java.util.function.Consumer<HsSubpartitionFileReader> fileReaderReleaser, ByteBuffer headerBuf)

Method Summary

Modifier and Type	Method	Description
int	compareTo(HsSubpartitionFileReader that)	Provides priority calculation logic for io scheduler.
Optional<ResultSubpartition.BufferAndBacklog>	consumeBuffer(int nextBufferToConsume, Collection<Buffer> buffersToRecycle)	Try to consume next buffer.
boolean	equals(Object o)
void	fail(Throwable failureCause)	Fail this HsSubpartitionFileReader caused by failureCause.
int	getBacklog()	Get the number of buffers whose Buffer.DataType is buffer.
Deque<HsSubpartitionFileReaderImpl.BufferIndexOrError>	getLoadedBuffers()
int	hashCode()
Buffer.DataType	peekNextToConsumeDataType(int nextBufferToConsume, Collection<Buffer> buffersToRecycle)	Get dataType of next buffer to consume.
void	prepareForScheduling()	Refresh downstream consumption progress for another round scheduling of reading.
void	readBuffers(Queue<org.apache.flink.core.memory.MemorySegment> buffers, BufferRecycler recycler)	Read subpartition data into buffers.
void	releaseDataView()	Release this HsDataView when related subpartition view is releasing.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HsSubpartitionFileReaderImpl

public HsSubpartitionFileReaderImpl(int subpartitionId,
                                    HsConsumerId consumerId,
                                    FileChannel dataFileChannel,
                                    HsSubpartitionConsumerInternalOperations operations,
                                    HsFileDataIndex dataIndex,
                                    int maxBufferReadAhead,
                                    java.util.function.Consumer<HsSubpartitionFileReader> fileReaderReleaser,
                                    ByteBuffer headerBuf)

Method Detail

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

readBuffers

public void readBuffers(Queue<org.apache.flink.core.memory.MemorySegment> buffers,
                        BufferRecycler recycler)
                 throws IOException

Read subpartition data into buffers.

This transfers the ownership of used buffers to this class. It's this class' responsibility to release the buffers using the recycler when no longer needed.

Calling this method does not always use up all the provided buffers. It's this class' decision when to stop reading. Currently, it stops reading when: 1) buffers are used up, or 2) reaches the end of the subpartition data within the region, or 3) enough data have been read ahead the downstream consuming offset.

Specified by:

readBuffers in interface HsSubpartitionFileReader

Parameters:

buffers - for reading, note that the ownership of the buffer taken out from the queue is transferred to this class, and the unused buffer must be returned.

recycler - to return buffer to read buffer pool.

Throws:

IOException

fail

public void fail(Throwable failureCause)

Description copied from interface: HsSubpartitionFileReader

Fail this HsSubpartitionFileReader caused by failureCause.

Specified by:

fail in interface HsSubpartitionFileReader

Parameters:

failureCause - represents the reason why it failed.

prepareForScheduling

public void prepareForScheduling()

Refresh downstream consumption progress for another round scheduling of reading.

Specified by:

prepareForScheduling in interface HsSubpartitionFileReader

compareTo

public int compareTo(HsSubpartitionFileReader that)

Provides priority calculation logic for io scheduler.

Specified by:

compareTo in interface Comparable<HsSubpartitionFileReader>

getLoadedBuffers

public Deque<HsSubpartitionFileReaderImpl.BufferIndexOrError> getLoadedBuffers()

consumeBuffer

public Optional<ResultSubpartition.BufferAndBacklog> consumeBuffer(int nextBufferToConsume,
                                                                   Collection<Buffer> buffersToRecycle)
                                                            throws Throwable

Description copied from interface: HsDataView

Try to consume next buffer.

Only invoked by consumer thread.

Specified by:

consumeBuffer in interface HsDataView

Parameters:

nextBufferToConsume - next buffer index to consume.

buffersToRecycle - buffers to recycle if needed.

Returns:

If the target buffer does exist, return buffer and next buffer's backlog, otherwise return Optional.empty().

Throws:

Throwable

peekNextToConsumeDataType

public Buffer.DataType peekNextToConsumeDataType(int nextBufferToConsume,
                                                 Collection<Buffer> buffersToRecycle)

Description copied from interface: HsDataView

Get dataType of next buffer to consume.

Specified by:

peekNextToConsumeDataType in interface HsDataView

Parameters:

nextBufferToConsume - next buffer index to consume

buffersToRecycle - buffers to recycle if needed.

Returns:

next buffer's dataType. If not found in memory, return Buffer.DataType.NONE.

releaseDataView

public void releaseDataView()

Description copied from interface: HsDataView

Release this HsDataView when related subpartition view is releasing.

Specified by:

releaseDataView in interface HsDataView

getBacklog

public int getBacklog()

Description copied from interface: HsDataView

Get the number of buffers whose Buffer.DataType is buffer.

Specified by:

getBacklog in interface HsDataView

Returns:

backlog of this view's corresponding subpartition.