Package org.apache.flink.streaming.runtime.io.recovery

Class RescalingStreamTaskNetworkInput<T>

  • All Implemented Interfaces:
    Closeable, AutoCloseable, org.apache.flink.runtime.io.AvailabilityProvider, PushingAsyncDataInput<T>, RecoverableStreamTaskInput<T>, StreamTaskInput<T>
    @Internal
public final class RescalingStreamTaskNetworkInput<T>
extends AbstractStreamTaskNetworkInput<T,​org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>>
implements RecoverableStreamTaskInput<T>
    A StreamTaskNetworkInput implementation that demultiplexes virtual channels.

    The demultiplexing works in two dimensions for the following cases. *

    • Subtasks of the current operator have been collapsed in a round-robin fashion.
    • The connected output operator has been rescaled (up and down!) and there is an overlap of channels (mostly relevant to keyed exchanges).

    In both cases, records from multiple old channels are received over one new physical channel, which need to demultiplex the record to correctly restore spanning records (similar to how StreamTaskNetworkInput works).

    Note that when both cases occur at the same time (downscaling of several operators), there is the cross product of channels. So if two subtasks are collapsed and two channels overlap from the output side, there is a total of 4 virtual channels.

    • Constructor Detail

      • RescalingStreamTaskNetworkInput

        public RescalingStreamTaskNetworkInput​(CheckpointedInputGate checkpointedInputGate,
                                       org.apache.flink.api.common.typeutils.TypeSerializer<T> inputSerializer,
                                       org.apache.flink.runtime.io.disk.iomanager.IOManager ioManager,
                                       StatusWatermarkValve statusWatermarkValve,
                                       int inputIndex,
                                       org.apache.flink.runtime.checkpoint.InflightDataRescalingDescriptor inflightDataRescalingDescriptor,
                                       Function<Integer,​StreamPartitioner<?>> gatePartitioners,
                                       org.apache.flink.api.common.TaskInfo taskInfo,
                                       StreamTask.CanEmitBatchOfRecordsChecker canEmitBatchOfRecords)

    • Method Detail

      • getActiveSerializer

        protected org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T> getActiveSerializer​(org.apache.flink.runtime.checkpoint.channel.InputChannelInfo channelInfo)
        Overrides:
        getActiveSerializer in class AbstractStreamTaskNetworkInput<T,​org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>>

      • processEvent

        protected DataInputStatus processEvent​(org.apache.flink.runtime.io.network.partition.consumer.BufferOrEvent bufferOrEvent)
        Overrides:
        processEvent in class AbstractStreamTaskNetworkInput<T,​org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>>

      • prepareSnapshot

        public CompletableFuture<Void> prepareSnapshot​(org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter channelStateWriter,
                                               long checkpointId)
                                        throws org.apache.flink.runtime.checkpoint.CheckpointException
        Description copied from interface: StreamTaskInput
        Prepares to spill the in-flight input buffers as checkpoint snapshot.
        Specified by:
        prepareSnapshot in interface StreamTaskInput<T>
        Throws:
        org.apache.flink.runtime.checkpoint.CheckpointException