Package org.apache.flink.configuration

Class NettyShuffleEnvironmentOptions

  • @PublicEvolving
public class NettyShuffleEnvironmentOptions
extends Object
    The set of configuration options relating to network stack.

    • Field Detail

      • DATA_PORT

        public static final ConfigOption<Integer> DATA_PORT
        The default network port the task manager expects to receive transfer envelopes on. The 0 means that the TaskManager searches for a free port.

      • DATA_BIND_PORT

        public static final ConfigOption<Integer> DATA_BIND_PORT
        The local network port that the task manager listen at for data exchange.

      • DATA_SSL_ENABLED

        public static final ConfigOption<Boolean> DATA_SSL_ENABLED
        Config parameter to override SSL support for taskmanager's data transport.

      • BATCH_SHUFFLE_COMPRESSION_ENABLED

        public static final ConfigOption<Boolean> BATCH_SHUFFLE_COMPRESSION_ENABLED
        Boolean flag indicating whether the shuffle data will be compressed for batch shuffle mode.

        Note: Data is compressed per buffer and compression can incur extra CPU overhead so it is more effective for IO bounded scenario when data compression ratio is high.

      • SHUFFLE_COMPRESSION_CODEC

        @Experimental
public static final ConfigOption<String> SHUFFLE_COMPRESSION_CODEC
        The codec to be used when compressing shuffle data.

      • NETWORK_DETAILED_METRICS

        public static final ConfigOption<Boolean> NETWORK_DETAILED_METRICS
        Boolean flag to enable/disable more detailed metrics about inbound/outbound network queue lengths.

      • MAX_NUM_TCP_CONNECTIONS

        public static final ConfigOption<Integer> MAX_NUM_TCP_CONNECTIONS
        The maximum number of tpc connections between taskmanagers for data communication.

      • NETWORK_READ_MAX_REQUIRED_BUFFERS_PER_GATE

        @Experimental
public static final ConfigOption<Integer> NETWORK_READ_MAX_REQUIRED_BUFFERS_PER_GATE
        The maximum number of network read buffers that are required by an input gate. (An input gate is responsible for reading data from all subtasks of an upstream task.) The number of buffers needed by an input gate is dynamically calculated in runtime, depending on various factors (e.g., the parallelism of the upstream task). Among the calculated number of needed buffers, the part below this configured value is required, while the excess part, if any, is optional. A task will fail if the required buffers cannot be obtained in runtime. A task will not fail due to not obtaining optional buffers, but may suffer a performance reduction.

      • NETWORK_BUFFERS_PER_CHANNEL

        public static final ConfigOption<Integer> NETWORK_BUFFERS_PER_CHANNEL
        Number of network buffers for each outgoing/incoming channel (subpartition/input channel). The minimum valid value for the option is 0. When the option is configured as 0, the exclusive network buffers used per downstream incoming channel will be 0, but for each upstream outgoing channel, max(1, configured value) will be used. In other words we ensure that, for performance reasons, at least one buffer is used per outgoing channel regardless of the configuration.

      • NETWORK_EXTRA_BUFFERS_PER_GATE

        public static final ConfigOption<Integer> NETWORK_EXTRA_BUFFERS_PER_GATE
        Number of floating network buffers for each outgoing/incoming gate (result partition/input gate).

      • NETWORK_SORT_SHUFFLE_MIN_BUFFERS

        public static final ConfigOption<Integer> NETWORK_SORT_SHUFFLE_MIN_BUFFERS
        Minimum number of network buffers required per blocking result partition for sort-shuffle.

      • NETWORK_SORT_SHUFFLE_MIN_PARALLELISM

        public static final ConfigOption<Integer> NETWORK_SORT_SHUFFLE_MIN_PARALLELISM
        Parallelism threshold to switch between sort-based blocking shuffle and hash-based blocking shuffle.

      • HYBRID_SHUFFLE_SPILLED_INDEX_REGION_GROUP_SIZE

        public static final ConfigOption<Integer> HYBRID_SHUFFLE_SPILLED_INDEX_REGION_GROUP_SIZE
        Region group size of hybrid spilled file data index.

      • HYBRID_SHUFFLE_NUM_RETAINED_IN_MEMORY_REGIONS_MAX

        public static final ConfigOption<Long> HYBRID_SHUFFLE_NUM_RETAINED_IN_MEMORY_REGIONS_MAX
        Max number of hybrid retained regions in memory.

      • NETWORK_MAX_BUFFERS_PER_CHANNEL

        public static final ConfigOption<Integer> NETWORK_MAX_BUFFERS_PER_CHANNEL
        Number of max buffers can be used for each output subpartition.

      • NETWORK_MAX_OVERDRAFT_BUFFERS_PER_GATE

        public static final ConfigOption<Integer> NETWORK_MAX_OVERDRAFT_BUFFERS_PER_GATE
        Number of max overdraft network buffers to use for each ResultPartition.

      • NETWORK_EXCLUSIVE_BUFFERS_REQUEST_TIMEOUT_MILLISECONDS

        public static final ConfigOption<Long> NETWORK_EXCLUSIVE_BUFFERS_REQUEST_TIMEOUT_MILLISECONDS
        The timeout for requesting exclusive buffers for each channel.

      • NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE

        @Experimental
public static final ConfigOption<Boolean> NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE
        The option to enable the new mode of hybrid shuffle.

      • NETWORK_HYBRID_SHUFFLE_REMOTE_STORAGE_BASE_PATH

        @Experimental
public static final ConfigOption<String> NETWORK_HYBRID_SHUFFLE_REMOTE_STORAGE_BASE_PATH
        The option to configure the base remote storage path for hybrid shuffle.

      • NETWORK_BLOCKING_SHUFFLE_TYPE

        public static final ConfigOption<String> NETWORK_BLOCKING_SHUFFLE_TYPE

      • TCP_CONNECTION_REUSE_ACROSS_JOBS_ENABLED

        public static final ConfigOption<Boolean> TCP_CONNECTION_REUSE_ACROSS_JOBS_ENABLED
        Whether to reuse tcp connections across multi jobs. If set to true, tcp connections will not be released after job finishes. The subsequent jobs will be free from the overhead of the connection re-establish. However, this may lead to an increase in the total number of connections on your machine. When it reaches the upper limit, you can set it to false to release idle connections.

        Note: To avoid connection leak, you must set MAX_NUM_TCP_CONNECTIONS to a smaller value before you enable tcp connection reuse.

      • CLIENT_CONNECT_TIMEOUT_SECONDS

        public static final ConfigOption<Integer> CLIENT_CONNECT_TIMEOUT_SECONDS

      • CLIENT_TCP_KEEP_IDLE_SECONDS

        public static final ConfigOption<Integer> CLIENT_TCP_KEEP_IDLE_SECONDS

      • CLIENT_TCP_KEEP_INTERVAL_SECONDS

        public static final ConfigOption<Integer> CLIENT_TCP_KEEP_INTERVAL_SECONDS

      • NETWORK_REQUEST_BACKOFF_INITIAL

        public static final ConfigOption<Integer> NETWORK_REQUEST_BACKOFF_INITIAL
        Minimum backoff for partition requests of input channels.

      • NETWORK_REQUEST_BACKOFF_MAX

        public static final ConfigOption<Integer> NETWORK_REQUEST_BACKOFF_MAX
        Maximum backoff for partition requests of input channels.