Package org.apache.flink.configuration

Class NettyShuffleEnvironmentOptions

  • @PublicEvolving
public class NettyShuffleEnvironmentOptions
extends Object
    The set of configuration options relating to network stack.

    • Field Detail

      • DATA_PORT

        public static final ConfigOption<Integer> DATA_PORT
        The default network port the task manager expects to receive transfer envelopes on. The 0 means that the TaskManager searches for a free port.

      • DATA_BIND_PORT

        public static final ConfigOption<String> DATA_BIND_PORT
        The local network port that the task manager listen at for data exchange.

      • DATA_SSL_ENABLED

        public static final ConfigOption<Boolean> DATA_SSL_ENABLED
        Config parameter to override SSL support for taskmanager's data transport.

      • BATCH_SHUFFLE_COMPRESSION_ENABLED

        @Deprecated
public static final ConfigOption<Boolean> BATCH_SHUFFLE_COMPRESSION_ENABLED
        Deprecated.
        This option is deprecated in 1.20 and will be removed in 2.0. Please set the SHUFFLE_COMPRESSION_CODEC to NONE to disable the compression.
        Boolean flag indicating whether the shuffle data will be compressed for batch shuffle mode.

        Note: Data is compressed per buffer and compression can incur extra CPU overhead so it is more effective for IO bounded scenario when data compression ratio is high.

      • NETWORK_DETAILED_METRICS

        public static final ConfigOption<Boolean> NETWORK_DETAILED_METRICS
        Boolean flag to enable/disable more detailed metrics about inbound/outbound network queue lengths.

      • MAX_NUM_TCP_CONNECTIONS

        @Deprecated
public static final ConfigOption<Integer> MAX_NUM_TCP_CONNECTIONS
        Deprecated.
        The option is unnecessary. It is deprecated in 1.20 and will be removed and hard-coded to 1 in 2.0.
        The maximum number of tpc connections between taskmanagers for data communication.

      • NETWORK_READ_MAX_REQUIRED_BUFFERS_PER_GATE

        @Experimental
public static final ConfigOption<Integer> NETWORK_READ_MAX_REQUIRED_BUFFERS_PER_GATE
        The maximum number of network read buffers that are required by an input gate. (An input gate is responsible for reading data from all subtasks of an upstream task.) The number of buffers needed by an input gate is dynamically calculated in runtime, depending on various factors (e.g., the parallelism of the upstream task). Among the calculated number of needed buffers, the part below this configured value is required, while the excess part, if any, is optional. A task will fail if the required buffers cannot be obtained in runtime. A task will not fail due to not obtaining optional buffers, but may suffer a performance reduction.

      • NETWORK_BUFFERS_PER_CHANNEL

        @Deprecated
public static final ConfigOption<Integer> NETWORK_BUFFERS_PER_CHANNEL
        Deprecated.
        This option is deprecated in 1.20 and will be removed in 2.0 to simplify the configuration of network buffers.
        Number of network buffers for each outgoing/incoming channel (subpartition/input channel). The minimum valid value for the option is 0. When the option is configured as 0, the exclusive network buffers used per downstream incoming channel will be 0, but for each upstream outgoing channel, max(1, configured value) will be used. In other words we ensure that, for performance reasons, at least one buffer is used per outgoing channel regardless of the configuration.

      • NETWORK_EXTRA_BUFFERS_PER_GATE

        @Deprecated
public static final ConfigOption<Integer> NETWORK_EXTRA_BUFFERS_PER_GATE
        Deprecated.
        This option is deprecated in 1.20 and will be removed in 2.0 to simplify the configuration of network buffers.
        Number of floating network buffers for each outgoing/incoming gate (result partition/input gate).

      • NETWORK_SORT_SHUFFLE_MIN_BUFFERS

        public static final ConfigOption<Integer> NETWORK_SORT_SHUFFLE_MIN_BUFFERS
        Minimum number of network buffers required per blocking result partition for sort-shuffle.

      • NETWORK_SORT_SHUFFLE_MIN_PARALLELISM

        @Deprecated
public static final ConfigOption<Integer> NETWORK_SORT_SHUFFLE_MIN_PARALLELISM
        Deprecated.
        The hash-based blocking shuffle is deprecated in 1.20 and will be totally removed in 2.0.
        Parallelism threshold to switch between sort-based blocking shuffle and hash-based blocking shuffle.

      • HYBRID_SHUFFLE_SPILLED_INDEX_REGION_GROUP_SIZE

        @Deprecated
public static final ConfigOption<Integer> HYBRID_SHUFFLE_SPILLED_INDEX_REGION_GROUP_SIZE
        Deprecated.
        The option is only used in the legacy hybrid shuffle mode. It is deprecated in 1.20 and will be totally removed in 2.0, as the legacy hybrid shuffle mode will be removed in 2.0.
        Region group size of hybrid spilled file data index.

      • HYBRID_SHUFFLE_NUM_RETAINED_IN_MEMORY_REGIONS_MAX

        @Deprecated
public static final ConfigOption<Long> HYBRID_SHUFFLE_NUM_RETAINED_IN_MEMORY_REGIONS_MAX
        Deprecated.
        The option is only used in the legacy hybrid shuffle mode. It is deprecated in 1.20 and will be totally removed in 2.0, as the legacy hybrid shuffle mode will be removed in 2.0.
        Max number of hybrid retained regions in memory.

      • NETWORK_MAX_BUFFERS_PER_CHANNEL

        @Deprecated
public static final ConfigOption<Integer> NETWORK_MAX_BUFFERS_PER_CHANNEL
        Deprecated.
        This option is deprecated in 1.20 and will be removed in 2.0 to simplify the configuration of network buffers.
        Number of max buffers can be used for each output subpartition.

      • NETWORK_MAX_OVERDRAFT_BUFFERS_PER_GATE

        @Deprecated
public static final ConfigOption<Integer> NETWORK_MAX_OVERDRAFT_BUFFERS_PER_GATE
        Deprecated.
        This option is deprecated in 1.20 and will be removed in 2.0 to simplify the configuration of network buffers.
        Number of max overdraft network buffers to use for each ResultPartition.

      • NETWORK_EXCLUSIVE_BUFFERS_REQUEST_TIMEOUT_MILLISECONDS

        @Deprecated
public static final ConfigOption<Long> NETWORK_EXCLUSIVE_BUFFERS_REQUEST_TIMEOUT_MILLISECONDS
        Deprecated.
        This option is deprecated in 1.20 and will be removed in 2.0 to simplify the configuration of network buffers. Please use NETWORK_BUFFERS_REQUEST_TIMEOUT instead.
        The timeout for requesting exclusive buffers for each channel.

      • NETWORK_BUFFERS_REQUEST_TIMEOUT

        public static final ConfigOption<java.time.Duration> NETWORK_BUFFERS_REQUEST_TIMEOUT
        The timeout for requesting buffers for each channel.

      • NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE

        @Deprecated
@Experimental
public static final ConfigOption<Boolean> NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE
        Deprecated.
        This option is deprecated in 1.20 and will be totally removed in 2.0, as the legacy hybrid shuffle mode will be removed in 2.0.
        The option to enable the new mode of hybrid shuffle.

      • NETWORK_HYBRID_SHUFFLE_EXTERNAL_REMOTE_TIER_FACTORY_CLASS_NAME

        @Experimental
public static final ConfigOption<String> NETWORK_HYBRID_SHUFFLE_EXTERNAL_REMOTE_TIER_FACTORY_CLASS_NAME
        The option to configure the tiered factory creator remote class name for hybrid shuffle.

      • NETWORK_HYBRID_SHUFFLE_REMOTE_STORAGE_BASE_PATH

        @Experimental
public static final ConfigOption<String> NETWORK_HYBRID_SHUFFLE_REMOTE_STORAGE_BASE_PATH
        The option to configure the base remote storage path for hybrid shuffle.

      • NETWORK_BLOCKING_SHUFFLE_TYPE

        @Deprecated
public static final ConfigOption<String> NETWORK_BLOCKING_SHUFFLE_TYPE
        Deprecated.
        The hash-based blocking shuffle is deprecated in 1.20 and will be totally removed in 2.0.

      • TCP_CONNECTION_REUSE_ACROSS_JOBS_ENABLED

        public static final ConfigOption<Boolean> TCP_CONNECTION_REUSE_ACROSS_JOBS_ENABLED
        Whether to reuse tcp connections across multi jobs. If set to true, tcp connections will not be released after job finishes. The subsequent jobs will be free from the overhead of the connection re-establish. However, this may lead to an increase in the total number of connections on your machine. When it reaches the upper limit, you can set it to false to release idle connections.

        Note: To avoid connection leak, you must set MAX_NUM_TCP_CONNECTIONS to a smaller value before you enable tcp connection reuse.

      • NUM_ARENAS

        @Deprecated
public static final ConfigOption<Integer> NUM_ARENAS
        Deprecated.
        It is not recommended to use the option. It is deprecated in 1.20 and will be totally removed in 2.0.

      • NUM_THREADS_SERVER

        @Deprecated
public static final ConfigOption<Integer> NUM_THREADS_SERVER
        Deprecated.
        It is not recommended to use the option. It is deprecated in 1.20 and will be totally removed in 2.0.

      • NUM_THREADS_CLIENT

        @Deprecated
public static final ConfigOption<Integer> NUM_THREADS_CLIENT
        Deprecated.
        It is not recommended to use the option. It is deprecated in 1.20 and will be totally removed in 2.0.

      • CONNECT_BACKLOG

        @Deprecated
public static final ConfigOption<Integer> CONNECT_BACKLOG
        Deprecated.
        It is not recommended to use the option. It is deprecated in 1.20 and will be totally removed in 2.0.

      • CLIENT_CONNECT_TIMEOUT_SECONDS

        public static final ConfigOption<Integer> CLIENT_CONNECT_TIMEOUT_SECONDS

      • SEND_RECEIVE_BUFFER_SIZE

        @Deprecated
public static final ConfigOption<Integer> SEND_RECEIVE_BUFFER_SIZE
        Deprecated.
        It is not recommended to use the option. It is deprecated in 1.20 and will be totally removed in 2.0.

      • TRANSPORT_TYPE

        @Deprecated
public static final ConfigOption<String> TRANSPORT_TYPE
        Deprecated.
        It is not recommended to use the option. It is deprecated in 1.20 and will be totally removed in 2.0.

      • CLIENT_TCP_KEEP_IDLE_SECONDS

        public static final ConfigOption<Integer> CLIENT_TCP_KEEP_IDLE_SECONDS

      • CLIENT_TCP_KEEP_INTERVAL_SECONDS

        public static final ConfigOption<Integer> CLIENT_TCP_KEEP_INTERVAL_SECONDS

      • NETWORK_REQUEST_BACKOFF_INITIAL

        public static final ConfigOption<Integer> NETWORK_REQUEST_BACKOFF_INITIAL
        Minimum backoff for partition requests of input channels.

      • NETWORK_REQUEST_BACKOFF_MAX

        public static final ConfigOption<Integer> NETWORK_REQUEST_BACKOFF_MAX
        Maximum backoff for partition requests of input channels.

      • NETWORK_PARTITION_REQUEST_TIMEOUT

        public static final ConfigOption<java.time.Duration> NETWORK_PARTITION_REQUEST_TIMEOUT
        The timeout for partition request listener in result partition manager.