Package org.apache.flink.yarn

Class Utils

  • public final class Utils
extends Object
    Utility class that provides helper methods to work with Apache Hadoop YARN.

    • Field Detail

      • KRB5_FILE_NAME

        public static final String KRB5_FILE_NAME
        KRB5 file name populated in YARN container for secure IT run.
        See Also:
        Constant Field Values

      • YARN_SITE_FILE_NAME

        public static final String YARN_SITE_FILE_NAME
        Yarn site xml file name populated in YARN container for secure IT run.
        See Also:
        Constant Field Values

    • Method Detail

      • setupYarnClassPath

        public static void setupYarnClassPath​(org.apache.hadoop.conf.Configuration conf,
                                      Map<String,​String> appMasterEnv)

      • deleteApplicationFiles

        public static void deleteApplicationFiles​(String applicationFilesDir)
        Deletes the YARN application files, e.g., Flink binaries, libraries, etc., from the remote filesystem.
        Parameters:
        applicationFilesDir - The application files directory.

      • addToEnvironment

        public static void addToEnvironment​(Map<String,​String> environment,
                                    String variable,
                                    String value)
        Copied method from org.apache.hadoop.yarn.util.Apps. It was broken by YARN-1824 (2.4.0) and fixed for 2.4.1 by https://issues.apache.org/jira/browse/YARN-1931

      • resolveKeytabPath

        public static String resolveKeytabPath​(String workingDir,
                                       String keytabPath)
        Resolve keytab path either as absolute path or relative to working directory.
        Parameters:
        workingDir - current working directory
        keytabPath - configured keytab path.
        Returns:
        resolved keytab path, or null if not found.

      • getTaskManagerShellCommand

        public static String getTaskManagerShellCommand​(org.apache.flink.configuration.Configuration flinkConfig,
                                                org.apache.flink.runtime.clusterframework.ContaineredTaskManagerParameters tmParams,
                                                String configDirectory,
                                                String logDirectory,
                                                boolean hasLogback,
                                                boolean hasLog4j,
                                                boolean hasKrb5,
                                                Class<?> mainClass,
                                                String mainArgs)
        Generates the shell command to start a task manager.
        Parameters:
        flinkConfig - The Flink configuration.
        tmParams - Parameters for the task manager.
        configDirectory - The configuration directory for the config.yaml
        logDirectory - The log directory.
        hasLogback - Uses logback?
        hasLog4j - Uses log4j?
        mainClass - The main class to start with.
        Returns:
        A String containing the task manager startup command.

      • getStartCommand

        public static String getStartCommand​(String template,
                                     Map<String,​String> startCommandValues)
        Replaces placeholders in the template start command with values from startCommandValues.

        If the default template ConfigConstants.DEFAULT_YARN_CONTAINER_START_COMMAND_TEMPLATE is used, the following keys must be present in the map or the resulting command will still contain placeholders:

        • java = path to the Java executable
        • jvmmem = JVM memory limits and tweaks
        • jvmopts = misc options for the Java VM
        • logging = logging-related configuration settings
        • class = main class to execute
        • args = arguments for the main class
        • redirects = output redirects
        Parameters:
        template - a template start command with placeholders
        startCommandValues - a replacement map placeholder -> value
        Returns:
        the start command with placeholders filled in

      • generateJvmOptsString

        public static String generateJvmOptsString​(org.apache.flink.configuration.Configuration conf,
                                           List<org.apache.flink.configuration.ConfigOption<String>> jvmOptions,
                                           boolean hasKrb5)

      • getQualifiedRemoteProvidedLibDirs

        public static List<org.apache.hadoop.fs.Path> getQualifiedRemoteProvidedLibDirs​(org.apache.flink.configuration.Configuration configuration,
                                                                                org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration)
                                                                         throws IOException
        Throws:
        IOException

      • isUsrLibDirectory

        public static boolean isUsrLibDirectory​(org.apache.hadoop.fs.FileSystem fileSystem,
                                        org.apache.hadoop.fs.Path path)
                                 throws IOException
        Throws:
        IOException

      • getYarnAndHadoopConfiguration

        public static org.apache.hadoop.yarn.conf.YarnConfiguration getYarnAndHadoopConfiguration​(org.apache.flink.configuration.Configuration flinkConfig)

      • getYarnConfiguration

        public static org.apache.hadoop.yarn.conf.YarnConfiguration getYarnConfiguration​(org.apache.flink.configuration.Configuration flinkConfig)
        Add additional config entries from the flink config to the yarn config.
        Parameters:
        flinkConfig - The Flink configuration object.
        Returns:
        The yarn configuration.

      • setAclsFor

        public static void setAclsFor​(org.apache.hadoop.yarn.api.records.ContainerLaunchContext amContainer,
                              org.apache.flink.configuration.Configuration flinkConfig)
        Sets the application ACLs for the given ContainerLaunchContext based on the values specified in the given Flink configuration. Only ApplicationAccessType.VIEW_APP and ApplicationAccessType.MODIFY_APP ACLs are set, and only if they are configured in the Flink configuration. If the viewAcls or modifyAcls string contains the WILDCARD_ACL constant, it will replace the entire string with the WILDCARD_ACL. The resulting map is then set as the application acls for the given container launch context.
        Parameters:
        amContainer - the ContainerLaunchContext to set the ACLs for.
        flinkConfig - the Flink configuration to read the ACL values from.

      • getPathFromLocalFile

        public static org.apache.hadoop.fs.Path getPathFromLocalFile​(File localFile)

      • getPathFromLocalFilePathStr

        public static org.apache.hadoop.fs.Path getPathFromLocalFilePathStr​(String localPathStr)