Package org.apache.flink.yarn

Class Utils

  • public final class Utils
extends Object
    Utility class that provides helper methods to work with Apache Hadoop YARN.

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static String KRB5_FILE_NAME
      KRB5 file name populated in YARN container for secure IT run.
      static String YARN_SITE_FILE_NAME
      Yarn site xml file name populated in YARN container for secure IT run.

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static void addToEnvironment​(Map<String,​String> environment, String variable, String value)
      Copied method from org.apache.hadoop.yarn.util.Apps.
      static void deleteApplicationFiles​(String applicationFilesDir)
      Deletes the YARN application files, e.g., Flink binaries, libraries, etc., from the remote filesystem.
      static List<org.apache.hadoop.fs.Path> getQualifiedRemoteProvidedLibDirs​(org.apache.flink.configuration.Configuration configuration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration)  
      static Optional<org.apache.hadoop.fs.Path> getQualifiedRemoteProvidedUsrLib​(org.apache.flink.configuration.Configuration configuration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration)  
      static org.apache.hadoop.yarn.conf.YarnConfiguration getYarnAndHadoopConfiguration​(org.apache.flink.configuration.Configuration flinkConfig)  
      static org.apache.hadoop.yarn.conf.YarnConfiguration getYarnConfiguration​(org.apache.flink.configuration.Configuration flinkConfig)
      Add additional config entries from the flink config to the yarn config.
      static boolean isUsrLibDirectory​(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path)  
      static String resolveKeytabPath​(String workingDir, String keytabPath)
      Resolve keytab path either as absolute path or relative to working directory.
      static void setAclsFor​(org.apache.hadoop.yarn.api.records.ContainerLaunchContext amContainer, org.apache.flink.configuration.Configuration flinkConfig)
      Sets the application ACLs for the given ContainerLaunchContext based on the values specified in the given Flink configuration.
      static void setupYarnClassPath​(org.apache.hadoop.conf.Configuration conf, Map<String,​String> appMasterEnv)  

    • Field Detail

      • KRB5_FILE_NAME

        public static final String KRB5_FILE_NAME
        KRB5 file name populated in YARN container for secure IT run.
        See Also:
        Constant Field Values

      • YARN_SITE_FILE_NAME

        public static final String YARN_SITE_FILE_NAME
        Yarn site xml file name populated in YARN container for secure IT run.
        See Also:
        Constant Field Values

    • Method Detail

      • setupYarnClassPath

        public static void setupYarnClassPath​(org.apache.hadoop.conf.Configuration conf,
                                      Map<String,​String> appMasterEnv)

      • deleteApplicationFiles

        public static void deleteApplicationFiles​(String applicationFilesDir)
        Deletes the YARN application files, e.g., Flink binaries, libraries, etc., from the remote filesystem.
        Parameters:
        applicationFilesDir - The application files directory.

      • addToEnvironment

        public static void addToEnvironment​(Map<String,​String> environment,
                                    String variable,
                                    String value)
        Copied method from org.apache.hadoop.yarn.util.Apps. It was broken by YARN-1824 (2.4.0) and fixed for 2.4.1 by https://issues.apache.org/jira/browse/YARN-1931

      • resolveKeytabPath

        public static String resolveKeytabPath​(String workingDir,
                                       String keytabPath)
        Resolve keytab path either as absolute path or relative to working directory.
        Parameters:
        workingDir - current working directory
        keytabPath - configured keytab path.
        Returns:
        resolved keytab path, or null if not found.

      • getQualifiedRemoteProvidedLibDirs

        public static List<org.apache.hadoop.fs.Path> getQualifiedRemoteProvidedLibDirs​(org.apache.flink.configuration.Configuration configuration,
                                                                                org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration)
                                                                         throws IOException
        Throws:
        IOException

      • isUsrLibDirectory

        public static boolean isUsrLibDirectory​(org.apache.hadoop.fs.FileSystem fileSystem,
                                        org.apache.hadoop.fs.Path path)
                                 throws IOException
        Throws:
        IOException

      • getYarnAndHadoopConfiguration

        public static org.apache.hadoop.yarn.conf.YarnConfiguration getYarnAndHadoopConfiguration​(org.apache.flink.configuration.Configuration flinkConfig)

      • getYarnConfiguration

        public static org.apache.hadoop.yarn.conf.YarnConfiguration getYarnConfiguration​(org.apache.flink.configuration.Configuration flinkConfig)
        Add additional config entries from the flink config to the yarn config.
        Parameters:
        flinkConfig - The Flink configuration object.
        Returns:
        The yarn configuration.

      • setAclsFor

        public static void setAclsFor​(org.apache.hadoop.yarn.api.records.ContainerLaunchContext amContainer,
                              org.apache.flink.configuration.Configuration flinkConfig)
        Sets the application ACLs for the given ContainerLaunchContext based on the values specified in the given Flink configuration. Only ApplicationAccessType.VIEW_APP and ApplicationAccessType.MODIFY_APP ACLs are set, and only if they are configured in the Flink configuration. If the viewAcls or modifyAcls string contains the WILDCARD_ACL constant, it will replace the entire string with the WILDCARD_ACL. The resulting map is then set as the application acls for the given container launch context.
        Parameters:
        amContainer - the ContainerLaunchContext to set the ACLs for.
        flinkConfig - the Flink configuration to read the ACL values from.