Package org.apache.flink.yarn

Class YarnClusterDescriptor

java.lang.Object

org.apache.flink.yarn.YarnClusterDescriptor

All Implemented Interfaces:

AutoCloseable, org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

public class YarnClusterDescriptor
extends Object
implements org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

The descriptor with deployment information for deploying a Flink cluster on Yarn.

Constructor Summary

Constructors

Constructor	Description
YarnClusterDescriptor(org.apache.flink.configuration.Configuration flinkConfiguration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration, org.apache.hadoop.yarn.client.api.YarnClient yarnClient, YarnClusterInformationRetriever yarnClusterInformationRetriever, boolean sharedYarnClient)

Method Summary

Modifier and Type	Method	Description
void	addShipFiles(List<org.apache.hadoop.fs.Path> shipFiles)	Adds the given files to the list of files to ship.
void	close()
org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId>	deployApplicationCluster(org.apache.flink.client.deployment.ClusterSpecification clusterSpecification, org.apache.flink.client.deployment.application.ApplicationConfiguration applicationConfiguration)
org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId>	deployJobCluster(org.apache.flink.client.deployment.ClusterSpecification clusterSpecification, org.apache.flink.runtime.jobgraph.JobGraph jobGraph, boolean detached)
org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId>	deploySessionCluster(org.apache.flink.client.deployment.ClusterSpecification clusterSpecification)
String	getClusterDescription()
org.apache.flink.configuration.Configuration	getFlinkConfiguration()
String	getNodeLabel()
org.apache.hadoop.yarn.client.api.YarnClient	getYarnClient()
protected String	getYarnJobClusterEntrypoint()	The class to start the application master with.
protected String	getYarnSessionClusterEntrypoint()	The class to start the application master with.
void	killCluster(org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
static void	logDetachedClusterInformation(org.apache.hadoop.yarn.api.records.ApplicationId yarnApplicationId, org.slf4j.Logger logger)
org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId>	retrieve(org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
void	setLocalJarPath(org.apache.hadoop.fs.Path localJarPath)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

YarnClusterDescriptor

public YarnClusterDescriptor(org.apache.flink.configuration.Configuration flinkConfiguration,
                             org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration,
                             org.apache.hadoop.yarn.client.api.YarnClient yarnClient,
                             YarnClusterInformationRetriever yarnClusterInformationRetriever,
                             boolean sharedYarnClient)

Method Detail

getYarnClient

public org.apache.hadoop.yarn.client.api.YarnClient getYarnClient()

getYarnSessionClusterEntrypoint

protected String getYarnSessionClusterEntrypoint()

The class to start the application master with. This class runs the main method in case of session cluster.

getYarnJobClusterEntrypoint

protected String getYarnJobClusterEntrypoint()

The class to start the application master with. This class runs the main method in case of the job cluster.

getFlinkConfiguration

public org.apache.flink.configuration.Configuration getFlinkConfiguration()

setLocalJarPath

public void setLocalJarPath(org.apache.hadoop.fs.Path localJarPath)

addShipFiles

public void addShipFiles(List<org.apache.hadoop.fs.Path> shipFiles)

Adds the given files to the list of files to ship.

Note that any file matching "flink-dist*.jar" will be excluded from the upload by YarnApplicationFileUploader.registerMultipleLocalResources(Collection, String, LocalResourceType) since we upload the Flink uber jar ourselves and do not need to deploy it multiple times.

Parameters:

shipFiles - files to ship

getNodeLabel

public String getNodeLabel()

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

retrieve

public org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> retrieve(org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
                                                                                                                 throws org.apache.flink.client.deployment.ClusterRetrieveException

Specified by:

retrieve in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

Throws:

org.apache.flink.client.deployment.ClusterRetrieveException

deploySessionCluster

public org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> deploySessionCluster(org.apache.flink.client.deployment.ClusterSpecification clusterSpecification)
                                                                                                                             throws org.apache.flink.client.deployment.ClusterDeploymentException

Specified by:

deploySessionCluster in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

Throws:

org.apache.flink.client.deployment.ClusterDeploymentException

deployApplicationCluster

public org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> deployApplicationCluster(org.apache.flink.client.deployment.ClusterSpecification clusterSpecification,
                                                                                                                                        org.apache.flink.client.deployment.application.ApplicationConfiguration applicationConfiguration)
                                                                                                                                 throws org.apache.flink.client.deployment.ClusterDeploymentException

Specified by:

deployApplicationCluster in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

Throws:

org.apache.flink.client.deployment.ClusterDeploymentException

deployJobCluster

public org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> deployJobCluster(org.apache.flink.client.deployment.ClusterSpecification clusterSpecification,
                                                                                                                                org.apache.flink.runtime.jobgraph.JobGraph jobGraph,
                                                                                                                                boolean detached)
                                                                                                                         throws org.apache.flink.client.deployment.ClusterDeploymentException

Specified by:

deployJobCluster in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

Throws:

org.apache.flink.client.deployment.ClusterDeploymentException

killCluster

public void killCluster(org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
                 throws org.apache.flink.util.FlinkException

Specified by:

killCluster in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

Throws:

org.apache.flink.util.FlinkException

getClusterDescription

public String getClusterDescription()

Specified by:

getClusterDescription in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

logDetachedClusterInformation

public static void logDetachedClusterInformation(org.apache.hadoop.yarn.api.records.ApplicationId yarnApplicationId,
                                                 org.slf4j.Logger logger)