Package org.apache.flink.client.program.rest

Class RestClusterClient<T>

    • Constructor Summary

      Constructors 
      Constructor Description
      RestClusterClient​(org.apache.flink.configuration.Configuration config, T clusterId)  
      RestClusterClient​(org.apache.flink.configuration.Configuration config, T clusterId, org.apache.flink.runtime.highavailability.ClientHighAvailabilityServicesFactory factory)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      CompletableFuture<org.apache.flink.runtime.messages.Acknowledge> cancel​(org.apache.flink.api.common.JobID jobID)
      Cancels a job identified by the job id.
      CompletableFuture<String> cancelWithSavepoint​(org.apache.flink.api.common.JobID jobId, String savepointDirectory, org.apache.flink.core.execution.SavepointFormatType formatType)
      Cancels a job identified by the job id and triggers a savepoint.
      void close()  
      CompletableFuture<org.apache.flink.runtime.messages.Acknowledge> disposeSavepoint​(String savepointPath)
      Dispose the savepoint under the given path.
      CompletableFuture<Map<String,​Object>> getAccumulators​(org.apache.flink.api.common.JobID jobID, ClassLoader loader)
      Requests and returns the accumulators for the given job identifier.
      T getClusterId()
      Returns the cluster id identifying the cluster to which the client is connected.
      CompletableFuture<org.apache.flink.runtime.rest.handler.legacy.messages.ClusterOverviewWithVersion> getClusterOverview()
      Get an overview of the Flink cluster.
      org.apache.flink.configuration.Configuration getFlinkConfiguration()
      Return the Flink configuration object.
      CompletableFuture<org.apache.flink.runtime.rest.messages.job.JobDetailsInfo> getJobDetails​(org.apache.flink.api.common.JobID jobId)
      Requests the job details.
      CompletableFuture<org.apache.flink.api.common.JobStatus> getJobStatus​(org.apache.flink.api.common.JobID jobId)
      Requests the JobStatus of the job with the given JobID.
      String getWebInterfaceURL()
      Returns an URL (as a string) to the cluster web interface.
      CompletableFuture<Void> invalidateClusterDataset​(org.apache.flink.util.AbstractID clusterDatasetId)
      Invalidate the cached intermediate dataset with the given id.
      CompletableFuture<Set<org.apache.flink.util.AbstractID>> listCompletedClusterDatasetIds()
      Return a set of ids of the completed cluster datasets.
      CompletableFuture<Collection<org.apache.flink.runtime.client.JobStatusMessage>> listJobs()
      Lists the currently running and finished jobs on the cluster.
      CompletableFuture<Void> reportHeartbeat​(org.apache.flink.api.common.JobID jobId, long expiredTimestamp)
      The client reports the heartbeat to the dispatcher for aliveness.
      CompletableFuture<org.apache.flink.runtime.jobmaster.JobResult> requestJobResult​(org.apache.flink.api.common.JobID jobId)
      Requests the JobResult for the given JobID.
      CompletableFuture<org.apache.flink.runtime.operators.coordination.CoordinationResponse> sendCoordinationRequest​(org.apache.flink.api.common.JobID jobId, org.apache.flink.runtime.jobgraph.OperatorID operatorId, org.apache.flink.runtime.operators.coordination.CoordinationRequest request)
      Sends out a request to a specified coordinator and return the response.
      <M extends org.apache.flink.runtime.rest.messages.MessageHeaders<R,​P,​U>,​U extends org.apache.flink.runtime.rest.messages.MessageParameters,​R extends org.apache.flink.runtime.rest.messages.RequestBody,​P extends org.apache.flink.runtime.rest.messages.ResponseBody>
      CompletableFuture<P>      		 sendRequest​(M messageHeaders, U messageParameters, R request)  
      void shutDownCluster()
      Shut down the cluster that this client communicate with.
      CompletableFuture<String> stopWithDetachedSavepoint​(org.apache.flink.api.common.JobID jobId, boolean advanceToEndOfTime, String savepointDirectory, org.apache.flink.core.execution.SavepointFormatType formatType)
      Stops a program on Flink cluster whose job-manager is configured in this client's configuration.
      CompletableFuture<String> stopWithSavepoint​(org.apache.flink.api.common.JobID jobId, boolean advanceToEndOfTime, String savepointDirectory, org.apache.flink.core.execution.SavepointFormatType formatType)
      Stops a program on Flink cluster whose job-manager is configured in this client's configuration.
      CompletableFuture<String> stopWithSavepoint​(org.apache.flink.api.common.JobID jobId, boolean advanceToEndOfTime, String savepointDirectory, org.apache.flink.core.execution.SavepointFormatType formatType, boolean isDetachedMode)  
      CompletableFuture<org.apache.flink.api.common.JobID> submitJob​(org.apache.flink.runtime.jobgraph.JobGraph jobGraph)
      Submit the given JobGraph to the cluster.
      CompletableFuture<Long> triggerCheckpoint​(org.apache.flink.api.common.JobID jobId, org.apache.flink.core.execution.CheckpointType checkpointType)
      Triggers a checkpoint for the job identified by the job id.
      CompletableFuture<String> triggerDetachedSavepoint​(org.apache.flink.api.common.JobID jobId, String savepointDirectory, org.apache.flink.core.execution.SavepointFormatType formatType)
      Triggers a detached savepoint for the job identified by the job id.
      CompletableFuture<String> triggerSavepoint​(org.apache.flink.api.common.JobID jobId, String savepointDirectory, org.apache.flink.core.execution.SavepointFormatType formatType)
      Triggers a savepoint for the job identified by the job id.
      CompletableFuture<org.apache.flink.runtime.messages.Acknowledge> updateJobResourceRequirements​(org.apache.flink.api.common.JobID jobId, org.apache.flink.runtime.jobgraph.JobResourceRequirements jobResourceRequirements)
      Update JobResourceRequirements of a given job.

    • Constructor Detail

      • RestClusterClient

        public RestClusterClient​(org.apache.flink.configuration.Configuration config,
                         T clusterId)
                  throws Exception
        Throws:
        Exception

      • RestClusterClient

        public RestClusterClient​(org.apache.flink.configuration.Configuration config,
                         T clusterId,
                         org.apache.flink.runtime.highavailability.ClientHighAvailabilityServicesFactory factory)
                  throws Exception
        Throws:
        Exception

    • Method Detail

      • getFlinkConfiguration

        public org.apache.flink.configuration.Configuration getFlinkConfiguration()
        Description copied from interface: ClusterClient
        Return the Flink configuration object.
        Specified by:
        getFlinkConfiguration in interface ClusterClient<T>
        Returns:
        The Flink configuration object

      • getJobDetails

        public CompletableFuture<org.apache.flink.runtime.rest.messages.job.JobDetailsInfo> getJobDetails​(org.apache.flink.api.common.JobID jobId)
        Requests the job details.
        Parameters:
        jobId - The job id
        Returns:
        Job details

      • getJobStatus

        public CompletableFuture<org.apache.flink.api.common.JobStatus> getJobStatus​(org.apache.flink.api.common.JobID jobId)
        Description copied from interface: ClusterClient
        Requests the JobStatus of the job with the given JobID.
        Specified by:
        getJobStatus in interface ClusterClient<T>

      • requestJobResult

        public CompletableFuture<org.apache.flink.runtime.jobmaster.JobResult> requestJobResult​(@Nonnull
                                                                                        org.apache.flink.api.common.JobID jobId)
        Requests the JobResult for the given JobID. The method retries multiple times to poll the JobResult before giving up.
        Specified by:
        requestJobResult in interface ClusterClient<T>
        Parameters:
        jobId - specifying the job for which to retrieve the JobResult
        Returns:
        Future which is completed with the JobResult once the job has completed or with a failure if the JobResult could not be retrieved.

      • submitJob

        public CompletableFuture<org.apache.flink.api.common.JobID> submitJob​(@Nonnull
                                                                      org.apache.flink.runtime.jobgraph.JobGraph jobGraph)
        Description copied from interface: ClusterClient
        Submit the given JobGraph to the cluster.
        Specified by:
        submitJob in interface ClusterClient<T>
        Parameters:
        jobGraph - to submit
        Returns:
        JobID of the submitted job

      • cancel

        public CompletableFuture<org.apache.flink.runtime.messages.Acknowledge> cancel​(org.apache.flink.api.common.JobID jobID)
        Description copied from interface: ClusterClient
        Cancels a job identified by the job id.
        Specified by:
        cancel in interface ClusterClient<T>
        Parameters:
        jobID - the job id

      • stopWithSavepoint

        public CompletableFuture<String> stopWithSavepoint​(org.apache.flink.api.common.JobID jobId,
                                                   boolean advanceToEndOfTime,
                                                   @Nullable
                                                   String savepointDirectory,
                                                   org.apache.flink.core.execution.SavepointFormatType formatType)
        Description copied from interface: ClusterClient
        Stops a program on Flink cluster whose job-manager is configured in this client's configuration. Stopping works only for streaming programs. Be aware, that the program might continue to run for a while after sending the stop command, because after sources stopped to emit data all operators need to finish processing.
        Specified by:
        stopWithSavepoint in interface ClusterClient<T>
        Parameters:
        jobId - the job ID of the streaming program to stop
        advanceToEndOfTime - flag indicating if the source should inject a MAX_WATERMARK in the pipeline
        savepointDirectory - directory the savepoint should be written to
        formatType - a binary format of the savepoint
        Returns:
        a CompletableFuture containing the path where the savepoint is located

      • stopWithDetachedSavepoint

        public CompletableFuture<String> stopWithDetachedSavepoint​(org.apache.flink.api.common.JobID jobId,
                                                           boolean advanceToEndOfTime,
                                                           @Nullable
                                                           String savepointDirectory,
                                                           org.apache.flink.core.execution.SavepointFormatType formatType)
        Description copied from interface: ClusterClient
        Stops a program on Flink cluster whose job-manager is configured in this client's configuration. Stopping works only for streaming programs. Be aware, that the program might continue to run for a while after sending the stop command, because after sources stopped to emit data all operators need to finish processing.
        Specified by:
        stopWithDetachedSavepoint in interface ClusterClient<T>
        Parameters:
        jobId - the job ID of the streaming program to stop
        advanceToEndOfTime - flag indicating if the source should inject a MAX_WATERMARK in the pipeline
        savepointDirectory - directory the savepoint should be written to
        formatType - a binary format of the savepoint
        Returns:
        the savepoint trigger id

      • cancelWithSavepoint

        public CompletableFuture<String> cancelWithSavepoint​(org.apache.flink.api.common.JobID jobId,
                                                     @Nullable
                                                     String savepointDirectory,
                                                     org.apache.flink.core.execution.SavepointFormatType formatType)
        Description copied from interface: ClusterClient
        Cancels a job identified by the job id and triggers a savepoint.
        Specified by:
        cancelWithSavepoint in interface ClusterClient<T>
        Parameters:
        jobId - the job id
        savepointDirectory - directory the savepoint should be written to
        formatType - a binary format of the savepoint
        Returns:
        future of path where the savepoint is located

      • triggerSavepoint

        public CompletableFuture<String> triggerSavepoint​(org.apache.flink.api.common.JobID jobId,
                                                  @Nullable
                                                  String savepointDirectory,
                                                  org.apache.flink.core.execution.SavepointFormatType formatType)
        Description copied from interface: ClusterClient
        Triggers a savepoint for the job identified by the job id. The savepoint will be written to the given savepoint directory, or CheckpointingOptions.SAVEPOINT_DIRECTORY if it is null.
        Specified by:
        triggerSavepoint in interface ClusterClient<T>
        Parameters:
        jobId - job id
        savepointDirectory - directory the savepoint should be written to
        formatType - a binary format of the savepoint
        Returns:
        path future where the savepoint is located

      • triggerCheckpoint

        public CompletableFuture<Long> triggerCheckpoint​(org.apache.flink.api.common.JobID jobId,
                                                 org.apache.flink.core.execution.CheckpointType checkpointType)
        Description copied from interface: ClusterClient
        Triggers a checkpoint for the job identified by the job id. The checkpoint will be written to the checkpoint directory for the job.
        Specified by:
        triggerCheckpoint in interface ClusterClient<T>
        Parameters:
        jobId - job id
        checkpointType - the checkpoint type (configured / full / incremental)

      • triggerDetachedSavepoint

        public CompletableFuture<String> triggerDetachedSavepoint​(org.apache.flink.api.common.JobID jobId,
                                                          @Nullable
                                                          String savepointDirectory,
                                                          org.apache.flink.core.execution.SavepointFormatType formatType)
        Description copied from interface: ClusterClient
        Triggers a detached savepoint for the job identified by the job id. The savepoint will be written to the given savepoint directory, or CheckpointingOptions.SAVEPOINT_DIRECTORY if it is null. Notice that: the detached savepoint will return with a savepoint trigger id instead of the path future, that means the client will return very quickly.
        Specified by:
        triggerDetachedSavepoint in interface ClusterClient<T>
        Parameters:
        jobId - job id
        savepointDirectory - directory the savepoint should be written to
        formatType - a binary format of the savepoint
        Returns:
        the savepoint trigger id

      • sendCoordinationRequest

        public CompletableFuture<org.apache.flink.runtime.operators.coordination.CoordinationResponse> sendCoordinationRequest​(org.apache.flink.api.common.JobID jobId,
                                                                                                                       org.apache.flink.runtime.jobgraph.OperatorID operatorId,
                                                                                                                       org.apache.flink.runtime.operators.coordination.CoordinationRequest request)
        Description copied from interface: ClusterClient
        Sends out a request to a specified coordinator and return the response.
        Specified by:
        sendCoordinationRequest in interface ClusterClient<T>
        Parameters:
        jobId - specifies the job which the coordinator belongs to
        operatorId - specifies which coordinator to receive the request
        request - the request to send
        Returns:
        the response from the coordinator

      • stopWithSavepoint

        public CompletableFuture<String> stopWithSavepoint​(org.apache.flink.api.common.JobID jobId,
                                                   boolean advanceToEndOfTime,
                                                   @Nullable
                                                   String savepointDirectory,
                                                   org.apache.flink.core.execution.SavepointFormatType formatType,
                                                   boolean isDetachedMode)

      • getAccumulators

        public CompletableFuture<Map<String,​Object>> getAccumulators​(org.apache.flink.api.common.JobID jobID,
                                                                   ClassLoader loader)
        Description copied from interface: ClusterClient
        Requests and returns the accumulators for the given job identifier. Accumulators can be requested while a is running or after it has finished.
        Specified by:
        getAccumulators in interface ClusterClient<T>
        Parameters:
        jobID - The job identifier of a job.
        loader - The class loader for deserializing the accumulator results.
        Returns:
        A Map containing the accumulator's name and its value.

      • listJobs

        public CompletableFuture<Collection<org.apache.flink.runtime.client.JobStatusMessage>> listJobs()
        Description copied from interface: ClusterClient
        Lists the currently running and finished jobs on the cluster.
        Specified by:
        listJobs in interface ClusterClient<T>
        Returns:
        future collection of running and finished jobs

      • getClusterId

        public T getClusterId()
        Description copied from interface: ClusterClient
        Returns the cluster id identifying the cluster to which the client is connected.
        Specified by:
        getClusterId in interface ClusterClient<T>
        Returns:
        cluster id of the connected cluster

      • disposeSavepoint

        public CompletableFuture<org.apache.flink.runtime.messages.Acknowledge> disposeSavepoint​(String savepointPath)
        Description copied from interface: ClusterClient
        Dispose the savepoint under the given path.
        Specified by:
        disposeSavepoint in interface ClusterClient<T>
        Parameters:
        savepointPath - path to the savepoint to be disposed
        Returns:
        acknowledge future of the dispose action

      • invalidateClusterDataset

        public CompletableFuture<Void> invalidateClusterDataset​(org.apache.flink.util.AbstractID clusterDatasetId)
        Description copied from interface: ClusterClient
        Invalidate the cached intermediate dataset with the given id.
        Specified by:
        invalidateClusterDataset in interface ClusterClient<T>
        Parameters:
        clusterDatasetId - id of the cluster dataset to be invalidated.
        Returns:
        Future which will be completed when the cached dataset is invalidated.

      • reportHeartbeat

        public CompletableFuture<Void> reportHeartbeat​(org.apache.flink.api.common.JobID jobId,
                                               long expiredTimestamp)
        Description copied from interface: ClusterClient
        The client reports the heartbeat to the dispatcher for aliveness.
        Specified by:
        reportHeartbeat in interface ClusterClient<T>
        Parameters:
        jobId - The jobId for the client and the job.
        Returns:

      • shutDownCluster

        public void shutDownCluster()
        Description copied from interface: ClusterClient
        Shut down the cluster that this client communicate with.
        Specified by:
        shutDownCluster in interface ClusterClient<T>

      • updateJobResourceRequirements

        public CompletableFuture<org.apache.flink.runtime.messages.Acknowledge> updateJobResourceRequirements​(org.apache.flink.api.common.JobID jobId,
                                                                                                      org.apache.flink.runtime.jobgraph.JobResourceRequirements jobResourceRequirements)
        Update JobResourceRequirements of a given job.
        Parameters:
        jobId - jobId specifies the job for which to change the resource requirements
        jobResourceRequirements - new resource requirements for the provided job
        Returns:
        Future which is completed upon successful operation.

      • getClusterOverview

        public CompletableFuture<org.apache.flink.runtime.rest.handler.legacy.messages.ClusterOverviewWithVersion> getClusterOverview()
        Get an overview of the Flink cluster.
        Returns:
        Future with the cluster overview.

      • sendRequest

        @VisibleForTesting
public <M extends org.apache.flink.runtime.rest.messages.MessageHeaders<R,​P,​U>,​U extends org.apache.flink.runtime.rest.messages.MessageParameters,​R extends org.apache.flink.runtime.rest.messages.RequestBody,​P extends org.apache.flink.runtime.rest.messages.ResponseBody> CompletableFuture<P> sendRequest​(M messageHeaders,
                                                                                                                                                                                                                                                                                                                                             U messageParameters,
                                                                                                                                                                                                                                                                                                                                             R request)