Package org.apache.flink.contrib.streaming.state

Class RocksDBStateBackend

  • All Implemented Interfaces:
    Serializable, org.apache.flink.runtime.state.CheckpointStorage, org.apache.flink.runtime.state.ConfigurableStateBackend, org.apache.flink.runtime.state.StateBackend
    @Deprecated
public class RocksDBStateBackend
extends org.apache.flink.runtime.state.AbstractManagedMemoryStateBackend
implements org.apache.flink.runtime.state.CheckpointStorage, org.apache.flink.runtime.state.ConfigurableStateBackend
    Deprecated.
    IMPORTANT RocksDBStateBackend is deprecated in favor of EmbeddedRocksDBStateBackend. and FileSystemCheckpointStorage. This change does not affect the runtime characteristics of your Jobs and is simply an API change to help better communicate the ways Flink separates local state storage from fault tolerance. Jobs can be upgraded without loss of state. If configuring your state backend via the StreamExecutionEnvironment please make the following changes. 
    
 		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
 		env.setStateBackend(new EmbeddedRocksDBStateBackend());
 		env.getCheckpointConfig().setCheckpointStorage("hdfs://checkpoints");

    If you are configuring your state backend via the config.yaml no changes are required.

    A State Backend that stores its state in RocksDB. This state backend can store very large state that exceeds memory and spills to disk.

    All key/value state (including windows) is stored in the key/value index of RocksDB. For persistence against loss of machines, checkpoints take a snapshot of the RocksDB database, and persist that snapshot in a file system (by default) or another configurable state backend.

    The behavior of the RocksDB instances can be parametrized by setting RocksDB Options using the methods setPredefinedOptions(PredefinedOptions) and setRocksDBOptions(RocksDBOptionsFactory).

    See Also:
    Serialized Form

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  RocksDBStateBackend.PriorityQueueStateType
      Deprecated.

      • Nested classes/interfaces inherited from interface org.apache.flink.runtime.state.StateBackend

        org.apache.flink.runtime.state.StateBackend.CustomInitializationMetrics, org.apache.flink.runtime.state.StateBackend.KeyedStateBackendParameters<K extends Object>, org.apache.flink.runtime.state.StateBackend.OperatorStateBackendParameters

    • Field Summary

      • Fields inherited from class org.apache.flink.runtime.state.AbstractStateBackend

        latencyTrackingConfigBuilder

    • Constructor Summary

      Constructors 
      Constructor Description
      RocksDBStateBackend​(String checkpointDataUri)
      Deprecated.
      Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.
      RocksDBStateBackend​(String checkpointDataUri, boolean enableIncrementalCheckpointing)
      Deprecated.
      Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.
      RocksDBStateBackend​(URI checkpointDataUri)
      Deprecated.
      Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.
      RocksDBStateBackend​(URI checkpointDataUri, boolean enableIncrementalCheckpointing)
      Deprecated.
      Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.
      RocksDBStateBackend​(org.apache.flink.runtime.state.AbstractStateBackend checkpointStreamBackend)
      Deprecated.
      Use RocksDBStateBackend(StateBackend) instead.
      RocksDBStateBackend​(org.apache.flink.runtime.state.AbstractStateBackend checkpointStreamBackend, boolean enableIncrementalCheckpointing)
      Deprecated.
      Use RocksDBStateBackend(StateBackend, TernaryBoolean) instead.
      RocksDBStateBackend​(org.apache.flink.runtime.state.StateBackend checkpointStreamBackend)
      Deprecated.
      Creates a new RocksDBStateBackend that uses the given state backend to store its checkpoint data streams.
      RocksDBStateBackend​(org.apache.flink.runtime.state.StateBackend checkpointStreamBackend, org.apache.flink.util.TernaryBoolean enableIncrementalCheckpointing)
      Deprecated.
      Creates a new RocksDBStateBackend that uses the given state backend to store its checkpoint data streams.

    • Constructor Detail

      • RocksDBStateBackend

        public RocksDBStateBackend​(String checkpointDataUri)
                    throws IOException
        Deprecated.
        Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.

        A state backend that stores checkpoints in HDFS or S3 must specify the file system host and port in the URI, or have the Hadoop configuration that describes the file system (host / high-availability group / possibly credentials) either referenced from the Flink config, or included in the classpath.

        Parameters:
        checkpointDataUri - The URI describing the filesystem and path to the checkpoint data directory.
        Throws:
        IOException - Thrown, if no file system can be found for the scheme in the URI.

      • RocksDBStateBackend

        public RocksDBStateBackend​(String checkpointDataUri,
                           boolean enableIncrementalCheckpointing)
                    throws IOException
        Deprecated.
        Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.

        A state backend that stores checkpoints in HDFS or S3 must specify the file system host and port in the URI, or have the Hadoop configuration that describes the file system (host / high-availability group / possibly credentials) either referenced from the Flink config, or included in the classpath.

        Parameters:
        checkpointDataUri - The URI describing the filesystem and path to the checkpoint data directory.
        enableIncrementalCheckpointing - True if incremental checkpointing is enabled.
        Throws:
        IOException - Thrown, if no file system can be found for the scheme in the URI.

      • RocksDBStateBackend

        public RocksDBStateBackend​(URI checkpointDataUri)
                    throws IOException
        Deprecated.
        Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.

        A state backend that stores checkpoints in HDFS or S3 must specify the file system host and port in the URI, or have the Hadoop configuration that describes the file system (host / high-availability group / possibly credentials) either referenced from the Flink config, or included in the classpath.

        Parameters:
        checkpointDataUri - The URI describing the filesystem and path to the checkpoint data directory.
        Throws:
        IOException - Thrown, if no file system can be found for the scheme in the URI.

      • RocksDBStateBackend

        public RocksDBStateBackend​(URI checkpointDataUri,
                           boolean enableIncrementalCheckpointing)
                    throws IOException
        Deprecated.
        Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.

        A state backend that stores checkpoints in HDFS or S3 must specify the file system host and port in the URI, or have the Hadoop configuration that describes the file system (host / high-availability group / possibly credentials) either referenced from the Flink config, or included in the classpath.

        Parameters:
        checkpointDataUri - The URI describing the filesystem and path to the checkpoint data directory.
        enableIncrementalCheckpointing - True if incremental checkpointing is enabled.
        Throws:
        IOException - Thrown, if no file system can be found for the scheme in the URI.

      • RocksDBStateBackend

        public RocksDBStateBackend​(org.apache.flink.runtime.state.StateBackend checkpointStreamBackend)
        Deprecated.
        Creates a new RocksDBStateBackend that uses the given state backend to store its checkpoint data streams. Typically, one would supply a filesystem or database state backend here where the snapshots from RocksDB would be stored.

        The snapshots of the RocksDB state will be stored using the given backend's CheckpointStorage.createCheckpointStorage(JobID).

        Parameters:
        checkpointStreamBackend - The backend write the checkpoint streams to.

      • RocksDBStateBackend

        public RocksDBStateBackend​(org.apache.flink.runtime.state.StateBackend checkpointStreamBackend,
                           org.apache.flink.util.TernaryBoolean enableIncrementalCheckpointing)
        Deprecated.
        Creates a new RocksDBStateBackend that uses the given state backend to store its checkpoint data streams. Typically, one would supply a filesystem or database state backend here where the snapshots from RocksDB would be stored.

        The snapshots of the RocksDB state will be stored using the given backend's StateBackend#createCheckpointStorage(JobID).

        Parameters:
        checkpointStreamBackend - The backend write the checkpoint streams to.
        enableIncrementalCheckpointing - True if incremental checkpointing is enabled.

    • Method Detail

      • configure

        public RocksDBStateBackend configure​(org.apache.flink.configuration.ReadableConfig config,
                                     ClassLoader classLoader)
        Deprecated.
        Creates a copy of this state backend that uses the values defined in the configuration for fields where that were not yet specified in this state backend.
        Specified by:
        configure in interface org.apache.flink.runtime.state.ConfigurableStateBackend
        Parameters:
        config - The configuration.
        classLoader - The class loader.
        Returns:
        The re-configured variant of the state backend

      • getCheckpointBackend

        public org.apache.flink.runtime.state.StateBackend getCheckpointBackend()
        Deprecated.
        Gets the state backend that this RocksDB state backend uses to persist its bytes to.

        This RocksDB state backend only implements the RocksDB specific parts, it relies on the 'CheckpointBackend' to persist the checkpoint and savepoint bytes streams.

      • supportsNoClaimRestoreMode

        public boolean supportsNoClaimRestoreMode()
        Deprecated.
        Specified by:
        supportsNoClaimRestoreMode in interface org.apache.flink.runtime.state.StateBackend

      • supportsSavepointFormat

        public boolean supportsSavepointFormat​(org.apache.flink.core.execution.SavepointFormatType formatType)
        Deprecated.
        Specified by:
        supportsSavepointFormat in interface org.apache.flink.runtime.state.StateBackend

      • resolveCheckpoint

        public org.apache.flink.runtime.state.CompletedCheckpointStorageLocation resolveCheckpoint​(String pointer)
                                                                                    throws IOException
        Deprecated.
        Specified by:
        resolveCheckpoint in interface org.apache.flink.runtime.state.CheckpointStorage
        Throws:
        IOException

      • createCheckpointStorage

        public org.apache.flink.runtime.state.CheckpointStorageAccess createCheckpointStorage​(org.apache.flink.api.common.JobID jobId)
                                                                               throws IOException
        Deprecated.
        Specified by:
        createCheckpointStorage in interface org.apache.flink.runtime.state.CheckpointStorage
        Throws:
        IOException

      • createKeyedStateBackend

        public <K> org.apache.flink.runtime.state.AbstractKeyedStateBackend<K> createKeyedStateBackend​(org.apache.flink.runtime.state.StateBackend.KeyedStateBackendParameters<K> parameters)
                                                                                        throws IOException
        Deprecated.
        Specified by:
        createKeyedStateBackend in interface org.apache.flink.runtime.state.StateBackend
        Specified by:
        createKeyedStateBackend in class org.apache.flink.runtime.state.AbstractManagedMemoryStateBackend
        Throws:
        IOException

      • createOperatorStateBackend

        public org.apache.flink.runtime.state.OperatorStateBackend createOperatorStateBackend​(org.apache.flink.runtime.state.StateBackend.OperatorStateBackendParameters parameters)
                                                                               throws Exception
        Deprecated.
        Specified by:
        createOperatorStateBackend in interface org.apache.flink.runtime.state.StateBackend
        Specified by:
        createOperatorStateBackend in class org.apache.flink.runtime.state.AbstractStateBackend
        Throws:
        Exception

      • setDbStoragePath

        public void setDbStoragePath​(String path)
        Deprecated.
        Sets the path where the RocksDB local database files should be stored on the local file system. Setting this path overrides the default behavior, where the files are stored across the configured temp directories.

        Passing null to this function restores the default behavior, where the configured temp directories will be used.

        Parameters:
        path - The path where the local RocksDB database files are stored.

      • setDbStoragePaths

        public void setDbStoragePaths​(String... paths)
        Deprecated.
        Sets the directories in which the local RocksDB database puts its files (like SST and metadata files). These directories do not need to be persistent, they can be ephemeral, meaning that they are lost on a machine failure, because state in RocksDB is persisted in checkpoints.

        If nothing is configured, these directories default to the TaskManager's local temporary file directories.

        Each distinct state will be stored in one path, but when the state backend creates multiple states, they will store their files on different paths.

        Passing null to this function restores the default behavior, where the configured temp directories will be used.

        Parameters:
        paths - The paths across which the local RocksDB database files will be spread.

      • getDbStoragePaths

        public String[] getDbStoragePaths()
        Deprecated.
        Gets the configured local DB storage paths, or null, if none were configured.

        Under these directories on the TaskManager, RocksDB stores its SST files and metadata files. These directories do not need to be persistent, they can be ephermeral, meaning that they are lost on a machine failure, because state in RocksDB is persisted in checkpoints.

        If nothing is configured, these directories default to the TaskManager's local temporary file directories.

      • isIncrementalCheckpointsEnabled

        public boolean isIncrementalCheckpointsEnabled()
        Deprecated.
        Gets whether incremental checkpoints are enabled for this state backend.

      • getPriorityQueueStateType

        public RocksDBStateBackend.PriorityQueueStateType getPriorityQueueStateType()
        Deprecated.
        Gets the type of the priority queue state. It will fallback to the default value, if it is not explicitly set.
        Returns:
        The type of the priority queue state.

      • setPriorityQueueStateType

        public void setPriorityQueueStateType​(RocksDBStateBackend.PriorityQueueStateType priorityQueueStateType)
        Deprecated.
        Sets the type of the priority queue state. It will fallback to the default value, if it is not explicitly set.

      • setPredefinedOptions

        public void setPredefinedOptions​(@Nonnull
                                 PredefinedOptions options)
        Deprecated.
        Sets the predefined options for RocksDB.

        If user-configured options within RocksDBConfigurableOptions is set (through config.yaml) or a user-defined options factory is set (via setRocksDBOptions(RocksDBOptionsFactory)), then the options from the factory are applied on top of the here specified predefined options and customized options.

        Parameters:
        options - The options to set (must not be null).

      • setRocksDBOptions

        public void setRocksDBOptions​(RocksDBOptionsFactory optionsFactory)
        Deprecated.
        Sets Options for the RocksDB instances. Because the options are not serializable and hold native code references, they must be specified through a factory.

        The options created by the factory here are applied on top of the pre-defined options profile selected via setPredefinedOptions(PredefinedOptions). If the pre-defined options profile is the default (PredefinedOptions.DEFAULT), then the factory fully controls the RocksDB options.

        Parameters:
        optionsFactory - The options factory that lazily creates the RocksDB options.

      • getNumberOfTransferThreads

        public int getNumberOfTransferThreads()
        Deprecated.
        Gets the number of threads used to transfer files while snapshotting/restoring.

      • setNumberOfTransferThreads

        public void setNumberOfTransferThreads​(int numberOfTransferThreads)
        Deprecated.
        Sets the number of threads used to transfer files while snapshotting/restoring.
        Parameters:
        numberOfTransferThreads - The number of threads used to transfer files while snapshotting/restoring.

      • getWriteBatchSize

        public long getWriteBatchSize()
        Deprecated.
        Gets the max batch size will be used in RocksDBWriteBatchWrapper.

      • setWriteBatchSize

        public void setWriteBatchSize​(long writeBatchSize)
        Deprecated.
        Sets the max batch size will be used in RocksDBWriteBatchWrapper, no positive value will disable memory size controller, just use item count controller.
        Parameters:
        writeBatchSize - The size will used to be used in RocksDBWriteBatchWrapper.