Package org.apache.flink.api.common.state

Class StateTtlConfig.Builder

    • Method Detail

      • cleanupIncrementally

        @Nonnull
public StateTtlConfig.Builder cleanupIncrementally​(@Nonnegative
                                                   int cleanupSize,
                                                   boolean runCleanupForEveryRecord)
        Cleanup expired state incrementally cleanup local state.

        Upon every state access this cleanup strategy checks a bunch of state keys for expiration and cleans up expired ones. It keeps a lazy iterator through all keys with relaxed consistency if backend supports it. This way all keys should be regularly checked and cleaned eventually over time if any state is constantly being accessed.

        Additionally to the incremental cleanup upon state access, it can also run per every record. Caution: if there are a lot of registered states using this option, they all will be iterated for every record to check if there is something to cleanup.

        Note: if no access happens to this state or no records are processed in case of runCleanupForEveryRecord, expired state will persist.

        Note: Time spent for the incremental cleanup increases record processing latency.

        Note: At the moment incremental cleanup is implemented only for Heap state backend. Setting it for RocksDB will have no effect.

        Note: If heap state backend is used with synchronous snapshotting, the global iterator keeps a copy of all keys while iterating because of its specific implementation which does not support concurrent modifications. Enabling of this feature will increase memory consumption then. Asynchronous snapshotting does not have this problem.

        Parameters:
        cleanupSize - max number of keys pulled from queue for clean up upon state touch for any key
        runCleanupForEveryRecord - run incremental cleanup per each processed record

      • cleanupInRocksdbCompactFilter

        @Nonnull
public StateTtlConfig.Builder cleanupInRocksdbCompactFilter​(long queryTimeAfterNumEntries)
        Cleanup expired state while Rocksdb compaction is running.

        RocksDB compaction filter will query current timestamp, used to check expiration, from Flink every time after processing queryTimeAfterNumEntries number of state entries. Updating the timestamp more often can improve cleanup speed but it decreases compaction performance because it uses JNI call from native code.

        Parameters:
        queryTimeAfterNumEntries - number of state entries to process by compaction filter before updating current timestamp

      • cleanupInRocksdbCompactFilter

        @Nonnull
public StateTtlConfig.Builder cleanupInRocksdbCompactFilter​(long queryTimeAfterNumEntries,
                                                            java.time.Duration periodicCompactionTime)
        Cleanup expired state while Rocksdb compaction is running.

        RocksDB compaction filter will query current timestamp, used to check expiration, from Flink every time after processing queryTimeAfterNumEntries number of state entries. Updating the timestamp more often can improve cleanup speed but it decreases compaction performance because it uses JNI call from native code.

        Periodic compaction could speed up expired state entries cleanup, especially for state entries rarely accessed. Files older than this value will be picked up for compaction, and re-written to the same level as they were before. It makes sure a file goes through compaction filters periodically.

        Parameters:
        queryTimeAfterNumEntries - number of state entries to process by compaction filter before updating current timestamp
        periodicCompactionTime - periodic compaction which could speed up expired state cleanup. 0 means turning off periodic compaction.