Package org.apache.flink.state.forst.fs.cache

Class FileBasedCache

java.lang.Object

org.apache.flink.state.forst.fs.cache.DoubleListLru<String,FileCacheEntry>

org.apache.flink.state.forst.fs.cache.FileBasedCache

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<org.apache.flink.api.java.tuple.Tuple2<String,FileCacheEntry>>

public final class FileBasedCache
extends DoubleListLru<String,FileCacheEntry>
implements Closeable

A file-granularity LRU cache. Only newly generated SSTs are written to the cache, the file reading from the remote will not. Newly generated SSTs are written to the original file system and cache simultaneously, so, the cached file can be directly deleted with persisting when evicting. The FileBasedCache, FileCacheEntry, CachedDataInputStream, and CachedDataOutputStream classes work together to implement a file-based caching mechanism in ForSt State Backend.

FileBasedCache manages multiple FileCacheEntry instances. Each FileCacheEntry represents a cached file and can open CachedDataInputStream for reading the file. CachedDataInputStream instances are created by FileCacheEntry and can read data from either the cached file or the original file, depending on the cache entry's state. It has internal stream status to indicate the current reading source. CachedDataOutputStream instances are created by FileBasedCache and write data to both the original and cached files, creating a new cache entry in the cache when the writing is finished.

Constructor Summary

Constructors

Constructor	Description
FileBasedCache(org.apache.flink.configuration.ReadableConfig configuration, CacheLimitPolicy cacheLimitPolicy, org.apache.flink.core.fs.FileSystem cacheFs, org.apache.flink.core.fs.Path basePath, org.apache.flink.metrics.MetricGroup metricGroup)

Method Summary

Modifier and Type	Method	Description
void	addFirst(String key, FileCacheEntry value)	Adds a new entry to the front of the cache.
void	addSecond(String key, FileCacheEntry value)	Inserts a new entry at the middle of the cache.
void	close()
CachedDataOutputStream	create(org.apache.flink.core.fs.FSDataOutputStream originalOutputStream, org.apache.flink.core.fs.Path path)
void	delete(org.apache.flink.core.fs.Path path)
FileCacheEntry	get(String key, boolean affectOrder)	Retrieves the value associated with the specified key.
void	incHitCounter()
void	incMissCounter()
static boolean	isFlinkThread()	Checks if the current thread is a Flink thread.
CachedDataInputStream	open(org.apache.flink.core.fs.Path path, org.apache.flink.core.fs.FSDataInputStream originalStream)
void	registerInCache(org.apache.flink.core.fs.Path originalPath, long size)	Directly insert in cache when restoring.
FileCacheEntry	remove(String key)	Removes the entry associated with the specified key from the cache.
static void	setFlinkThread()	Sets the current thread as a Flink thread.
static void	unsetFlinkThread()

Methods inherited from class org.apache.flink.state.forst.fs.cache.DoubleListLru

descendingIterator, getSecondSize, isEmpty, iterator, moveMiddleBack, moveMiddleFront, size

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

FileBasedCache

public FileBasedCache(org.apache.flink.configuration.ReadableConfig configuration,
                      CacheLimitPolicy cacheLimitPolicy,
                      org.apache.flink.core.fs.FileSystem cacheFs,
                      org.apache.flink.core.fs.Path basePath,
                      org.apache.flink.metrics.MetricGroup metricGroup)

Method Detail

setFlinkThread

public static void setFlinkThread()

Sets the current thread as a Flink thread. This method is used to mark the thread as a Flink thread, which can be used to determine whether the file access would affect the LRU cache order, or metrics updates.

unsetFlinkThread

@VisibleForTesting
public static void unsetFlinkThread()

isFlinkThread

public static boolean isFlinkThread()

Checks if the current thread is a Flink thread. This method returns a boolean indicating whether the current thread has been marked as a Flink thread using the setFlinkThread() method.

Returns:

true if the current thread is a Flink thread, false otherwise.

incHitCounter

public void incHitCounter()

incMissCounter

public void incMissCounter()

open

public CachedDataInputStream open(org.apache.flink.core.fs.Path path,
                                  org.apache.flink.core.fs.FSDataInputStream originalStream)
                           throws IOException

Throws:

IOException

create

public CachedDataOutputStream create(org.apache.flink.core.fs.FSDataOutputStream originalOutputStream,
                                     org.apache.flink.core.fs.Path path)
                              throws IOException

Throws:

IOException

delete

public void delete(org.apache.flink.core.fs.Path path)

get

public FileCacheEntry get(String key,
                          boolean affectOrder)

Description copied from class: DoubleListLru

Retrieves the value associated with the specified key. Optionally affects the order of the entries in the cache.

Overrides:

get in class DoubleListLru<String,FileCacheEntry>

Parameters:

key - the key of the entry

affectOrder - true if the order of the entries should be affected, false otherwise

Returns:

the value associated with the key, or null if the key is not found

addFirst

public void addFirst(String key,
                     FileCacheEntry value)

Description copied from class: DoubleListLru

Adds a new entry to the front of the cache.

Overrides:

addFirst in class DoubleListLru<String,FileCacheEntry>

Parameters:

key - the key of the entry

value - the value of the entry

addSecond

public void addSecond(String key,
                      FileCacheEntry value)

Description copied from class: DoubleListLru

Inserts a new entry at the middle of the cache.

Overrides:

addSecond in class DoubleListLru<String,FileCacheEntry>

Parameters:

key - the key of the entry

value - the value of the entry

remove

public FileCacheEntry remove(String key)

Description copied from class: DoubleListLru

Removes the entry associated with the specified key from the cache.

Overrides:

remove in class DoubleListLru<String,FileCacheEntry>

Parameters:

key - the key of the entry to be removed

Returns:

the value of the removed entry, or null if the key is not found

registerInCache

public void registerInCache(org.apache.flink.core.fs.Path originalPath,
                            long size)

Directly insert in cache when restoring.

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException