public abstract class CheckedFileIO extends Object implements AutoCloseable
For backward compatibility purpose, this class also supports no checksum mode in which case, the application can continue using the package inheriting this class but there will not be any checksum computed or stored for the file. And, the file will behave like any normal non-checksum file.
Other than abstracting checksum capabilities, this class also provides functionality for migration from non-checksum to checksum or vice versa, and checksum validation of file content.
NOTE: This class is not thread-safe nor it guarantees a single instantiation on a server. It will be responsibility of the caller to make sure they take care of concurrency and exclusive access related dependencies, otherwise the checksum will be corrupted.
Usage: ------ - Application / user is supposed to decide the super block size. - For NO_CHECKSUM algorithm type this will be 0 (as the super block is not present and the data is written starting at position 0). - The maximum supported super block size for checksum enabled algorithms is defined in SUPER_BLOCK_LENGTH_MAX as 4KB. ChecksumAlgorithm CHECKSUM_ALGORITHM = CheckedFileIO.ChecksumAlgorithm.ADLER; short CHECKSUM_SUPER_BLOCK_LENGTH = 2000; - Application needs to ensure that the file being opened ends with a suffix based on the CHECKSUM_ALGORITHM. - The helper method getValidPath(...) will validate and if needed will generate the desired filename. Path actualPath = getValidPath(CHECKSUM_ALGORITHM, path) CheckedFileIO fileIO = CheckedFileIO.open(CHECKSUM_ALGORITHM, CHECKSUM_SUPER_BLOCK_LENGTH, actualPath, ...) - Write operation. fileIO.write(0, ByteBuffer.wrap("hello".getBytes())); - Read data from file into buffer ByteBuffer bb = ByteBuffer.allocate(5).order(BYTE_ORDER); fileIO.read(bb, 0); bb.flip(); fileIO.close() - If needed to migrate from one format to another, application can use the static method migrateFile(...). - This method is generally used during application bootstrap before it starts operating on checksum supported file. CheckedFileIO.migrateFile(srcPath, destPath, destSuperBlockLength, ....) NOTE: Application should always open a file with the same value for super block size. For only reading StandardOpenOption.READ should be provided, while for writing both StandardOpenOption.READ, StandardOpenOption.WRITE should be providedModifier and Type | Method and Description |
---|---|
Optional<Long> |
checksum()
Optionally return the checksum value.
|
abstract void |
close()
Closes the file channel
|
static void |
create(Path path,
Algorithm algo,
short superBlockLength)
Create a new file based on the algorithm passed as parameter.
|
abstract void |
flush()
Flush method is used to flush the file onto storage.
|
static boolean |
isValidPath(Algorithm algo,
Path path)
isValidPath is a helper util to check if the input file is in algorithm specific format
if algo = ChecksumAlgorithm.ADLER
somePath/tierMetadata.adler -> true
somePath/tierMetadata -> false
if algo = ChecksumAlgorithm.NO_CHECKSUM
somePath/tierMetadata -> true
somePath/tierMetadata.adler -> false
|
static void |
migrateFile(Path srcPath,
Path destPath,
short destSuperBlockLength)
migrateFile provides migration between various checksum formats (such as ADLER -> NO_CHECKSUM or vice versa)
This method atomically migrates source file into destination file based on the destination checksum algorithm.
|
static CheckedFileIO |
open(Path path,
boolean skipMultiChecksumValidation,
OpenOption... options)
Opens a file and returns a CheckedFileIO instance to access the file.
|
static CheckedFileIO |
open(Path path,
OpenOption... options) |
static CheckedFileIO |
openOrCreate(Path path,
Algorithm algo,
short superBlockLength,
boolean skipMultiChecksumValidation,
OpenOption... options)
Opens or creates a file, returning a CheckedFileIO instance to access the file.
|
static CheckedFileIO |
openOrCreate(Path path,
Algorithm algo,
short superBlockLength,
OpenOption... options) |
abstract long |
position()
Returns this channel's file relative position.
|
abstract void |
position(long newPosition)
Sets this channel's file position from the input relative position.
|
abstract void |
read(ByteBuffer dst,
long position)
Read data from the channel to the given byte buffer from the input relative position until there are no bytes
remaining in the buffer or the end of the file has been reached.
|
abstract long |
size()
Size of file channel based on relative position.
|
void |
transferFrom(CheckedFileIO source,
long position,
long count)
Transfer bytes to this file channel's from a readable byte channel's input relative position
|
void |
transferTo(long position,
long count,
CheckedFileIO target)
Transfer bytes from this file channel's input relative position to a writable byte channel
|
abstract void |
truncate(long size)
Truncate file channel to given size based on relative position.
|
abstract boolean |
validate()
Validate compares with the value stored in header of the file with the checksum computed of the entire file.
|
static Path |
validPath(Algorithm algo,
Path path)
getValidPath converts given input path based on the algorithm parameter to a valid algorithm specific path format
algo = ADLER
case-1a: path = somePath/tierMetadata -> somePath/tierMetadata.adler
case-1b: path = somePath/tierMetadata.adler -> somePath/tierMetadata.adler
algo = NO_CHECKSUM
case-2a: path = somePath/tierMetadata -> somePath/tierMetadata
case-2b: path = somePath/tierMetadata.adler -> somePath/tierMetadata
|
static void |
verifyOnlyValidFileExists(Algorithm algo,
Path path)
verifyOnlyValidFileExists is a helper util to check if the given input path is a valid ADLER or NO_CHECKSUM path.
|
abstract void |
write(ByteBuffer src)
Write the given buffer into the file channel at the relative file channel position
NOTE: File position is updated with the number of bytes written
|
abstract void |
write(ByteBuffer src,
long position)
Write the given buffer into the file channel at the input relative position in file channel.
|
public static void migrateFile(Path srcPath, Path destPath, short destSuperBlockLength) throws IOException
srcPath
- the path to source file to migrate fromdestPath
- the path to destination file for migrationdestSuperBlockLength
- the destination super block lengthIOException
- If any I/O error occurspublic abstract boolean validate() throws IOException, InstantiationException, IllegalAccessException
IOException
- If any I/O error occursInstantiationException
IllegalAccessException
public static Path validPath(Algorithm algo, Path path)
algo
- the checksum algorithm typepath
- the input path to be convertedpublic static void verifyOnlyValidFileExists(Algorithm algo, Path path) throws FileNotFoundException
algo
- the input algorithmpath
- the path to fileorg.apache.kafka.common.errors.MultiChecksumTypeException
- if there exist files in multiple checksum algorithm formatsorg.apache.kafka.common.errors.OtherChecksumTypeException
- if there exists a file in a different checksum algorithm formatFileNotFoundException
- if no file exists supporting any checksum algorithm formatpublic static boolean isValidPath(Algorithm algo, Path path)
algo
- the checksum algorithm to verify withpath
- the input path to verifypublic Optional<Long> checksum()
public static CheckedFileIO openOrCreate(Path path, Algorithm algo, short superBlockLength, boolean skipMultiChecksumValidation, OpenOption... options) throws IOException
path
- The path of the file to open or createalgo
- The checksum algorithmsuperBlockLength
- The super block lengthskipMultiChecksumValidation
- Boolean flag denoting to skip validation to check for multiple format filesoptions
- Options specifying how the file is openedIOException
- If any I/O error occurspublic static CheckedFileIO openOrCreate(Path path, Algorithm algo, short superBlockLength, OpenOption... options) throws IOException
IOException
public static CheckedFileIO open(Path path, boolean skipMultiChecksumValidation, OpenOption... options) throws IOException
path
- The path of the file to open or createskipMultiChecksumValidation
- Boolean flag denoting to skip validation to check for multiple format filesoptions
- Options specifying how the file is openedIOException
- If any I/O error occurspublic static CheckedFileIO open(Path path, OpenOption... options) throws IOException
IOException
public static void create(Path path, Algorithm algo, short superBlockLength) throws IOException
path
- The path of the file to createalgo
- The checksum algorithmsuperBlockLength
- The super block lengthIOException
- If any I/O error occurspublic void transferTo(long position, long count, CheckedFileIO target) throws IOException
position
- The position from which transfer is to begincount
- The maximum number of bytes to be transferredtarget
- The target channelIOException
- If any I/O error occurspublic void transferFrom(CheckedFileIO source, long position, long count) throws IOException
source
- The source channelposition
- The position in source file channel from which transfer is to begincount
- The maximum number of bytes to be transferredIOException
- If any I/O error occurspublic abstract void read(ByteBuffer dst, long position) throws IOException
dst
- The byte buffer to which bytes are to be transferredposition
- The file position at which the transfer is to beginIOException
- If any I/O error occurspublic abstract void write(ByteBuffer src) throws IOException
src
- The buffer from which bytes are to be transferredIOException
- If any I/O error occurspublic abstract void write(ByteBuffer src, long position) throws IOException
src
- The buffer from which bytes are to be transferredposition
- The file position at which the transfer is to beginIOException
- If any I/O error occurspublic abstract long position() throws IOException
IOException
- If any I/O error occurspublic abstract void position(long newPosition) throws IOException
newPosition
- The new positionIOException
- If any I/O error occurspublic abstract long size() throws IOException
IOException
- If any I/O error occurspublic abstract void truncate(long size) throws IOException
size
- The new size of file channelIOException
- If any I/O error occurspublic abstract void flush() throws IOException
IOException
- If any I/O error occurspublic abstract void close() throws IOException
close
in interface AutoCloseable
IOException
- If any I/O error occurs