Package org.apache.flink.fs.s3hadoop

Class HadoopS3AccessHelper

java.lang.Object

org.apache.flink.fs.s3hadoop.HadoopS3AccessHelper

All Implemented Interfaces:

org.apache.flink.fs.s3.common.writer.S3AccessHelper

public class HadoopS3AccessHelper
extends Object
implements org.apache.flink.fs.s3.common.writer.S3AccessHelper

An implementation of the S3AccessHelper for the Hadoop S3A filesystem.

This implementation uses Hadoop's WriteOperationHelper to perform S3 operations, similar to the SDK v1 implementation. This provides retry logic, error handling, S3A statistics integration, and auditing support.

Constructor Summary

Constructors

Constructor	Description
HadoopS3AccessHelper(org.apache.hadoop.fs.s3a.S3AFileSystem s3a, org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method	Description
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse	commitMultiPartUpload(String destKey, String uploadId, List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags, long length, AtomicInteger errorCount)
boolean	deleteObject(String key)
long	getObject(String key, File targetLocation)
software.amazon.awssdk.services.s3.model.HeadObjectResponse	getObjectMetadata(String key)
software.amazon.awssdk.services.s3.model.PutObjectResponse	putObject(String key, File inputFile)
String	startMultiPartUpload(String key)
software.amazon.awssdk.services.s3.model.UploadPartResponse	uploadPart(String key, String uploadId, int partNumber, File inputFile, long length)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopS3AccessHelper

public HadoopS3AccessHelper(org.apache.hadoop.fs.s3a.S3AFileSystem s3a,
                            org.apache.hadoop.conf.Configuration conf)

Method Detail

startMultiPartUpload

public String startMultiPartUpload(String key)
                            throws IOException

Specified by:

startMultiPartUpload in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Throws:

IOException

uploadPart

public software.amazon.awssdk.services.s3.model.UploadPartResponse uploadPart(String key,
                                                                              String uploadId,
                                                                              int partNumber,
                                                                              File inputFile,
                                                                              long length)
                                                                       throws IOException

Specified by:

uploadPart in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Throws:

IOException

putObject

public software.amazon.awssdk.services.s3.model.PutObjectResponse putObject(String key,
                                                                            File inputFile)
                                                                     throws IOException

Specified by:

putObject in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Throws:

IOException

commitMultiPartUpload

public software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse commitMultiPartUpload(String destKey,
                                                                                                      String uploadId,
                                                                                                      List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags,
                                                                                                      long length,
                                                                                                      AtomicInteger errorCount)
                                                                                               throws IOException

Specified by:

commitMultiPartUpload in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Throws:

IOException

deleteObject

public boolean deleteObject(String key)
                     throws IOException

Specified by:

deleteObject in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Throws:

IOException

getObject

public long getObject(String key,
                      File targetLocation)
               throws IOException

Specified by:

getObject in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Throws:

IOException

getObjectMetadata

public software.amazon.awssdk.services.s3.model.HeadObjectResponse getObjectMetadata(String key)
                                                                              throws IOException

Specified by:

getObjectMetadata in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Throws:

IOException