Interface S3TransferManager

All Superinterfaces:
AutoCloseable, SdkAutoCloseable
@ThreadSafe public interface S3TransferManager extends SdkAutoCloseable
The S3 Transfer Manager offers a simple API that allows you to transfer a single object or a set of objects to and from Amazon S3 with enhanced throughput and reliability. It leverages Amazon S3 multipart upload and byte-range fetches to perform transfers in parallel. In addition, the S3 Transfer Manager also enables you to monitor a transfer's progress in real-time, as well as pause the transfer for execution at a later time.

Instantiate the S3 Transfer Manager

Create a transfer manager instance with SDK default settings. Note that it's recommended to add software.amazon.awssdk.crt:aws-crt dependency so that automatic multipart feature can be enabled 
     S3TransferManager transferManager = S3TransferManager.create();
Create an S3 Transfer Manager instance with custom settings 
        S3AsyncClient s3AsyncClient = S3AsyncClient.crtBuilder()
                                                   .credentialsProvider(DefaultCredentialsProvider.create())
                                                   .region(Region.US_WEST_2)
                                                   .targetThroughputInGbps(20.0)
                                                   .minimumPartSizeInBytes(8 * MB)
                                                   .build();

        S3TransferManager transferManager =
            S3TransferManager.builder()
                             .s3Client(s3AsyncClient)
                             .build();
Create an S3 Transfer Manager with S3 Multipart Async Client. The S3 Multipart Async Client is an alternative to the CRT client that offers the same multipart upload/download feature. 
     S3AsyncClient s3AsyncClient = s3AsyncClient.builder()
                                                .multipartEnabled(true)
                                                .multipartConfiguration(conf -> conf.apiCallBufferSizeInBytes(32 * MB))
                                                .build();

     S3TransferManager transferManager =
         S3TransferManager.builder()
                          .s3Client(s3AsyncClient)
                          .build();

Common Usage Patterns

Upload a file to S3 
        S3TransferManager transferManager = S3TransferManager.create();

        UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
                                                               .putObjectRequest(req -> req.bucket("bucket").key("key"))
                                                               .addTransferListener(LoggingTransferListener.create())
                                                               .source(Paths.get("myFile.txt"))
                                                               .build();

        FileUpload upload = transferManager.uploadFile(uploadFileRequest);
        upload.completionFuture().join();
Download an S3 object to a local file 
        S3TransferManager transferManager = S3TransferManager.create();

        DownloadFileRequest downloadFileRequest =
               DownloadFileRequest.builder()
                                  .getObjectRequest(req -> req.bucket("bucket").key("key"))
                                                              .destination(Paths.get("myFile.txt"))
                                                              .addTransferListener(LoggingTransferListener.create())
                                                              .build();

        FileDownload download = transferManager.downloadFile(downloadFileRequest);

        // Wait for the transfer to complete
        download.completionFuture().join();
Upload a local directory to an S3 bucket 
        S3TransferManager transferManager = S3TransferManager.create();
        DirectoryUpload directoryUpload =
            transferManager.uploadDirectory(UploadDirectoryRequest.builder()
                                                                  .source(Paths.get("source/directory"))
                                                                  .bucket("bucket")
                                                                  .s3Prefix("prefix")
                                                                  .build());

        // Wait for the transfer to complete
        CompletedDirectoryUpload completedDirectoryUpload = directoryUpload.completionFuture().join();

        // Print out any failed uploads
        completedDirectoryUpload.failedTransfers().forEach(System.out::println);
Download S3 objects to a local directory 
      S3TransferManager transferManager = S3TransferManager.create();
      DirectoryDownload directoryDownload =
            transferManager.downloadDirectory(
                 DownloadDirectoryRequest.builder()
                                         .destination(Paths.get("destination/directory"))
                                         .bucket("bucket")
                                         .listObjectsV2RequestTransformer(l -> l.prefix("prefix"))
                                         .build());
        // Wait for the transfer to complete
        CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();

        // Print out any failed downloads
        completedDirectoryDownload.failedTransfers().forEach(System.out::println);
Copy an S3 object to a different location in S3 
        S3TransferManager transferManager = S3TransferManager.create();
        CopyObjectRequest copyObjectRequest = CopyObjectRequest.builder()
                                                               .sourceBucket("source_bucket")
                                                               .sourceKey("source_key")
                                                               .destinationBucket("dest_bucket")
                                                               .destinationKey("dest_key")
                                                               .build();
        CopyRequest copyRequest = CopyRequest.builder()
                                             .copyObjectRequest(copyObjectRequest)
                                             .build();

        Copy copy = transferManager.copy(copyRequest);
        // Wait for the transfer to complete
        CompletedCopy completedCopy = copy.completionFuture().join();
The automatic parallel transfer feature (multipart upload/download) is available through the AWS-CRT based S3 client S3AsyncClient.crtBuilder().build) and Java-based S3 multipart client S3AsyncClient.builder().multipartEnabled(true).build(). If no client is configured, AWS-CRT based S3 client will be used if AWS CRT is in the classpath, otherwise, Java-based S3 multipart client will be used.

  • Method Details

    • downloadFile

      default FileDownload downloadFile(DownloadFileRequest downloadRequest)
      Downloads an object identified by the bucket and key from S3 to a local file. For non-file-based downloads, you may use download(DownloadRequest) instead.

      The SDK will create a new file if the provided one doesn't exist. The default permission for the new file depends on the file system and platform. Users can configure the permission on the file using Java API by themselves. If the file already exists, the SDK will replace it. In the event of an error, the SDK will NOT attempt to delete the file, leaving it as-is.

      Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners.

      Usage Example: 

              S3TransferManager transferManager = S3TransferManager.create();

        DownloadFileRequest downloadFileRequest =
               DownloadFileRequest.builder()
                                  .getObjectRequest(req -> req.bucket("bucket").key("key"))
                                                              .destination(Paths.get("myFile.txt"))
                                                              .addTransferListener(LoggingTransferListener.create())
                                                              .build();

        FileDownload download = transferManager.downloadFile(downloadFileRequest);

        // Wait for the transfer to complete
        download.completionFuture().join();
      See Also:

    • downloadFile

      default FileDownload downloadFile(Consumer<DownloadFileRequest.Builder> request)
      This is a convenience method that creates an instance of the DownloadFileRequest builder, avoiding the need to create one manually via DownloadFileRequest.builder().
      See Also:

    • resumeDownloadFile

      default FileDownload resumeDownloadFile(ResumableFileDownload resumableFileDownload)
      Resumes a downloadFile operation. This download operation uses the same configuration as the original download. Any content that has already been fetched since the last pause will be skipped and only the remaining data will be downloaded from Amazon S3.

      If it is determined that the source S3 object or the destination file has be modified since the last pause, the SDK will download the object from the beginning as if it is a new DownloadFileRequest.

      Usage Example: 

              S3TransferManager transferManager = S3TransferManager.create();

        DownloadFileRequest downloadFileRequest = DownloadFileRequest.builder()
                                                                     .getObjectRequest(req -> req.bucket("bucket").key
                                                                     ("key"))
                                                                     .destination(Paths.get("myFile.txt"))
                                                                     .build();

        // Initiate the transfer
        FileDownload download =
            transferManager.downloadFile(downloadFileRequest);

        // Pause the download
        ResumableFileDownload resumableFileDownload = download.pause();

        // Optionally, persist the download object
        Path path = Paths.get("resumableFileDownload.json");
        resumableFileDownload.serializeToFile(path);

        // Retrieve the resumableFileDownload from the file
        resumableFileDownload = ResumableFileDownload.fromFile(path);

        // Resume the download
        FileDownload resumedDownload = transferManager.resumeDownloadFile(resumableFileDownload);

        // Wait for the transfer to complete
        resumedDownload.completionFuture().join();
      Parameters:
      resumableFileDownload - the download to resume.
      Returns:
      A new FileDownload object to use to check the state of the download.
      See Also:

    • resumeDownloadFile

      default FileDownload resumeDownloadFile(Consumer<ResumableFileDownload.Builder> resumableFileDownload)
      This is a convenience method that creates an instance of the ResumableFileDownload builder, avoiding the need to create one manually via ResumableFileDownload.builder().
      See Also:

    • download

      default <ResultT> Download<ResultT> download(DownloadRequest<ResultT> downloadRequest)
      Downloads an object identified by the bucket and key from S3 through the given AsyncResponseTransformer. For downloading to a file, you may use downloadFile(DownloadFileRequest) instead.

      Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners.

      Usage Example (this example buffers the entire object in memory and is not suitable for large objects): 

              S3TransferManager transferManager = S3TransferManager.create();

        DownloadRequest<ResponseBytes<GetObjectResponse>> downloadRequest =
            DownloadRequest.builder()
                           .getObjectRequest(req -> req.bucket("bucket").key("key"))
                           .responseTransformer(AsyncResponseTransformer.toBytes())
                           .build();

        // Initiate the transfer
        Download<ResponseBytes<GetObjectResponse>> download =
            transferManager.download(downloadRequest);
        // Wait for the transfer to complete
        download.completionFuture().join();

      See the static factory methods available in AsyncResponseTransformer for other use cases.

      Type Parameters:
      ResultT - The type of data the AsyncResponseTransformer produces
      Parameters:
      downloadRequest - the download request, containing a GetObjectRequest and AsyncResponseTransformer
      Returns:
      A Download that can be used to track the ongoing transfer
      See Also:

    • uploadFile

      default FileUpload uploadFile(UploadFileRequest uploadFileRequest)
      Uploads a local file to an object in S3. For non-file-based uploads, you may use upload(UploadRequest) instead.

      Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners. Upload a local file to an object in S3. For non-file-based uploads, you may use upload(UploadRequest) instead.

      Usage Example: 

              S3TransferManager transferManager = S3TransferManager.create();

        UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
                                                               .putObjectRequest(req -> req.bucket("bucket").key("key"))
                                                               .addTransferListener(LoggingTransferListener.create())
                                                               .source(Paths.get("myFile.txt"))
                                                               .build();

        FileUpload upload = transferManager.uploadFile(uploadFileRequest);
        upload.completionFuture().join();
      See Also:

    • uploadFile

      default FileUpload uploadFile(Consumer<UploadFileRequest.Builder> request)
      This is a convenience method that creates an instance of the UploadFileRequest builder, avoiding the need to create one manually via UploadFileRequest.builder().
      See Also:

    • resumeUploadFile

      default FileUpload resumeUploadFile(ResumableFileUpload resumableFileUpload)
      Resumes uploadFile operation. This upload operation will use the same configuration provided in ResumableFileUpload. The SDK will skip the data that has already been upload since the last pause and only upload the remaining data from the source file.

      If it is determined that the source file has be modified since the last pause, the SDK will upload the object from the beginning as if it is a new UploadFileRequest.

      Usage Example: 

              S3TransferManager transferManager = S3TransferManager.create();

        UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
                                                               .putObjectRequest(req -> req.bucket("bucket").key("key"))
                                                               .source(Paths.get("myFile.txt"))
                                                               .build();

        // Initiate the transfer
        FileUpload upload =
            transferManager.uploadFile(uploadFileRequest);
        // Pause the upload
        ResumableFileUpload resumableFileUpload = upload.pause();

        // Optionally, persist the resumableFileUpload
        Path path = Paths.get("resumableFileUpload.json");
        resumableFileUpload.serializeToFile(path);

        // Retrieve the resumableFileUpload from the file
        ResumableFileUpload persistedResumableFileUpload = ResumableFileUpload.fromFile(path);

        // Resume the upload
        FileUpload resumedUpload = transferManager.resumeUploadFile(persistedResumableFileUpload);

        // Wait for the transfer to complete
        resumedUpload.completionFuture().join();
      Parameters:
      resumableFileUpload - the upload to resume.
      Returns:
      A new FileUpload object to use to check the state of the download.
      See Also:

    • resumeUploadFile

      default FileUpload resumeUploadFile(Consumer<ResumableFileUpload.Builder> resumableFileUpload)
      This is a convenience method that creates an instance of the ResumableFileUpload builder, avoiding the need to create one manually via ResumableFileUpload.builder().
      See Also:

    • upload

      default Upload upload(UploadRequest uploadRequest)
      Uploads the given AsyncRequestBody to an object in S3. For file-based uploads, you may use uploadFile(UploadFileRequest) instead.

      Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners.

      Usage Example: 

              S3TransferManager transferManager = S3TransferManager.create();

        UploadRequest uploadRequest = UploadRequest.builder()
                                                   .requestBody(AsyncRequestBody.fromString("Hello world"))
                                                   .putObjectRequest(req -> req.bucket("bucket").key("key"))
                                                   .build();

        Upload upload = transferManager.upload(uploadRequest);
        // Wait for the transfer to complete
        upload.completionFuture().join();
      See the static factory methods available in AsyncRequestBody for other use cases.
      Parameters:
      uploadRequest - the upload request, containing a PutObjectRequest and AsyncRequestBody
      Returns:
      An Upload that can be used to track the ongoing transfer
      See Also:

    • upload

      default Upload upload(Consumer<UploadRequest.Builder> request)
      This is a convenience method that creates an instance of the UploadRequest builder, avoiding the need to create one manually via UploadRequest.builder().
      See Also:

    • uploadDirectory

      default DirectoryUpload uploadDirectory(UploadDirectoryRequest uploadDirectoryRequest)
      Uploads all files under the given directory to the provided S3 bucket. The key name transformation depends on the optional prefix and delimiter provided in the UploadDirectoryRequest. By default, all subdirectories will be uploaded recursively, and symbolic links are not followed automatically. This behavior can be configured in at request level via UploadDirectoryRequest.Builder.followSymbolicLinks(Boolean) or client level via S3TransferManager.Builder.uploadDirectoryFollowSymbolicLinks(Boolean) Note that request-level configuration takes precedence over client-level configuration.

      By default, the prefix is an empty string and the delimiter is "/". Assume you have a local directory "/test" with the following structure: 

         
      |- test
         |- sample.jpg
         |- photos
             |- 2022
                 |- January
                     |- sample.jpg
                 |- February
                     |- sample1.jpg
                     |- sample2.jpg
                     |- sample3.jpg
      Give a request to upload directory "/test" to an S3 bucket, the target bucket will have the following S3 objects:
      • sample.jpg
      • photos/2022/January/sample.jpg
      • photos/2022/February/sample1.jpg
      • photos/2022/February/sample2.jpg
      • photos/2022/February/sample3.jpg

      The returned CompletableFuture only completes exceptionally if the request cannot be attempted as a whole (the source directory provided does not exist for example). The future completes successfully for partial successful requests, i.e., there might be failed uploads in the successfully completed response. As a result, you should check for errors in the response via CompletedDirectoryUpload.failedTransfers() even when the future completes successfully.

      The current user must have read access to all directories and files.

      Usage Example: 

              S3TransferManager transferManager = S3TransferManager.create();
        DirectoryUpload directoryUpload =
            transferManager.uploadDirectory(UploadDirectoryRequest.builder()
                                                                  .source(Paths.get("source/directory"))
                                                                  .bucket("bucket")
                                                                  .s3Prefix("prefix")
                                                                  .build());

        // Wait for the transfer to complete
        CompletedDirectoryUpload completedDirectoryUpload = directoryUpload.completionFuture().join();

        // Print out any failed uploads
        completedDirectoryUpload.failedTransfers().forEach(System.out::println);
      Parameters:
      uploadDirectoryRequest - the upload directory request
      See Also:

    • uploadDirectory

      default DirectoryUpload uploadDirectory(Consumer<UploadDirectoryRequest.Builder> requestBuilder)
      This is a convenience method that creates an instance of the UploadDirectoryRequest builder, avoiding the need to create one manually via UploadDirectoryRequest.builder().
      See Also:

    • downloadDirectory

      default DirectoryDownload downloadDirectory(DownloadDirectoryRequest downloadDirectoryRequest)
      Downloads all objects under a bucket to the provided directory. By default, all objects in the entire bucket will be downloaded. You can modify this behavior by providing a DownloadDirectoryRequest.listObjectsRequestTransformer() and/or a DownloadDirectoryRequest.filter() in DownloadDirectoryRequest to limit the S3 objects to download.

      The downloaded directory structure will match with the provided S3 virtual bucket. For example, assume that you have the following keys in your bucket:

      • sample.jpg
      • photos/2022/January/sample.jpg
      • photos/2022/February/sample1.jpg
      • photos/2022/February/sample2.jpg
      • photos/2022/February/sample3.jpg
      Give a request to download the bucket to a destination with path of "/test", the downloaded directory would look like this 
         
      |- test
         |- sample.jpg
         |- photos
             |- 2022
                 |- January
                     |- sample.jpg
                 |- February
                     |- sample1.jpg
                     |- sample2.jpg
                     |- sample3.jpg

      The returned CompletableFuture only completes exceptionally if the request cannot be attempted as a whole (the downloadDirectoryRequest is invalid for example). The future completes successfully for partial successful requests, i.e., there might be failed downloads in a successfully completed response. As a result, you should check for errors in the response via CompletedDirectoryDownload.failedTransfers() even when the future completes successfully.

      The SDK will create the destination directory if it does not already exist. If a specific file already exists, the existing content will be replaced with the corresponding S3 object content.

      The current user must have write access to all directories and files

      Usage Example: 

             S3TransferManager transferManager = S3TransferManager.create();
         DirectoryDownload directoryDownload =
            transferManager.downloadDirectory(
                 DownloadDirectoryRequest.builder()
                                         .destination(Paths.get("destination/directory"))
                                         .bucket("bucket")
                                          // only download objects with prefix "photos"
                                          .listObjectsV2RequestTransformer(l -> l.prefix("photos"))
                                         .build());
        // Wait for the transfer to complete
        CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();

        // Print out any failed downloads
        completedDirectoryDownload.failedTransfers().forEach(System.out::println);
      Parameters:
      downloadDirectoryRequest - the download directory request
      See Also:

    • downloadDirectory

      default DirectoryDownload downloadDirectory(Consumer<DownloadDirectoryRequest.Builder> requestBuilder)
      This is a convenience method that creates an instance of the DownloadDirectoryRequest builder, avoiding the need to create one manually via DownloadDirectoryRequest.builder().
      See Also:

    • copy

      default Copy copy(CopyRequest copyRequest)
      Creates a copy of an object that is already stored in S3.

      Depending on the underlying S3Client, S3TransferManager may intelligently use plain CopyObjectRequests for smaller objects, and multiple parallel UploadPartCopyRequests for larger objects. If multipart copy is supported by the underlying S3Client, this behavior can be configured via S3CrtAsyncClientBuilder.minimumPartSizeInBytes(Long). Note that for multipart copy request, existing metadata stored in the source object is NOT copied to the destination object; if required, you can retrieve the metadata from the source object and set it explicitly in the @link CopyObjectRequest.Builder#metadata(Map)}.

      While this API supports TransferListeners, they will not receive bytesTransferred callback-updates due to the way the CopyObjectRequest API behaves. When copying an object, S3 performs the byte copying on your behalf while keeping the connection alive. The progress of the copy is not known until it fully completes and S3 sends a response describing the outcome.

      If you are copying an object to a bucket in a different region, you need to enable cross region access on the S3AsyncClient.

      Usage Example: 

              S3AsyncClient s3AsyncClient =
           S3AsyncClient.crtBuilder()
                        // enable cross-region access, only required if you are making cross-region copy
                        .crossRegionAccessEnabled(true)
                        .build();
        S3TransferManager transferManager = S3TransferManager.builder()
                                                             .s3Client(s3AsyncClient)
                                                             .build();
        CopyObjectRequest copyObjectRequest = CopyObjectRequest.builder()
                                                               .sourceBucket("source_bucket")
                                                               .sourceKey("source_key")
                                                               .destinationBucket("dest_bucket")
                                                               .destinationKey("dest_key")
                                                               .build();
        CopyRequest copyRequest = CopyRequest.builder()
                                             .copyObjectRequest(copyObjectRequest)
                                             .build();

        Copy copy = transferManager.copy(copyRequest);
        // Wait for the transfer to complete
        CompletedCopy completedCopy = copy.completionFuture().join();
      Parameters:
      copyRequest - the copy request, containing a CopyObjectRequest
      Returns:
      A Copy that can be used to track the ongoing transfer
      See Also:

    • copy

      default Copy copy(Consumer<CopyRequest.Builder> copyRequestBuilder)
      This is a convenience method that creates an instance of the CopyRequest builder, avoiding the need to create one manually via CopyRequest.builder().
      See Also:

    • create

      static S3TransferManager create()
      Create an S3TransferManager using the default values.

      The type of S3AsyncClient used depends on if AWS Common Runtime (CRT) library software.amazon.awssdk.crt:aws-crt is in the classpath. If AWS CRT is available, an AWS CRT-based S3 client will be created via (S3AsyncClient.crtCreate()). Otherwise, a standard S3 client(S3AsyncClient.create()) will be created. Note that only AWS CRT-based S3 client supports parallel transfer, i.e, leveraging multipart upload/download for now, so it's recommended to add AWS CRT as a dependency.

    • builder

      static S3TransferManager.Builder builder()
      Creates a default builder for S3TransferManager.