Package software.amazon.awssdk.services.sagemaker.model

Class S3DataSource

java.lang.Object
software.amazon.awssdk.services.sagemaker.model.S3DataSource
All Implemented Interfaces:
Serializable, SdkPojo, ToCopyableBuilder<S3DataSource.Builder,S3DataSource>
@Generated("software.amazon.awssdk:codegen") public final class S3DataSource extends Object implements SdkPojo, Serializable, ToCopyableBuilder<S3DataSource.Builder,S3DataSource>

Describes the S3 data source.

Your input bucket must be in the same Amazon Web Services region as your training job.

See Also:

  • Method Details

    • s3DataType

      public final S3DataType s3DataType()

      If you choose S3Prefix, S3Uri identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.

      If you choose ManifestFile, S3Uri identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.

      If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile can only be used if the Channel's input mode is Pipe.

      If the service returns an enum value that is not available in the current SDK version, s3DataType will return S3DataType.UNKNOWN_TO_SDK_VERSION. The raw value returned by the service is available from s3DataTypeAsString().

      Returns:
      If you choose S3Prefix, S3Uri identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.

      If you choose ManifestFile, S3Uri identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.

      If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile can only be used if the Channel's input mode is Pipe.

      See Also:

    • s3DataTypeAsString

      public final String s3DataTypeAsString()

      If you choose S3Prefix, S3Uri identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.

      If you choose ManifestFile, S3Uri identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.

      If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile can only be used if the Channel's input mode is Pipe.

      If the service returns an enum value that is not available in the current SDK version, s3DataType will return S3DataType.UNKNOWN_TO_SDK_VERSION. The raw value returned by the service is available from s3DataTypeAsString().

      Returns:
      If you choose S3Prefix, S3Uri identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.

      If you choose ManifestFile, S3Uri identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.

      If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile can only be used if the Channel's input mode is Pipe.

      See Also:

    • s3Uri

      public final String s3Uri()

      Depending on the value specified for the S3DataType, identifies either a key name prefix or a manifest. For example:

      • A key name prefix might look like this: s3://bucketname/exampleprefix/

      • A manifest might look like this: s3://bucketname/example.manifest

        A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of S3Uri. Note that the prefix must be a valid non-empty S3Uri that precludes users from specifying a manifest whose individual S3Uri is sourced from different S3 buckets.

        The following code example shows a valid manifest format:

        [ {"prefix": "s3://customer_bucket/some/prefix/"},

        "relative/path/to/custdata-1",

        "relative/path/custdata-2",

        ...

        "relative/path/custdata-N"

        ]

        This JSON is equivalent to the following S3Uri list:

        s3://customer_bucket/some/prefix/relative/path/to/custdata-1

        s3://customer_bucket/some/prefix/relative/path/custdata-2

        ...

        s3://customer_bucket/some/prefix/relative/path/custdata-N

        The complete set of S3Uri in this manifest is the input data for the channel for this data source. The object that each S3Uri points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.

      Your input bucket must be located in same Amazon Web Services region as your training job.

      Returns:
      Depending on the value specified for the S3DataType, identifies either a key name prefix or a manifest. For example:

      • A key name prefix might look like this: s3://bucketname/exampleprefix/

      • A manifest might look like this: s3://bucketname/example.manifest

        A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of S3Uri. Note that the prefix must be a valid non-empty S3Uri that precludes users from specifying a manifest whose individual S3Uri is sourced from different S3 buckets.

        The following code example shows a valid manifest format:

        [ {"prefix": "s3://customer_bucket/some/prefix/"},

        "relative/path/to/custdata-1",

        "relative/path/custdata-2",

        ...

        "relative/path/custdata-N"

        ]

        This JSON is equivalent to the following S3Uri list:

        s3://customer_bucket/some/prefix/relative/path/to/custdata-1

        s3://customer_bucket/some/prefix/relative/path/custdata-2

        ...

        s3://customer_bucket/some/prefix/relative/path/custdata-N

        The complete set of S3Uri in this manifest is the input data for the channel for this data source. The object that each S3Uri points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.

      Your input bucket must be located in same Amazon Web Services region as your training job.

    • s3DataDistributionType

      public final S3DataDistribution s3DataDistributionType()

      If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated.

      If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.

      Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.

      In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode is set to File), this copies 1/n of the number of objects.

      If the service returns an enum value that is not available in the current SDK version, s3DataDistributionType will return S3DataDistribution.UNKNOWN_TO_SDK_VERSION. The raw value returned by the service is available from s3DataDistributionTypeAsString().

      Returns:
      If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated.

      If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.

      Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.

      In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode is set to File), this copies 1/n of the number of objects.

      See Also:

    • s3DataDistributionTypeAsString

      public final String s3DataDistributionTypeAsString()

      If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated.

      If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.

      Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.

      In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode is set to File), this copies 1/n of the number of objects.

      If the service returns an enum value that is not available in the current SDK version, s3DataDistributionType will return S3DataDistribution.UNKNOWN_TO_SDK_VERSION. The raw value returned by the service is available from s3DataDistributionTypeAsString().

      Returns:
      If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated.

      If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.

      Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.

      In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode is set to File), this copies 1/n of the number of objects.

      See Also:

    • hasAttributeNames

      public final boolean hasAttributeNames()
      For responses, this returns true if the service returned a value for the AttributeNames property. This DOES NOT check that the value is non-empty (for which, you should check the isEmpty() method on the property). This is useful because the SDK will never return a null collection or map, but you may need to differentiate between the service returning nothing (or null) and the service returning an empty collection or map. For requests, this returns true if a value for the property was specified in the request builder, and false if a value was not specified.

    • attributeNames

      public final List<String> attributeNames()

      A list of one or more attribute names to use that are found in a specified augmented manifest file.

      Attempts to modify the collection returned by this method will result in an UnsupportedOperationException.

      This method will never return null. If you would like to know whether the service returned this field (so that you can differentiate between null and empty), you can use the hasAttributeNames() method.

      Returns:
      A list of one or more attribute names to use that are found in a specified augmented manifest file.

    • hasInstanceGroupNames

      public final boolean hasInstanceGroupNames()
      For responses, this returns true if the service returned a value for the InstanceGroupNames property. This DOES NOT check that the value is non-empty (for which, you should check the isEmpty() method on the property). This is useful because the SDK will never return a null collection or map, but you may need to differentiate between the service returning nothing (or null) and the service returning an empty collection or map. For requests, this returns true if a value for the property was specified in the request builder, and false if a value was not specified.

    • instanceGroupNames

      public final List<String> instanceGroupNames()

      A list of names of instance groups that get data from the S3 data source.

      Attempts to modify the collection returned by this method will result in an UnsupportedOperationException.

      This method will never return null. If you would like to know whether the service returned this field (so that you can differentiate between null and empty), you can use the hasInstanceGroupNames() method.

      Returns:
      A list of names of instance groups that get data from the S3 data source.

    • toBuilder

      public S3DataSource.Builder toBuilder()
      Description copied from interface: ToCopyableBuilder
      Take this object and create a builder that contains all of the current property values of this object.
      Specified by:
      toBuilder in interface ToCopyableBuilder<S3DataSource.Builder,S3DataSource>
      Returns:
      a builder for type T

    • builder

      public static S3DataSource.Builder builder()

    • serializableBuilderClass

      public static Class<? extends S3DataSource.Builder> serializableBuilderClass()

    • hashCode

      public final int hashCode()
      Overrides:
      hashCode in class Object

    • equals

      public final boolean equals(Object obj)
      Overrides:
      equals in class Object

    • equalsBySdkFields

      public final boolean equalsBySdkFields(Object obj)
      Description copied from interface: SdkPojo
      Indicates whether some other object is "equal to" this one by SDK fields. An SDK field is a modeled, non-inherited field in an SdkPojo class, and is generated based on a service model.

      If an SdkPojo class does not have any inherited fields, equalsBySdkFields and equals are essentially the same.

      Specified by:
      equalsBySdkFields in interface SdkPojo
      Parameters:
      obj - the object to be compared with
      Returns:
      true if the other object equals to this object by sdk fields, false otherwise.

    • toString

      public final String toString()
      Returns a string representation of this object. This is useful for testing and debugging. Sensitive data will be redacted from this string using a placeholder value.
      Overrides:
      toString in class Object

    • getValueForField

      public final <T> Optional<T> getValueForField(String fieldName, Class<T> clazz)

    • sdkFields

      public final List<SdkField<?>> sdkFields()
      Specified by:
      sdkFields in interface SdkPojo
      Returns:
      List of SdkField in this POJO. May be empty list but should never be null.