AWS Batch
AWS Batch is a fully managed, regional Amazon Web Services batch processing service that lets developers, scientists, and engineers execute large amounts of batch computing workloads efficiently, without managing the underlying cluster.
Key points
- Provisions the right amount of memory and compute to efficiently run large-scale batch computing workloads across managed compute environments built on:
- Amazon EC2 — On-Demand or Spot Instances.
- AWS Fargate — including Fargate Spot.
- Amazon EKS — for jobs running on an existing EKS cluster.
- Jobs are submitted to a job queue and scheduled onto a managed computing environment.
- A job is a unit of work such as a shell script, a Linux executable, or a Docker container image.
- Multi-node parallel jobs, which span multiple EC2 instances for tightly coupled HPC or distributed GPU training workloads, require EC2 On-Demand compute environments — they aren’t supported on Spot Instances or on Fargate.
- Supports queuing Amazon SageMaker Training jobs with configurable priorities, so ML training jobs run automatically as capacity becomes available.
- Can integrate with data stores such as Amazon S3 or Amazon DynamoDB to securely retrieve and write data.