Amazon EMR
Amazon EMR (Elastic MapReduce) is a managed big-data platform from Amazon Web Services that processes and analyzes large volumes of data using frameworks such as Apache Hadoop, Apache Hive, Apache Spark, and Apache Flink across dynamically expandable Amazon EC2 instance clusters. It decouples compute from storage by keeping cluster data on Amazon S3, and integrates with Amazon DynamoDB, Amazon Redshift, and Amazon Kinesis Data Streams.
AI Practitioner focus
- Use EMR for distributed preparation and analysis of very large datasets with frameworks such as Spark; it can feed an AI/ML pipeline but is not itself a managed model-training API.
- Choose EMR Serverless when the processing job should scale without cluster management; choose an EMR cluster or EMR on EKS when framework/runtime control matters.
Key points
- Managed Hadoop framework enabling simple, quick, cost-effective, data-intensive processing across dynamically expandable Amazon EC2 instances and S3 (per AWS Terminology.xlsx).
- The cluster is the main component, made up of Amazon EC2 instances known as nodes.
- Decouples the compute and storage layers, scaling independently, with cluster data stored on Amazon S3.
- Controls network access for instances by configuring instance firewall settings.
- Provides basic cluster maintenance: monitoring, replacing failed instances, and bug fixes.
- Analyzes machine learning workloads with Apache Spark MLlib and TensorFlow, clickstream workloads with Apache Spark and Apache Hive, and real-time streaming workloads from Amazon Kinesis using Apache Flink.
- Executes on Amazon EC2, Amazon EKS (EMR on EKS), AWS Outposts, or EMR Serverless.
- Accessible via the EMR Console, the AWS Command Line Interface (AWS CLI), an SDK, or the Web Service API.
- Integrates with DynamoDB (EMR-DynamoDB Connector), Amazon Redshift (copy from HDFS), Amazon OpenSearch Service (OpenSearch Connector), Amazon Kinesis (Streaming Data Connector), RDS (JDBC data with Spark SQL), and S3 (EMR File System / EMRFS).
- EMR Serverless runs Spark, Hive, and other supported jobs without provisioning or managing clusters: it automatically determines, acquires, and scales the workers a job needs, then releases them when the job finishes.
- EMR on EKS runs Spark jobs on an existing Amazon EKS cluster, letting teams standardize on Kubernetes to manage clusters across applications or run multiple framework versions side by side — distinct from EMR Serverless, which avoids cluster management entirely.