AWS Glue
AWS Glue is a serverless ETL (extract, transform, and load) service from Amazon Web Services used to discover, categorize, transform, and move data between data stores and streams for analytics, machine learning, and other application development.
AI Practitioner focus
- Glue supports the data collection/preparation side of AI/ML: discover schemas with crawlers, store technical metadata in the Data Catalog, transform data with ETL/Visual ETL, and apply data-quality rules.
- The Data Catalog stores metadata rather than the underlying records. Use AWS Glue DataBrew for analyst-friendly visual cleaning and AWS Lake Formation for fine-grained data-lake governance.
Key points
- Serverless data integration service that prepares and combines data for analytics, machine learning, and application development.
- Central repository, the AWS Glue Data Catalog, automatically generates Python or Scala code.
- Processes semi-structured data using a “dynamic frame” in ETL scripts, similar to an Apache Spark data frame that organizes data into rows and columns.
- Dynamic frames and Spark data frames can be converted between each other so AWS Glue and Spark can be used together for analysis.
- Discovers data and stores its associated metadata in the AWS Glue Data Catalog while executing Apache Spark environment ETL jobs.
- Lets organizations work together on data integration tasks such as extraction, normalization, combining, loading, and running ETL workloads.
- Manages multiple types of data for customers and helps AWS services discover, transform, and visualize the data.
- Works with Amazon Redshift for data warehouses, Amazon S3 for data lakes, and Amazon RDS or Amazon EC2 instances for data stores.
- Typical ETL flow: S3 Bucket (Extract) → AWS Glue (Transform) → Amazon Redshift (Load); also Amazon RDS (Extract) → AWS Glue.
- Distinct from AWS Lake Formation, which is the higher-level service for standing up a whole secure data lake (identifying sources, crawling, cataloging, and applying permissions); Lake Formation is built on top of, and shares, the Glue Data Catalog.
- Not a query engine — for ad hoc SQL directly against S3 data, use Amazon Athena instead.