Amazon Managed Streaming for Apache Kafka (MSK)
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a managed cluster service from Amazon Web Services for building and running Apache Kafka applications that process streaming data, feeding machine learning applications, analytical applications, data lakes, and database change streams via Apache Kafka APIs.
Key points
- Removes the manual tasks otherwise needed to configure Apache Kafka applications.
- Manages replacing servers during failures, handling server patches and upgrades with no downtime, maintenance of Apache Kafka clusters, maintenance of Apache ZooKeeper, multi-AZ replication for Apache Kafka clusters, and planning scaling events.
- Provides multiple kinds of security for Apache Kafka clusters, including AWS Identity and Access Management (IAM) for API authorization, encryption at rest, and Apache Kafka Access Control Lists (ACLs).
- Integrates with AWS Glue to run an Apache Spark job on the MSK cluster, Amazon Kinesis Data Analytics to run an Apache Flink job on the MSK cluster, and AWS Lambda functions.
- Not serverless by default — you provision and size the broker instances yourself (Amazon MSK Serverless is a separate, serverless variant that scales automatically).
- MSK Connect runs fully managed Apache Kafka Connect workloads — compatible with connectors built for Apache Kafka Connect 2.7.1 and above — and auto-scales workers to move data between MSK clusters and external systems without provisioning infrastructure.
- MSK Express brokers provide up to 3x the throughput per broker, scale up to 20x faster, recover 90% quicker, and support up to 5x more partitions per broker than standard MSK brokers.