Amazon Kinesis Data Firehose
Amazon Data Firehose (renamed from Amazon Kinesis Data Firehose in February 2024) is a fully managed, serverless service from Amazon Web Services that captures, transforms, and reliably delivers real-time streaming data to destinations such as Amazon S3, Amazon Redshift, or Amazon OpenSearch Service (per AWS Terminology.xlsx). The rename changed only the name shown in the console, documentation, and service pages — endpoints, APIs, the AWS CLI, IAM policies, and CloudWatch metrics were unaffected.
Key points
- Synchronously replicates data across three Availability Zones while delivering it to destinations.
- Enables real-time analysis with existing business intelligence tools, and can transform, batch, compress, and encrypt data before delivery.
- Each delivery stream keeps data records for up to 24 hours if the destination is unavailable and the source is Direct PUT (retention follows the source stream’s configuration when Kinesis Data Streams is the source).
- Buffer size hints range from 1 MB to 128 MB (default 5 MB) and buffer interval hints range from 0 to 900 seconds (default 300 seconds). A zero buffering option, added December 2023, lets streams to S3, OpenSearch Service, Redshift, and HTTP endpoints deliver in as little as 5 seconds — the older 60-second minimum latency no longer applies (older CLF-C02 material still cites a 60-second/32 MB minimum).
- Sources include Amazon Kinesis Data Streams, Amazon MSK, and Amazon CloudWatch Logs/Events, delivered via SDK, Direct PUT, or the Kinesis Producer Library (KPL).
- Delivers streaming data to Amazon S3, Amazon Redshift, Amazon OpenSearch Service (including OpenSearch Serverless), Splunk, Snowflake, Apache Iceberg tables, and generic HTTP endpoints.