Amazon Kinesis Data Streams
Amazon Kinesis Data Streams (KDS) is a scalable, fully managed real-time data-streaming service from Amazon Web Services that captures gigabytes of data per second from sources such as website clickstreams, event streams, and social-media feeds, serving as an alternative to Apache Kafka. Producer applications write data records into shards, and custom consumer applications read from the stream, often forwarding results into Amazon Kinesis Data Analytics (now Amazon Managed Service for Apache Flink) or Amazon Kinesis Data Firehose (now Amazon Data Firehose).
Key points
- Part of the Kinesis family alongside Amazon Kinesis Data Analytics (renamed Amazon Managed Service for Apache Flink), Amazon Kinesis Data Firehose (renamed Amazon Data Firehose), and Kinesis Video Streams.
- Producers write data via the Kinesis Streams API, the Kinesis Producer Library (KPL), or the Kinesis Agent.
- Consumers are custom-built Kinesis Data Streams applications that read data from a stream as data records.
- Streams are divided into Shards (Partitions); a shard is the base throughput unit.
- Default data retention is 24 hours, extendable up to a maximum of 8,760 hours (365 days) via extended/long-term retention (older CLF-C02 material still cites a 7-day ceiling).
- Each shard provides 1 MB/second (1,000 records/second) of input capacity and 2 MB/second of output capacity via the standard
GetRecordsAPI; enhanced fan-out consumers get a dedicated 2 MB/second pipe per consumer per shard, with up to 20 registered consumers per stream. - Fully managed for real-time processing of streaming data at massive scale, with producer applications putting data into a stream as a series of data records (per AWS Terminology.xlsx).
- Typical flow: Producers → Kinesis Data Streams → Amazon Managed Service for Apache Flink → Amazon Data Firehose → destinations such as an S3 bucket or Amazon OpenSearch Service.
- Distinct from Amazon SQS: KDS is for continuous, ordered ingestion of streaming data by custom consumer applications, not point-to-point messaging between application components — for sending, storing, and receiving discrete messages, use SQS instead.
- On-Demand capacity mode requires no shard planning — it automatically scales to accommodate up to double the previous 30-day peak write throughput. Provisioned mode requires manually specifying and resizing the shard count; a stream can switch between the two modes twice within 24 hours.