A lab uses IoT sensors to monitor humidity, temperature, and pressure for a project. The sensors send 100 KB of data every 10 seconds. A downstream process will read the data from an Amazon S3 bucket every 30 seconds.
Which solution will deliver the data to the S3 bucket with the LEAST latency?
To achieve the least latency when delivering the data to the S3 bucket, using Amazon Kinesis Data Streams coupled with the Kinesis Client Library (KCL) and a minimal buffer interval is the most efficient approach. The Kinesis Data Streams captures the data continuously, while the KCL enables timely processing and flexible delivery configurations. By setting a 5-second buffer interval from an application, it ensures that data is delivered to S3 with minimal delay. This method avoids the added complexity and potential latency overhead introduced by other services such as Apache Flink.
Kinesis Data Streams cannot deliver directly to S3. Data has to go through Firehose. A is correct but is not lowest latency. I would go with D, as we can set the buffer interval to a low value. We do not need Flink, tho. That's a bit confusing.
C - This option ensures low latency by using a short buffer interval (5 seconds). The use of KCL allows for customized processing logic and timely delivery of data to S3. This makes it a strong candidate for minimal latency. D - While this option provides low latency with a 5-second buffer interval, it introduces unnecessary complexity by using Apache Flink for what seems to be a straightforward data ingestion task. This option is overkill for the given use case and may add more operational overhead than necessary.
I think the answer is C. Kinesis Data Firehose has a minimum buffer interval of 60 seconds (1 minute) or 1 MB of data.
Fyi, Firehose now supports 0 buffering: https://aws.amazon.com/about-aws/whats-new/2023/12/amazon-kinesis-data-firehose-zero-buffering/
As per option A, "Use the default buffer interval for Kinesis Data Firehose". Default buffer interval for Kinesis Data Firehose is 300seconds where S3 is the destination. Flink is not required here. Hence, option D is not suitable.
https://docs.aws.amazon.com/firehose/latest/dev/buffering-hints.html
https://aws.amazon.com/about-aws/whats-new/2023/12/amazon-kinesis-data-firehose-zero-buffering/
its C - option D uses 1/ Analytics which summarizes data and gence has delay then passses to 2/ Firehose for deliver and Firehose doesnt say its using zero buffering
Firehose uses multi-part upload for S3 destination when you configure a buffer time interval less than 60 seconds to offer lower latencies. Due to multi-part upload for S3 destination, you will see some increase in S3 PUT API costs if you choose a buffer time interval less than 60 seconds.