A manufacturing company wants to collect data from sensors. A data engineer needs to implement a solution that ingests sensor data in near real time.
The solution must store the data to a persistent data store. The solution must store the data in nested JSON format. The company must have the ability to query from the data store with a latency of less than 10 milliseconds.
Which solution will meet these requirements with the LEAST operational overhead?
To meet the requirements of near real-time data ingestion, persistent storage, nested JSON format, and low-latency queries with the least operational overhead, using Amazon Kinesis Data Streams to capture the sensor data and storing it in Amazon DynamoDB for querying is the most suitable option. Amazon Kinesis Data Streams is a fully managed service that supports real-time data ingestion, and Amazon DynamoDB provides single-digit millisecond latency for queries, supports nested JSON, and is also fully managed. This minimizes operational overhead compared to alternatives that require more complex setups or self-management.
Option C is the best solution to meet the requirements with the least operational overhead: Use Amazon Kinesis Data Streams to ingest real-time sensor data Store the nested JSON data in Amazon DynamoDB for low latency queries The key advantages of Option C are: Kinesis Data Streams fully manages real-time data ingestion with auto-scaling and persistence DynamoDB provides single digit millisecond latency for queries DynamoDB natively supports nested JSON data models Fully managed services minimize operational overhead In contrast: Option A requires managing Kafka clusters Option B uses Lambda which can't provide persistent storage Option D requires integrating SQS, Glue, and RDS leading to complexity
Option C, using Amazon Kinesis Data Streams to capture the sensor data and storing it in Amazon DynamoDB for querying, is the best solution to meet the requirements with the least operational overhead. This solution is well-optimized for real-time data ingestion, supports the desired data format, and provides the necessary query performance.
Amazon Kinesis Data Streams is a fully managed service that allows for seamless integration of diverse data sources, including IoT sensors. By using Kinesis Data Streams as the ingestion mechanism, the company can avoid the overhead of setting up and managing an Apache Kafka cluster or other data ingestion pipelines.
near real time = Kinesis Data streams
to be more accurate, Kinesis Data streams = real time Kinesis Data Firehose = near real time
Option C is the best solution to meet the requirements