A company stores its application logs in an Amazon CloudWatch Logs log group. A new policy requires the company to store all application logs in Amazon OpenSearch Service (Amazon Elasticsearch Service) in near-real time.
Which solution will meet this requirement with the LEAST operational overhead?
Configuring a CloudWatch Logs subscription to stream the logs to Amazon OpenSearch Service provides a direct and efficient way to meet the requirement with minimal operational overhead. This solution enables near-real-time streaming of logs without the need for additional services or complex configurations, ensuring simplicity and reliability in the log transfer process.
answer is A https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html > You can configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in NEAR REAL-TIME through a CloudWatch Logs subscription least overhead compared to kinesis
Great link. Convinced me
Option A (Configure a CloudWatch Logs subscription to stream the logs to Amazon OpenSearch Service (Amazon Elasticsearch Service)) is not a suitable option, as a CloudWatch Logs subscription is designed to send log events to a destination such as an Amazon Simple Notification Service (Amazon SNS) topic or an AWS Lambda function. It is not designed to write logs directly to Amazon Elasticsearch Service (Amazon ES).
that is not true, you can stream logs from CloudWatch Logs directly to OpenSearch
Zerotn3 is right! There should be a Lambda for writing into ES
that is not true, you can stream logs from CloudWatch Logs directly to OpenSearch
good enough for me
Thank you for the link - clear answer
The correct answer is C: Create an Amazon Kinesis Data Firehose delivery stream. Configure the log group as the delivery stream source. Configure Amazon OpenSearch Service (Amazon Elasticsearch Service) as the delivery stream's destination. This solution uses Amazon Kinesis Data Firehose, which is a fully managed service for streaming data to Amazon OpenSearch Service (Amazon Elasticsearch Service) and other destinations. You can configure the log group as the source of the delivery stream and Amazon OpenSearch Service as the destination. This solution requires minimal operational overhead, as Kinesis Data Firehose automatically scales and handles data delivery, transformation, and indexing.
Option A: Configure a CloudWatch Logs subscription to stream the logs to Amazon OpenSearch Service (Amazon Elasticsearch Service) would also work, but it may require more operational overhead as you would need to set up and manage the subscription and ensure that the logs are delivered in near-real time. Option B: Create an AWS Lambda function. Use the log group to invoke the function to write the logs to Amazon OpenSearch Service (Amazon Elasticsearch Service) would also work, but it may require more operational overhead as you would need to set up and manage the Lambda function and ensure that it scales to handle the incoming logs. Option D: Install and configure Amazon Kinesis Agent on each application server to deliver the logs to Amazon Kinesis Data Streams. Configure Kinesis Data Streams to deliver the logs to Amazon OpenSearch Service (Amazon Elasticsearch Service) would also work, but it may require more operational overhead as you would need to install and configure the Kinesis Agent on each application server and set up and manage the Kinesis Data Streams.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
ANSWER A https://docs.aws.amazon.com/opensearch-service/latest/developerguide/integrations.html You can use CloudWatch or Kinesis, but in the Kinesis description it never says real time, however in the Cloudwatch description it does say Real time ""You can load streaming data from CloudWatch Logs to your OpenSearch Service domain by using a CloudWatch Logs subscription . For information about Amazon CloudWatch subscriptions, see Real-time processing of log data with subscriptions.""
Answer is A CloudWatch has a native feature to stream logs to OpenSearch, when you enable this setting it creates a Lambda Function automatically with pre-populated code which streams the logs to OpenSearch Cluster. The question here needs a solution with LEAST operational overhead, therefore the answer should be A REF: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html You'll need to have destination arn (not mentioned under option A) - either Lambda or Kinesis Firehose. The Amazon Resource Name (ARN) of the Kinesis stream, Kinesis Data Firehose stream, or Lambda function you want to use as the destination of the subscription feed. Option B) does not mention the Subscription Filter. Looks more towards Option C)
Option A (Configure a CloudWatch Logs subscription to stream the logs to Amazon OpenSearch Service (Amazon Elasticsearch Service)) is not a suitable option, as a CloudWatch Logs subscription is designed to send log events to a destination such as an Amazon Simple Notification Service (Amazon SNS) topic or an AWS Lambda function. It is not designed to write logs directly to Amazon Elasticsearch Service (Amazon ES).
You're totally right
B seems to be the right answer https://computingforgeeks.com/stream-logs-in-aws-from-cloudwatch-to-elasticsearch/
Please tell me why not C? https://docs.aws.amazon.com/opensearch-service/latest/developerguide/integrations.html#integrations-fh
LEAST operational overhead
The answer is C. The " in near-real time" makes it more accurate and least operational overhead.
A. Configure a CloudWatch Logs subscription to stream the logs to Amazon OpenSearch Service (Amazon Elasticsearch Service). This solution meets the requirement of storing all application logs in Amazon OpenSearch Service (Amazon Elasticsearch Service) with the least operational overhead. A CloudWatch Logs subscription allows you to automatically stream logs from CloudWatch Logs to a destination such as Elasticsearch Service, Kinesis Data Streams, or Lambda without the need for additional configurations and management. It eliminates the need for additional infrastructure, Lambda functions and configurations, or separate agents to handle the logs transfer to Elasticsearch Service.
The correct answer remains A. Kindly check the link for a confirmation. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
Must be C, https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html "You can configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in near real-time through a CloudWatch Logs subscription. For more information, see Real-time processing of log data with subscriptions.". https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html "You can use subscriptions to get access to a real-time feed of log events from CloudWatch Logs and have it delivered to other services such as an Amazon Kinesis stream, an Amazon Kinesis Data Firehose stream, or AWS Lambda for custom processing, analysis, or loading to other systems." CloudWatch cannot stream directly to Amazon OpenSearch Service.
The link above supports answer A not C, there is no mention of Kinesis
By configuring a CloudWatch Logs subscription, you can stream the logs from CloudWatch Logs to Amazon OpenSearch Service in near-real-time. This solution requires minimal operational overhead as it leverages the built-in functionality of CloudWatch Logs and Amazon OpenSearch Service for log streaming and indexing. Option B (Creating an AWS Lambda function) would involve additional development effort and maintenance of a custom Lambda function to write the logs to Amazon OpenSearch Service. Option C (Creating an Amazon Kinesis Data Firehose delivery stream) introduces an additional service (Kinesis Data Firehose) that may not be necessary for this specific requirement, adding unnecessary complexity. Option D (Installing and configuring Amazon Kinesis Agent) also introduces additional overhead in terms of manual installation and configuration on each application server, which may not be needed if the logs are already stored in CloudWatch Logs. In summary, option A is the correct choice as it provides a straightforward and efficient way to stream logs from CloudWatch Logs to Amazon OpenSearch Service with minimal operational overhead.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html
Near Real Time: Cloud watch logs --> Subscription Filter --> Kinesis data fire house --> S3 Real Time: Cloud watch logs --> Subscription Filter -->Lmabda --> S3
Since the scenario perfectly fits this description: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
I think C is correct because the cloud watch subscription can't stream directly to OpenSearch, it is via Lambda, SNS, FireHouse,....
Should be A
B looks good: https://d1.awsstatic.com/whitepapers/whitepaper-use-amazon-elasticsearch-to-log-and-monitor-almost-everything.pdf Chapter: Pushing Amazon CloudWatch Logs into Amazon ES: "... integration makes it easy to send data to Elasticsearch if source data exists in CloudWatch Logs" Approach with Amazon Kinesis Data Firehose requires installation of Amazon Kinesis agent on the EC2 instances, hence it's not considered as LEAST operational complex
Did you mean A?
They mentioned near real time
A is also near real time. plus A is least operational overhead
You can configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in near real-time through a CloudWatch Logs subscription.
Option A has least amount of changes needed to achieve this. But D is also possible would be better long term solution as it will avoid the duplication of the logs going into Cloudwatch and then moving to opensearch.
Ans c is correct note :- Kinesis Data Firehose (Near real-time (buffer time min. 60 sec))
choose C after seeing all comments from community
C for me and ChatGPT
A is wrong because subscriptions cannot be sent directly to Opensearch, see 'destination arn' in https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html Correct answer is C
We need to consider the “least operation overhead” and with that said Cloudwatch log Group and opersearch is already existing in the system and needs integration. Kinesics is preferable for near real time streaming but it will be additional overhead..Hence answer should be A
You can configure a CloudWatch Logs log group to stream data it receives to Amazon OpenSearch Service cluster in near real-time through a CloudWatch Logs subscription. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
"A new policy requires the company to store all application logs in Amazon OpenSearch Service (Amazon Elasticsearch Service) in !!!near-real time!!!!." Amazon Kinesis Data Firehose captures and loads data in near real time. It loads new data into Amazon S3, Amazon Redshift, and Amazon OpenSearch Service within 60 seconds after the data is sent to the service. As a result, you can access new data sooner and react to business and operational events faster.
Following these key words: - near-real time. - LEAST operational overhead and the fact that CloudWatch loggroup support OpenSearch Service subscription filter
Arratum: Obviously option A according to my wording
Amazon Kinesis Data Firehose can automatically deliver logs from CloudWatch Logs to Amazon OpenSearch Service without requiring you to manage and configure additional components or write custom code. It simplifies the process and reduces operational overhead
Correct Answer: C. Create an Amazon Kinesis Data Firehose delivery stream. Configure the log group as the delivery stream's source. Configure Amazon OpenSearch Service (Amazon Elasticsearch Service) as the delivery stream's destination. Explanation: Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon OpenSearch Service. It requires minimal setup and management, making it a low-overhead solution. By configuring the log group as the source for the Kinesis Data Firehose delivery stream and Amazon OpenSearch Service as the destination, logs can be delivered in near-real time with built-in reliability and scalability.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
C for sure Simplicity: Kinesis Data Firehose is a managed service that handles the task of capturing, transforming, and loading data into destinations like Amazon OpenSearch Service. This eliminates the need for complex configuration and management. Scalability: Kinesis Data Firehose can automatically scale to handle varying data volumes, ensuring that logs are ingested in near-real time. Cost-effectiveness: Kinesis Data Firehose is a pay-as-you-go service, making it a cost-effective option for log ingestion and analysis.
CloudWatch Logs subscription filter: This is the most straightforward way to stream logs from a CloudWatch Logs group to Amazon OpenSearch Service (Amazon Elasticsearch Service) in near real-time. It eliminates the need for additional components or complex configurations, reducing operational overhead. Direct integration: CloudWatch Logs can directly stream logs to OpenSearch Service without requiring intermediate services, making it a simple and efficient solution. Low operational overhead: Once set up, the subscription filter automatically forwards logs to OpenSearch Service with minimal maintenance.
LEAST Operational Overhead "https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html" Answer: A
No doubt C will work, but seems A is cheaper
Answer : A Based on Keywords and Documentation : A is the Answer You can configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in "near real-time through a CloudWatch Logs subscription"
But CloudWatch Logs log group does NOT support store(write) performance. It just stream data to Amazon OpenSearch Service.
Option C (Create an Amazon Kinesis Data Firehose delivery stream. Configure the log group as the delivery stream's sources. Configure Amazon OpenSearch Service (Amazon Elasticsearch Service) as the delivery stream's destination) would be the best option as it allows to easily and securely stream logs from CloudWatch Logs to Amazon Elasticsearch Service in near-real time with minimal operational overhead. Data Firehose is designed specifically for data stream processing and can automatically handle tasks such as data transformation, data validation, and data loading, simplifying the process of sending logs to Amazon Elasticsearch Service.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
This should be C. OpenSearch is one of the main destinations for Kinesis Data Firehose.
@six _fingers is right!!!! You can configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in near real-time through a CloudWatch Logs subscription. answer is A https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
I vote for C. Solution A add unnecessary hop
OPTION C You can use subscriptions to get access to a real-time feed of log events from CloudWatch Logs and have it delivered to other services such as an Amazon Kinesis stream, an Amazon Kinesis Data Firehose stream, or AWS Lambda for custom processing, analysis, or loading to other systems. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html
Answer is A . The question says near real time and not real time You can also use a CloudWatch Logs subscription to stream log data in near real time to an Amazon OpenSearch Service cluster. For more information, see Streaming CloudWatch Logs data to Amazon OpenSearch Service.
C is the correct answer. Using Kinesis Data Firehose will allow near real-time delivery of the CloudWatch logs to Amazon Elasticsearch Service with the least operational overhead compared to the other options. Firehose can be configured to automatically ingest data from CloudWatch Logs into Elasticsearch without needing to run Lambda functions or install agents on the application servers. This makes it the most operationally simple way to meet the stated requirements.
The answer is C
100% C. CloudWatch logs cannot be send to OpenSearch directly, need KDS or KDF works in the middle. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html
Create an AWS Lambda function. Use the log group to invoke the function to write the logs to Amazon Open Search Service (Amazon Elasticsearch Service) would also work, but it may require more operational overhead as you would need to set up and manage the Lambda function and ensure that it scales to handle the incoming logs. Option D: Install and configure Amazon Kinesis Agent on each application server to deliver the logs to Amazon Kinesis Data Streams. Configure Kinesis Data Streams to deliver the logs to Amazon Open Search Service (Amazon Elasticsearch Service) would also work, but it may require more operational overhead as you would need to install and configure the https://2048-cupcakes.org/
You need kinesis as a buffer in between, otherwise, the logs will be lost if anything goes wrong.
It is possible to configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in near realtime through a CloudWatch Logs subscription which implies les ops overhead.
You need real time buffer like Kinesis, otherwise you are going to lose data.
Pretty sure A supports near-real-time transfer
They both do, but C has the least operational overhead.
They both do, but C has the least operational overhead.
A, C can both support near-real-time logs transfer to OpenSearch. But it depends on the current needs. Based on the context of question, Option A is the best one. For Option C: This Kinesis Data Firehose offers additional benefits like easy scaling, built-in failure handling, and potential for data transformation if needed. But these are not required by the question. It only requires LEAST overhead-operation and near-real-time transfer then A is straightforward.
The solution that will meet the requirement with the least operational overhead is: **Option A**: Configure a CloudWatch Logs subscription to stream the logs to Amazon OpenSearch Service (Amazon Elasticsearch Service). This option allows you to directly stream logs from CloudWatch to OpenSearch Service (Elasticsearch Service) in near-real time without the need for additional services or resources, thus minimizing operational overhead. The other options involve additional services (Lambda, Kinesis Data Firehose, Kinesis Data Streams) and would therefore require more operational management.
A, You can configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in near real-time through a CloudWatch Logs subscription
You can configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in near real-time through a CloudWatch Logs subscription. This is the solution that requires the least operational overhead. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
A is correct You can configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in near real-time through a CloudWatch Logs subscription. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
Answer A. This doc clarifies the subject: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
You can configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in near real-time through a CloudWatch Logs subscription. here is the link/; https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html
It can natively connect to CloudWatch Logs as a source and OpenSearch Service as a destination, handling the delivery of logs efficiently and with minimal setup. This approach offers the least operational overhead by simplifying the data transfer pipeline with automatic scaling and error handling.
A for sure
easy enough to figure out. Option A
A is the correct answer, CloudWatch offers a subscription where you can stream data to other AWS services
Configure a CloudWatch Logs log group to stream data directly to the Amazon OpenSearch Service cluster. This can be done through a CloudWatch Logs subscription, which allows for real-time processing of log data.
You can configure a log group in Amazon CloudWatch Logs, so you can stream data to your Amazon OpenSearch Service cluster in near real-time.
In summary, the CloudWatch Logs → Kinesis Data Firehose → Amazon OpenSearch Service (option C) integration is the path recommended by AWS for this type of case. It allows for near real-time transmission, automatic scaling and relatively simple configuration, with the lowest operational overhead .