MLS-C01 Exam QuestionsBrowse all questions from this exam

MLS-C01 Exam - Question 173


A retail company uses a machine learning (ML) model for daily sales forecasting. The company's brand manager reports that the model has provided inaccurate results for the past 3 weeks.

At the end of each day, an AWS Glue job consolidates the input data that is used for the forecasting with the actual daily sales data and the predictions of the model. The AWS Glue job stores the data in Amazon S3. The company's ML team is using an Amazon SageMaker Studio notebook to gain an understanding about the source of the model's inaccuracies.

What should the ML team do on the SageMaker Studio notebook to visualize the model's degradation MOST accurately?

Show Answer
Correct Answer: D

To understand the model's degradation most accurately, it is important to visualize the relationship between the actual daily sales and the model errors over time. A scatter plot of daily sales versus model error for the last 3 weeks, along with a scatter plot from before that period, allows the ML team to see how the model's performance correlates with the actual sales data both recently and historically. This comparison can highlight changes in the pattern of errors, offering insights into whether the model's inaccuracies are associated with specific sales ranges or temporal shifts.

Discussion

16 comments
Sign in to comment
exam_prep
May 25, 2022

C is the correct answer.

weixing
Jun 6, 2022

B, I guess

wolfsong
Feb 16, 2023

No, it's neither A or B as they use a histogram which would plot the distribution of errors. It will not tell you anything about how the model degrades over time, as the histogram will have no time component. You need a line chart for this. So it's C.

Jerry84Option: C
Jan 16, 2023

C is correct. We could view the "Degradation" as a trend. Line charts are usually very helpful to show if there is any trend in the data over the period of time under analysis. Histogram is normally used to visualizing distributions in your data.

ValcilioOption: C
Mar 6, 2023

C is the answer, line plots are good solutions for time series analysis.

Mickey321Option: B
Jul 31, 2023

option B with histograms of model errors for the specific time periods is the most accurate and appropriate visualization to understand the model's degradation and identify the reasons behind the inaccuracies in the daily sales forecasting.

DimLam
Oct 31, 2023

Is there a reason to create weekly MAE plot, if the prediction is made on daily granularity?

kyuhuckOption: B
Feb 8, 2024

The best option to visualize the model's degradation most accurately would be to compare the model's errors over the relevant periods. This directly addresses the issue of model accuracy and allows for a clear comparison of model performance before and after the reported period of inaccuracy. Therefore, the most appropriate approach would be: B. Create a histogram of the model errors over the last 3 weeks. In addition, create a histogram of the model errors from before that period. This approach will allow the ML team to see if the distribution of errors has changed recently, indicating a degradation in model performance.

AIWaveOption: D
Feb 25, 2024

Degradation over time: line or scatter plot (options C, D) C is aggregrate weekly view and doesn't give any additional details. D compares the model's errors during 3 week period to the errors from before that period giving an accurate picture of anomalies

DD4
Aug 30, 2022

Should be A because it is daily forecasting and histograms before and after will show the comparable degradation.

venimus_vidimus_vicimus
Dec 12, 2022

It only states to plot daily sales.. how should that help with the error? You need a plot of the actual and predicted values or or the errors - def not A

kaike_reisOption: C
Aug 10, 2023

C is correct.

loictOption: C
Sep 8, 2023

A. NO - Daily sales histogram does not help to see model error B. NO - Histogram of the model errors is good, but no point to have one for the first 3 weeks and another for older data C. YES - one chart of model errors is perfect D. NO - no point to have 2 charts again

teka112233Option: C
Sep 12, 2023

this is the key sentence : At the end of each day, an AWS Glue job consolidates the input data that is used for the forecasting with the actual daily sales data and the predictions of the model and that is exactly what MAE do: mean absolute error (MAE) is a statistical measure of the difference between two continuous variables. It is calculated as the average of the absolute differences between the predicted and actual values so the answer is C

3eb0542Option: D
Feb 18, 2024

GPT: To accurately visualize the degradation of the model over time and understand the source of inaccuracies, the ML team should focus on comparing the model's performance before and after the reported period of inaccuracies. The most appropriate option is: D. Create a scatter plot of daily sales versus model error for the last 3 weeks. In addition, create a scatter plot of daily sales versus model error from before that period.

F1Fan
Mar 20, 2024

Claude 3 Sonnet: Based on the evidence from AWS documentation and best practices, Option B: Create a histogram of the model errors over the last 3 weeks. In addition, create a histogram of the model errors from before that period, is the most accurate approach for the ML team to visualize the model's degradation. Histograms of model errors directly visualize the distribution and patterns of the model's inaccuracies, which is crucial for understanding the source of the problem. By comparing the error distributions before and after the 3-week period, the ML team can identify any significant shifts or changes that may indicate the cause of the model's degradation. This approach aligns with AWS best practices for model monitoring and visualization, as recommended by the Amazon SageMaker Model Monitor documentation. It provides a clear and focused visualization of the model's performance, enabling the ML team to gain insights and take appropriate actions to address the inaccuracies.

3eb0542Option: D
Apr 22, 2024

Weekly MAE aggregates the error metrics over a larger time window, which can mask fluctuations and specific patterns in the model's performance on a daily basis. In situations where there are sudden changes or degradation in the model's accuracy within a week, this visualization might not capture those nuances effectively.

f3a4b7cOption: D
May 29, 2024

D is the correct answer

learningcloud1Option: B
Jul 2, 2024

Most accurately. C is a tempting answer. You will see the model degradation over time. You could see if it's slowly getting worse or was it sudden. B is more accurate. You will only have two histograms to compare, but you will easily see which direction the error move: Are we over or underestimating. In practice you would use both. I'm terms of the exam - most accurate - most added information, B gives more information than C. It's a preference though.

learningcloud1
Jul 3, 2024

Although, following the docs it will be C: https://docs.aws.amazon.com/forecast/latest/dg/predictor-monitoring-results.html