Professional Machine Learning Engineer Exam - Question 141

Question

You work for a magazine publisher and have been tasked with predicting whether customers will cancel their annual subscription. In your exploratory data analysis, you find that 90% of individuals renew their subscription every year, and only 10% of individuals cancel their subscription. After training a NN Classifier, your model predicts those who cancel their subscription with 99% accuracy and predicts those who renew their subscription with 82% accuracy. How should you interpret these results?

Examice · Accepted Answer

The model's performance should be assessed against a simple baseline, which in this case, would be predicting that everyone renews their subscription. Given that 90% of individuals renew, this naive prediction would result in a 90% accuracy rate. The model’s accuracy of 82% for renewals is actually lower than this baseline, indicating that the model is not performing well overall. Despite the high accuracy of 99% in predicting cancellations, this subgroup only makes up 10% of the population, making it less impactful. Hence, the model is performing worse than simply predicting renewal for everyone.

TNT87 · Answer

Answer C

Voyager2 · Answer

Went with C: This is a good result because predicting those who cancel their subscription is more difficult, since there is less data for this group
 My Reason: "You have been tasked with predicting whether customers will cancel their annual subscription." And in that task you are getting 99% of accuracy

Scipione_ · Answer

I think C is the only way

ares81 · Answer

Logically, it should be C.

John_Pongthorn · Answer

We can consider it as follows reasonably.
A: it doesn't make any sense,  given that cancel=99% but renew =82%, how did you make renew class (82%) beat the Cancel class(99%), it must be 100% accuracy ( bullshit)
B:  Cancel class have more accuracy than renew (99%>82%)
D: You can justify, both are good 80% if we have a balance class.

So it left us with C.   This model predicts well upon imbalanced class circumstances.
target class =10 samles meanwhile  the another =90 samples

M25 · Answer

#ResponsibleAI, predicting the majority class (imbalanced data) topic: “the model [82% accuracy for renew] is performing worse than predicting that people will always [90% accuracy] renew their subscription”.
https://developers.google.com/machine-learning/crash-course/classification/check-your-understanding-accuracy-precision-recall
“A deadly, but curable, medical condition afflicts .01% of the population. An ML model (…) predicts (…) with an accuracy of 99.99%. (…) After all, even a "dumb" model that always predicts "not sick" would still be 99.99% accurate.“

ShePiDai · Answer

Task is to predict whether customer will cancel subscription, so both renew and cancel predictions are important. The overall accuracy is 99% x 10% + 82% x 90% = 83%, while guessing always renew has 90% accuracy.

tavva_prudhvi · Answer

since there is less data for this group. While the accuracy for predicting subscription renewals is lower, it is still above chance and may still be useful. Additionally, the high accuracy for predicting cancellations is promising, as this is the group of interest for the publisher. However, it would still be important to assess the model's precision and recall to fully evaluate its performance.

pico · Answer

Here's the reasoning:

The overall renewal rate is 90%, meaning that if the model simply predicted that everyone would renew, it would have an accuracy of 90%. The model's accuracy for predicting renewals (82%) is lower than this baseline accuracy.
The model's accuracy for predicting cancellations is high (99%), but this could be misleading. If only 10% of individuals cancel their subscription, a model that predicts no cancellations at all would still have a high accuracy of 90%. Therefore, the high accuracy for cancellations may not be very informative.
In summary, the model is not performing well, especially when compared to a simple baseline of always predicting renewals.

Dataspire · Answer

Since 90% of dataset represent customer who will renew subscription, accuracy should have been greater than 82%

lucaluca1982 · Answer

The 82% accuracy for renewals is lower than a naive model that always predicts renewals (which would have a 90% accuracy).

Professional Machine Learning Engineer Exam - Question 141

Discussion