Professional Machine Learning Engineer Exam - Question 159

Question

Your organization manages an online message board. A few months ago, you discovered an increase in toxic language and bullying on the message board. You deployed an automated text classifier that flags certain comments as toxic or harmful. Now some users are reporting that benign comments referencing their religion are being misclassified as abusive. Upon further inspection, you find that your classifier's false positive rate is higher for comments that reference certain underrepresented religious groups. Your team has a limited budget and is already overextended. What should you do?

Examice · Accepted Answer

To address the issue of the classifier misclassifying benign comments referencing certain underrepresented religious groups as toxic, raising the threshold for comments to be considered toxic or harmful is a practical and immediate solution. By increasing the threshold, the classifier will require a higher level of toxicity to flag a comment as harmful, thereby reducing false positives. While other options like adding synthetic training data or replacing the model could theoretically improve the classifier's accuracy, they would require significant time and resources — something the team does not have. Human moderation, though accurate, isn't feasible with the limited budget. Raising the threshold specifically tackles the problem at hand with minimal additional workload, making it the most viable option under the given constraints.

[Removed] · Answer

By raising the threshold for comments to be considered toxic or harmful, you will decrease the number of false positives.

B is wrong because we are taking a Google MLE exam :) A and C are wrong because both of them involve a good amount of additional work, either for extending the dataset or training/experimenting with a new model. Considering your team is already over the budget and has too many tasks on their plate (overextended), these two options are not available for you.

powerby35 · Answer

"Your team has a limited budget and is already overextended"

daidai75 · Answer

Your team has a limited budget and is already overextended, that means the re-training is hardly possible.

pinimichele01 · Answer

Your team has a limited budget and is already overextended

PST21 · Answer

A. Add synthetic training data where those phrases are used in non-toxic ways.

In this situation, where your automated text classifier is misclassifying benign comments referencing certain underrepresented religious groups as toxic or harmful, adding synthetic training data where those phrases are used in non-toxic ways can be a cost-effective solution to improve the model's performance.

tavva_prudhvi · Answer

In the long run, usually we go with A, but Option D could be a temporary solution to reduce false positives, while being aware that it may allow some genuinely toxic comments to go unnoticed. However, this may be a necessary trade-off until your team has the resources to improve the classifier or find a better solution.

edoo · Answer

B and C are non sense, I don't want to risk potentially increasing the FNR by reducing the FPR (Raise the threshold). Thus A.

7cb0ab3 · Answer

I went fo A because it directly tackels the issue of misclassification and improving the models unterstanding of religious references. B and C don't make sense.
D would generally reduce the number of comments flagged as toxic, which could decrease the false positive rate. However, this approach risks allowing genuinely harmful comments to go unflagged. It addresses the symptom (high false positive rate) rather than the underlying cause

gscharly · Answer

agree with daidai75

Simple_shreedhar · Answer

A option directly addresses the bias issue without incurring significant ongoing costs or burdening the moderation team. By augmenting the training dataset with synthetic examples where phrases related to underrepresented religious groups are used in non-toxic ways, the classifier can learn to distinguish between toxic and benign comments more accurately.

AzureDP900 · Answer

option A (Add synthetic training data where those phrases are used in non-toxic ways) directly addresses the specific issue of bias and improves the model's accuracy by providing more contextually relevant training examples. This approach is more targeted and has a lower risk of introducing new biases or negatively impacting other aspects of comment moderation.

I hope this additional explanation helps clarify why option D might not be the best choice in this scenario!

Professional Machine Learning Engineer Exam - Question 159

Discussion