You recently joined a machine learning team that will soon release a new project. As a lead on the project, you are asked to determine the production readiness of the ML components. The team has already tested features and data, model development, and infrastructure. Which additional readiness check should you recommend to the team?
To ensure that a machine learning model is ready for production, it is critical to have mechanisms in place for monitoring its performance. This allows the team to detect and respond to any changes in model performance that may occur once the model is deployed. Having a monitoring system ensures that the model consistently performs well in the live environment, maintaining the reliability and accuracy of its predictions. This step is essential for maintaining quality and for making any necessary adjustments post-deployment.
I think it should be C
performance monitoring is a continuous effort that happens all time. but reproducibility makes more sense to be added to model QA
The question was not about model QA but production readiness, thus I think the answer is C because monitor model performance in production is important. As regard to A, I would I argue it could fall under "model development", since reproducible training is already important during model development.
To my understanding, I think A might be correct since model performance monitoring is happens "in production". but the question said the project "will soon release" which means right now is before launching, so to me testing the reproducible would make more sense. (I was confused about A and C for a long time) reference: - Testing reproducibility: https://developers.google.com/machine-learning/testing-debugging/pipeline/deploying - Testing in Production: https://developers.google.com/machine-learning/testing-debugging/pipeline/production
I also think is C: reference : https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/aad9f93b86b7addfea4c419b9100c6cdd26cacea.pdf
A - important one before moving to the production
Testing for Deploying Machine Learning Models: - Test Model Updates with Reproducible Training https://developers.google.com/machine-learning/testing-debugging/pipeline/deploying
Hey! all guys A+B+D=The team has already tested features and data, model development, and infrastructure. we are about to go live with production. Monitoring readiness is the last thing to account for. It will be very rediculous if you launch model as production regardless of how we will have about monitoring. you will lauch model as production for while and will make plan to model performance monitoring later ??? you are too reckless. Pls . Read it carefully https://developers.google.com/machine-learning/testing-debugging/pipeline/production https://developers.google.com/machine-learning/testing-debugging/pipeline/overview#what-is-an-ml-pipeline. You Most guys prefer A : https://developers.google.com/machine-learning/testing-debugging/pipeline/deploying I think that it is all about model development prior to deploying .
A makes more sense than C.
With the specific focus on "production readiness" as stated, I'd pick C above the others.
It's mentioned that the team has already tested features and data, implying that data generation is reproducible. If you have to test features data has to be reproducible to compare model outputs. ( https://developers.google.com/machine-learning/data-prep/construct/sampling-splitting/randomization). Hence C makes more sense
Monitoring is crucial. So - C
"production readiness" means that we are still in dev-test phase , and "performance monitoring" happens in production , and what if monitoring is applied but the model re-train is difficult , so "A" is the best answer
I think it's C. A is related to infrastructure, B is related to model development and D is related to Data and features. It clearly mentioned that team has already tested for model development, data and features and infrastructure.
https://cloud.google.com/ai-platform/docs/ml-solutions-overview
C, for me.
Reproducible Training is more likely to be in the Deployment step in that it referred to the question "The team has already tested features and data, model development" but the question focuses on Production readiness https://developers.google.com/machine-learning/testing-debugging/pipeline/production Monitor section is part of this above link
I think that your team ensure that all hypermarameters were turned yet when tested features... i think that it's more important that they ensure that model performance is monitored than thaining is reproducible for best practices. https://cloud.google.com/architecture/ml-on-gcp-best-practices
I'll go with C. Monitoring model performance is an important aspect of production readiness. It allows the team to detect and respond to changes in performance that may affect the quality of the model. The other options are also important, but they are more focused on the development phase of the project rather than the production phase.
Went with C
C is not a readiness check. Monitoring is a continuous effort. IMO A is the correct answer. If the training is not reproducible it's not ready for production. If any error happens, data drifts / skews, then there is no way to recreate the model. This is a check BEFORE going to production. Once it's in production, then yes C is important.
C) Model monitoring