A data science team deployed a gradient-boosted model to detect fraudulent credit card transactions. The model, trained on historical data from the previous year, achieved a 95% F1-score during validation. After six months in production, monitoring systems indicate a drop in the F1-score to 78%, accompanied by a significant increase in false negatives. Analysis of the live inference data reveals that the statistical distribution of features like 'transaction amount' and 'time of day' has shifted compared to the original training dataset. However, the fundamental patterns defining a fraudulent transaction are believed to be unchanged.
Which of the following best identifies the primary cause of the model's performance degradation and the most appropriate initial action?
The model is experiencing concept drift. The team should perform extensive hyperparameter tuning on the existing model architecture to adapt to the new fraud patterns.
The model is experiencing data drift. The most appropriate initial action is to retrain the model using a more recent dataset that includes the last six months of production data.
The performance drop is likely due to multicollinearity in the new data. The team should focus on advanced feature engineering to create new, uncorrelated variables.
The original model was overfitted to the training data. The best course of action is to simplify the model by reducing its complexity and then redeploying.
The correct answer identifies the issue as data drift and the solution as retraining with recent data. The scenario explicitly states that the input data's statistical distribution has changed, while the underlying relationship between inputs and the fraudulent outcome has not. This is the definition of data drift (also known as covariate shift). The most direct and standard initial approach to correct for data drift is to retrain the model on data that reflects the new distribution, which in this case would be the most recent production data.
Concept drift is incorrect because the scenario states that the fundamental patterns of fraud are unchanged, meaning the 'concept' the model is trying to predict has not changed.
Overfitting is incorrect because the model performed well initially after deployment, which would not be the case if it had failed to generalize from the start. The degradation occurred over time.
Multicollinearity is a problem with the relationships between predictor variables and is typically addressed during model development. It is not the most likely cause for a gradual performance drop over time as new data distributions emerge.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is data drift in machine learning?
Open an interactive chat with Bash
How does data drift differ from concept drift?
Open an interactive chat with Bash
Why is retraining a model important when dealing with data drift?