Spent 6 weeks debugging a model drift issue that was a data pipeline typo
Back in March, I was working on a customer churn prediction model for our retail clients. The model kept losing accuracy every week, and I assumed it was concept drift or changing customer behavior. I spent six weeks retraining, tuning hyperparameters, and even swapping algorithms. Finally, a coworker pointed out that a date field in our ETL job was hardcoded to start from February instead of March, so the training data was missing a whole month. Fixing that one line cut my error rate in half overnight. I felt equal parts relieved and stupid. Has anyone else chased a model performance problem for weeks only to find it was a data issue upstream, not the model itself?