My prototype hit 100% accuracy on a test set of 500 images and I still don't trust it

Turns out the training data had a hidden label pattern from the lighting setup in my garage, so the model was just memorizing shadows instead of the actual objects.

2 comments

2 Comments

claire_ramirez2218d ago

Garage lighting tax. Classic. Bet the shadows looked real crisp in those 500 photos though. You basically built a fancy shadow detector that also happened to see objects sometimes. That's a whole new kind of overfitting. At least now you know your model has a strong future in forensic lighting analysis. Silver linings.

-1

the_sarah18d ago

Wait, has anyone actually considered what kind of lighting you'd need to make a model that's genuinely robust across all these garage situations? I mean, I've been down this rabbit hole with my own setup and the real kicker is everyone's garage has slightly different bulb temperatures, fixture heights, and reflective surfaces. So those 500 photos probably taught your model some wild biases about specific shadow angles that'll never show up in the real world. You could totally flip this into a data augmentation problem though, like train on synthetic shadows overlaid on clean object images. That might actually give you a detector that works when somebody parks a car in front of a window or turns on a single bulb in a dark corner. Maybe the next step isn't more photos, but literally simulating every possible light source placement you can think of.