Just realized my old mentor was right about layer priorities after a convo with an intern

Kid fresh out of school said he was taught to stack layers by complexity instead of sequence, and it made me question everything I learned from Jim back in 2015. Has anyone else had to unlearn something from an old timer that actually turned out to be outdated?

2 comments

2 Comments

amygonzalez18d ago

Honestly, that complexity versus sequence thing is wild to think about. It makes total sense when you break it down though, because back in 2015 we were all working with pretty simple networks so sequencing was fine. Now with all this deep learning stuff going on, stacking by complexity actually helps the model handle nonlinear patterns way better because each layer builds on different features. Jim probably had good reasons for teaching it that way based on what he knew then, the industry just moved fast. Tbh I bet if you run both methods on a modern dataset you'd see a real difference in loss curves.

tylerw7218d ago

The loss curves comparison is a good point, I bet you'd see a noticeable gap with bigger datasets especially. It really makes me wonder how many other "rules" from back then are holding people back now without us even realizing it.