Unfortunately, I fear I'll always be cheap regarding model size. Instinctively I still am "*millions of parameters?!?!*"
2
0
17
0
0
@francoisfleuret For humongous data, you need something that can absorb a lot of info. For now, this is parameters. Otherwise, it's our current best way of making optimization easier. Hopefully we'll find better ways eventually.
@giffmana @francoisfleuret Parameter-efficient learning is often taken for granted. I wish we go beyond the "fitting" paradigm and learn more with less.
@ahatamiz1 @francoisfleuret You may like our distillation paper then, which does exactly that: scholar.google.ch/scholar?q=dist…