I propose that we adopt the term "Large Self-Supervised Models (LSSMs)" as a replacement for "Foundation Models" and "LLMs". "LLMs" don't capture non-linguistic data and "Foundation Models" is too grandiose. Thoughts? @percyliang

@tdietterich The beauty of language is that you can have multiple terms that highlight different aspects of the same object. You don't have to choose. I use "LLM" to talk about LLMs, "self-supervised" for their construction, and "foundation model" for their function. No term can be replaced.

@percyliang Yes, but as you know, "Foundation" is too close to "Foundational", and many of us find that troubling. That is why I'm proposing a more neutral term. For use, maybe we could just call them "Upstream models".

@giffmana @francoisfleuret @tdietterich @percyliang Pretty sure it is quite important... We wouldn't be able to scale close to that scale with fully supervised models.

@giffmana @francoisfleuret @tdietterich @percyliang ­čĺ» Aren't AlexNet/ResNet FM? Their function wasn't limit to image classification. Also used in object detection,...,robotics? @percyliang and Co might establish a rigours comparison in their paper. Read it & let us know ­čśů. Feedback about the document can be shared in @BlindDou

@giffmana @tdietterich @percyliang A causal auto-regressive model certainly gets a lot of bits of supervision with one cross-entropy score per token, with zero ml-specific cost for labeling.

@giffmana @francoisfleuret In vision, there seems to be much less benefit in self-supervised pre-training compared to language.

@ggdupont @francoisfleuret @tdietterich @percyliang Funny you tell me that, because I have several papers doing exactly that...

@francoisfleuret @tdietterich @percyliang That's correct, but orthogonal to the discussed point.