If you don't know at least one of: tiling, row/col major, or cache-lines and why they are relevant to deep learning, highly recommend reading this. In general I recommend following Horace! twitter.com/cHHillee/statu…

@cHHillee @ID_AA_Carmack +1 very well chosen example. I think matmul is more commonly used because cs.utexas.edu/~flame/pubs/Go… but your A+B.T keeps the essence while removing the extras, I really like it and will definitely steal it in the future :)

@ShayneRedford @YiTayML I see, now it makes more sense, my instant reaction was "wut", as I just had my vision dataset work in mind, which takes me at the very least an hour per dataset!

@YiTayML @ShayneRedford How on earth do you create 1800+ tasks or unify this many datasets to a common format?

@ducha_aiki Lol what a twist

@francoisfleuret @hos31n It seems to be mostly those that are present many times (duplicates) or outliers, both of which we know from recent papers that (not generative) deep models so memorize. While this result was not obvious a priori, it's also not hugely surprising, but super important nonetheless!

@wightmanr @rasbt @PyTorch Especially that cifar is not indicative of what's good for "regular resolution" images at all.

@ylecun @francoisfleuret This is not even an arbitrary exemple: when regressing angles and putting mod 360, something like this realistically happens.

Before you go work with some professor, ask yourself: do they just promote themself? or do they also work for their students? Looking at someone's website is a pretty good tell.

@david_picard I think it's less about native or not, and more that some people just can't help being verbose and others are naturally concise, in their mothertongue too.

@david_picard @ChrisGr93091552 The second, I've seen it a lot lately.

@peratham I like this one more because unlike convnext, it also mentions all the things that did *not* work, which is very valuable info. (I also like convnext)

@karpathy I like this paper because we had a similar experience between BiT and ViT, trying A LOT of ResNet "improvements" and none held water (except SqeezeEx). Now a similar thing seems to be happening with ViT sadly.

@RHebbalaguppe @neu_rips Categorical, not ordinal.

@cthorrez @theshawwn @francoisfleuret Separate

@theshawwn @francoisfleuret mobilenet, effnet, convnext, nfnet, etc

@y0b1byte Wow yes. And consistently 3 per year. I'm very impressed.

@cHHillee @theshawwn @francoisfleuret Yeah that should be exposed as low level thing then (à la jax.lax) and not guide high level API design imo.

@m__dehghani @neu_rips Yeah I did come up with a few heuristics (models) before asking, but though "this sounds like it must already exist". Seems like nope so far though.

@advadnoun @zacharynado Don't worry, that's why I shitpost without developing agi in bigco, will even thing out

@theshawwn @francoisfleuret That's the thing, having one somehow "freezes" this exact combination or at least conv-normalizer-activation and disincentives trying other stuff. At least that's what I noticed happens.

@theshawwn @francoisfleuret I'm instantly put off when a library or framework considers the non linearity to be part (say an option) of another layer

@francoisfleuret XLA has exactly one such mega-general conv op! jax.readthedocs.io/en/latest/_aut…

@yuricampbll @neu_rips Yep that's one of the heuristics someone suggested and give reasonably useful results for me

@y0b1byte Wow this is labour of love

@DigThatData @roydanroy @neu_rips Joan had a slightly better version of such heuristic, on top of which I built my current best heuristic: twitter.com/giffmana/statu… I can do heuristics but was really curious if something "proper" or established already exists.

@advadnoun phew dodged a bullet by putting in a bit of effort :D

@DigThatData @roydanroy @neu_rips Yup, that's a good approach, but linear wouldn't work, it's a set-based input and definitely a non-linear function that I want. Overall, making this almost a whole project on its own - can't afford that time for what I'm doing with it, and will prefer a closed-form heuristic.

@advadnoun I am part of the problem 🫥

@fhuszar @neu_rips This sounded promising when I first read it, but required some fiddling and is meh. Also it is not something that's TTTTTFFFFF as I initially thought, see example prints.