@wightmanr The perfect batch size

@SamuelAinsworth @zacharynado @konstmish hahahahaha for real tho

@SamuelAinsworth @zacharynado @konstmish Yeah. Because in BN the examples in a minibatch are not independent anymore, so you can't interchange f([Xi]) = [f(Xi)] anymore, and the whole ERM formulation changes.

@SamuelAinsworth @zacharynado @konstmish It would be interesting to check empirically whether there's qualitatively different behaviour for BN models vs non-BN models.

@SamuelAinsworth @zacharynado @konstmish I remember reading this take, but don't remember where: As soon as you use BN, *the batch is an example* and the "dataset" is actually the collection of all possible unique batches. From this perspective, there should be a sweet spot BS that corresponds to the largest "dataset".

@francoisfleuret Great poll! Now I wonder for those who voted "still talk", whether it's "sad" or "great" :)

@ikostrikov I bet no computer scientist did, because from information gain perspective, it's a pretty inefficient word to start with!

@theshawwn @ducha_aiki @amy_tabb It's literally an example in every framework, I'd hardly call that esoteric O.o github.com/pytorch/exampl… github.com/google/flax/tr… - this one even has a nice timing table, takes 4.3h on 8xV100, meaning 1½ days! 2080 is not far: lambdalabs.com/blog/best-gpu-… But Dmytro didn't disagree?

@ducha_aiki @amy_tabb @theshawwn Yep

@ducha_aiki @amy_tabb @theshawwn yeah, so we agree? It was feasible years ago for a lone phd student with 1 GPU, and is even moreso nowadays. Just need a little bit of patience.

@theshawwn you are underestimating my age quite a bit :D But the code was very simple and didn't take long, it was more-or-less this: github.com/lucasb-eyer/lb… Also: twitter.com/giffmana/statu…

@code_star @amy_tabb @theshawwn That sounds more like an ablation or hparam search, note that I mention "ablations on smaller scale is OK". If you claimed the hparams to be "mostly robust" then you should be able to do just one run on ImageNet. If not, then you may want to try finding "early indicator" metrics.

@amy_tabb @code_star @theshawwn 2. cloud is the 5-30$ 20min option for the impatient when it works. When it doesn't, there's still the 5yo single GPU, just have 2w of patience (worst case here, single 5yo GPU, I don't believe that's current standard!) One can prepare the dreaded coursework while it's training:)

@amy_tabb @code_star @theshawwn (I'm not saying it's easy) rebuttal to all points: 1. most papers take pride in using the baseline's (r50) exact hparams for theirs, or highlight their robustness -> no own baseline training and no big hparam search needed. (ablations on smaller scale is OK)

@amy_tabb @theshawwn And engineering cost/skill I would have agreed, if we weren't talking about _the_ example that every big framework comes with.

@amy_tabb @theshawwn I read your linked post and agree with everything, but it does not contradict my statement? 1. If you do DL, you have min 1 GPU already 2. R50-i1k took me 2w on 1 GPU 5 ago, now much better. 3. It is _the_ gold standard, but a paper can't spend 2 gpu-weeks or 5$ cloud on it? huh?

@david_picard 3200 is made up, but I remember having some in the lower thousands during my phd, so it must have been either CV or Robotics back then.

@slow_nd_furious

While we're on the topic, a little plug that you can apply for free TPU access for your research: sites.research.google/trc/about/ (I have no idea what's the accept rate)

@theshawwn Yeah I don't blame you for thinking so, but I've also ran ImageNet on my single or dual GPU(s) in the uni lab back when it was GTX780 times, and gotten insights from it... and would do it again.

But I should add the disclaimer that I have not tried their library 😅

And I'm sure we will *still* see "we cannot afford an ImageNet experiment" in papers for years to come. The 3200$ publication fee and 2000$ conference travel are no problemo though. twitter.com/aleks_madry/st…

PS A few examples: in 30 min, we can train ResNet-18 to 67% ImageNet acc on *one A100*. In 20 mins, ResNet-50 to 75.6% on a p4d AWS machine (<$5!). CIFAR costs 2 cents/model. My students tell me this is fast ;) [More seriously, we haven't seen anything in PyTorch that compares.]

@KyleLiang5 Yup, and that's one of the reasons I really like my current workplace/group: Neither Jeff Dean nor Zoubin are last authors on all the papers just because they run the lab!

@yoshitomo_cs for doing ML, one really needs patience in all aspects of it =)

I really appreciate that OpenAI actually did that in the CLIP paper, by reporting the result on a 15M subset of YFCC + releasing the actual subset. It allowed us nice comparisons and sanity-checks in a recent paper, and allows trustable independent reproduction à la OpenCLIP. 👍 twitter.com/pbloemesquire/…

@DZhang50 I had not seen this paper yet, interesting approach. Not sure yet if I like or dislike this direction, as it risks locking us into a specific arch for a long time, although the approach itself certainly seems flexible.

@giffmana I've faced this too while working on image to image translation. Replaced all convs in the generator with depthwise + pointwise convs. To my surprise the inference time went UP by 30% on the GPU and 120% on the GPU!

@jekbradbury @elonmusk @askerlee Thanks for your answer, I almost did not see it! Are you aware of any (real) benchmarks? Because, sure, one can always get closer to peak by reducing peak.

@yoshitomo_cs Yes as a learning experience absolutely. But always pick (at least) one number from the paper and try to get as close as possible. Otherwise you may not learn much. I actually originally spent ~1mo doing this for BN when it first came out.