Some interesting discussion on r/machinelearning about EfficientNet and CNN efficiency. reddit.com/r/MachineLearn… TBH, I think FLOPS as a measurement of models sometimes gets a bad rap. It has its downsides, but it's one of the harder metrics to "game".
@cHHillee Wow, that person really hates EfficientNets... but clearly they have no clue what the current state of things on latest GPUs and software stacks. It's pretty much in line with other heavily optimized architectures now.
@cHHillee Let's be honest. Neither MACs nor throughput are good performance measurement metrics individually, particularly in the deep learning applications. I think in the MLSys community you [should] find little to no attention to such metrics.
@cHHillee We should be optimizing for minimal memory bandwidth. Unfortunately, its very hard to do. First, utilized memory bandwidth is dependent on both hardware and software stack. Secondly, there aren't simple models for it, you need to understand the whole stack.