Yi Wei @yiwei8

Back in 2019 Yura Malitsky and I wrote a paper about adaptive estimation of gradient Lipschitzness in gradient descent. In 2023 we tightened the constant factor in the estimate from 1/2 to 1/√2. We couldn't figure out if it could be further tightened to exactly 1. Finally gpt 5.6 pro gave us the answer: even with a constant factor of 0.98 the method would not converge. It created a PEP-style counterexample, which I couldn't do when I tried using PEP back in 2019. It's not anyone's big conjecture but I think it gives a better example of how LLMs are useful for doing math.

This spring I taught a grad course on “All of Convex Quadratic Optimization.” That may sound narrow, but quadratic problems are simple enough to analyze precisely, yet rich enough to reveal many of the core phenomena that drive modern optimization and learning. I wrote up the course notes here: ddrusvyat.github.io/DSC-243/

I was responding to a question about what's been happening in optimization and realized someone else might find it useful, so here it is, my biased(!) perspective with links.

(1/n) Introducing Hyperball — an optimizer wrapper that keeps weight & update norm constant and lets you control the effective (angular) step size directly. Result: sustained speedups across scales + strong hyperparameter transfer.

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by @BanghuaZ, Assistant Professor at the University of Washington @UW, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here: deeplearning.ai/short-courses/…

A student reached out asking for advice on research directions in optimization, so I wrote a long response with pointers to interesting papers. I thought it'd be worth sharing it here too: 1. Adaptive optimization. There has been a lot going on in the last year, below are some papers I personally found interesting. First of all, this paper by Li and Lan on Nesterov's acceleration of adaptive gradient descent: arxiv.org/abs/2310.10082 Check Corollary 1 for a simple description of their method. There is one thing I don't like about it: the amount by which we can increase the stepsize at each iteration decreases as t grows. That being said, I don't know if this restriction can be lifted, and perhaps it's the best thing we can get. Yura Malitsky and I also did some work on adaptive gradient descent, making the stepsizes a bit larger, roughly sqrt(2) improvement over our previous result: arxiv.org/abs/2308.02261 We still don't know if that's the best we can do or if a tighter analysis can give us better methods. I should also mention that there is more push in the literature on Polyak stepsize, see for instance these two papers: arxiv.org/abs/2407.04358 (a stepsize very similar to Polyak) arxiv.org/abs/2406.04142 (Polyak stepsize with momentum) 2. Adagrad-like methods still can be studied, I believe it's an underexplored direction. I wish there was more papers on studying the importance of coordinate-wise stepsizes. One paper on the topic I really liked is this study of when Adam is more useful than SGD: arxiv.org/abs/2402.19449 There is also some research on new practical methods, for instance, acceleration of DoG is interesting: arxiv.org/abs/2404.00666 And I also enjoyed reading this paper by Rodomanov et al. on line-search-inspired stochastic methods: arxiv.org/abs/2402.03210 3. I also like the direction of getting better assumptions for optimization theory and studying the implications. A good example is the gradient clipping literature: arxiv.org/abs/1905.11881 ((L₀, L₁)-smoothness) arxiv.org/abs/2305.01588 (same revisited) arxiv.org/abs/2406.04443 (on heavy-tailed noise) We need to bridge optimization assumptions with what we know about neural networks, so read about properties of neural networks themselves like this: arxiv.org/abs/2405.14813 (on scales of layers and how their type affects Lipschitz constants) 4. These days, people are using deep networks of all scales for their tasks, and they have discovered a lot of tricks that haven't been studied thoroughly in optimization literature: quantization, Straight-Through Estimator, (arxiv.org/abs/1903.05662), low-rank techniques such as LoRA, learning-rate warm-up, etc. You should expose yourself to those tricks to get a better understanding of what the current theory is lacking. If you're considering choosing optimization as the topic for your PhD, here are some extra thoughts. Right now there is less activity than about 5 years ago, most low-hanging fruits seem to have been taken, and the remaining questions seem quite challenging. So if you're looking for a field where it is easy to get publications, it might not be perfect. However, it's still a good field to produce meaningful theory. It's also important who you would work with, i.e. if you can find a good advisor, that often affects one's satisfaction to a larger degree than the topic itself, so make your decision carefully. As my last word of advice, I definitely encourage testing new methods on neural networks (and preferably not on CIFAR10/CIFAR100, because they give misleading results), at least something like nanoGPT (github.com/karpathy/nanoG…). When I was a PhD student, I did a lot of theoretical research testing my methods on logistic regression and that was useful to understand the theory, but I also had the wrong impression about what works and what doesn't because of that. If you can, do both, understand the theory as much as you can, but also learn its limits and failure modes.

New lecture recordings on RL+LLM! 📺 This spring, I gave a lecture series titled **Reinforcement Learning of Large Language Models**. I have decided to re-record these lectures and share them on YouTube. (1/7)

I believe successful neural network training represents cases of "near convexity": the optimization landscape, while technically non-convex, behaves enough like a convex problem that standard convex optimization is often applicable. At the same time, *in general* neural nets can't be trained due to non-convexity and non-smoothness, it's an emprical fact too. Try removing all normalization layers from ResNet-18 and watch it completely fail to train on CIFAR10 with SGD. Yet, we can train deep networks across countless tasks, and when it does work, it resembles convex optimization remarkably closely. Things such as averaging the weights or momentum, which aren't supposed to help on non-convex problems, somehow give good results. The convex theory of decaying the learning rates is tightly matching the experiments. And it all works for minimizing the loss, not just getting a small gradient norm like non-convex optimization theory would predict. So in a way, convex optimization theory is terrible for explaining why neural networks should train at all, but it's also almost perfect for predicting how the training goes when it does work.

Shalev @Shalev_lif

a year ago

The neural network objective function is a very complicated objective function. It's very non convex, and there are no mathematical guarantees whatsoever about its success. And so if you were to speak to somebody who studies optimization from a theoretical point of view, they

29 102 995 205K 486

Ever wonder why switching optimizers never actually helps? Introducing HeavyBall Benchmark

Anyone working on adaptive optimization methods and replacements for Adam should check this paper.

If you are interested in the latest advances related to asynchronous SGD and/or strategies for handling device (data, compute & communication) heterogeneity in distributed/federated/collaborative/decentralized learning, you may wish to watch my recent FLOW talk: youtube.com/watch?v=2YiYhw…

We just released a new optimizer Prodigy. Based on D-Adaptation, Prodigy has stronger theoretical guarantees and we derive how to estimate the stepsize for Adam. We tested it on a wide range of deep nets, so I can confidently say: it does work in practice. arxiv.org/abs/2306.06101

Workshops are the best way to get feedback on your ideas and I personally like attending them more than main conferences. If you have some work on optimization, please consider submitting it to the opt workshop at NeurIPS: opt-ml.org/cfp.html Submission deadline: 27 September

Is optimization still a good topic to work on for a PhD? Here is my detailed response to a student who reached out asking this question.