@karpathy's insights on the complexities of training LLMs really hit home. Keeping the #llama3 training alive was a journey filled with hard challenges across the entire tech stack. Reading it kept me sane, knowing others have a similar experience.
Having a massive fleet of GPUs is only a part of the story, you absolutely need to have an amazing team who can make it work.