smalltalkman @smalltalkman2
Joined May 2019-
Tweets2
-
Followers1
-
Following48
-
Likes7
The (true) story of development and inspiration behind the "attention" operator, the one in "Attention is All you Need" that introduced the Transformer. From personal email correspondence with the author @DBahdanau ~2 years ago, published here and now (with permission) following some fake news about how it was developed that circulated here over the last few days. Attention is a brilliant (data-dependent) weighted average operation. It is a form of global pooling, a reduction, communication. It is a way to aggregate relevant information from multiple nodes (tokens, image patches, or etc.). It is expressive, powerful, has plenty of parallelism, and is efficiently optimizable. Even the Multilayer Perceptron (MLP) can actually be almost re-written as Attention over data-indepedent weights (1st layer weights are the queries, 2nd layer weights are the values, the keys are just input, and softmax becomes elementwise, deleting the normalization). TLDR Attention is awesome and a *major* unlock in neural network architecture design. It's always been a little surprising to me that the paper "Attention is All You Need" gets ~100X more err ... attention... than the paper that actually introduced Attention ~3 years earlier, by Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio: "Neural Machine Translation by Jointly Learning to Align and Translate". As the name suggests, the core contribution of the Attention is All You Need paper that introduced the Transformer neural net is deleting everything *except* Attention, and basically just stacking it in a ResNet with MLPs (which can also be seen as ~attention per the above). But I do think the Transformer paper stands on its own because it adds many additional amazing ideas bundled up all together at once - positional encodings, scaled attention, multi-headed attention, the isotropic simple design, etc. And the Transformer has imo stuck around basically in its 2017 form to this day ~7 years later, with relatively few and minor modifications, maybe with the exception better positional encoding schemes (RoPE and friends). Anyway, pasting the full email below, which also hints at why this operation is called "attention" in the first place - it comes from attending to words of a source sentence while emitting the words of the translation in a sequential manner, and was introduced as a term late in the process by Yoshua Bengio in place of RNNSearch (thank god? :D). It's also interesting that the design was inspired by a human cognitive process/strategy, of attending back and forth over some data sequentially. Lastly the story is quite interesting from the perspective of nature of progress, with similar ideas and formulations "in the air", with a particular mentions to the work of Alex Graves (NMT) and Jason Weston (Memory Networks) around that time. Thank you for the story @DBahdanau !
"[A]n RFC that adds support for Rust to the Linux kernel" lkml.org/lkml/2021/4/14…
katahiromz🐜 @katahiromz
3K Followers 5K Following 画像カラスンポ: https://t.co/3qGCvFF1WK 万能プラカードジェネレータ: https://t.co/ajp49K3bLm クロスワードギバー: https://t.co/aG7R4RIk6s
Andrej Karpathy @karpathy
4.3M Followers 1K Following
Eric S. Raymond @esrtweet
62K Followers 262 Following Yes, I *am* that ESR. Well, it's the question people usually ask. Programmer, wandering philosopher, accidental anthropologist, troublemaker for liberty.
crablang @crablang
2K Followers 78 Following not associated or affiliated with @rustlang or the @rust_foundation • 5k ⭐️https://t.co/I4GoUujKjh
Spack @spackpm
2K Followers 3K Following A flexible package manager designed to support multiple versions, configurations, platforms, and compilers. Join us at https://t.co/aT6zWYxXKz!
Standard C++ @isocpp
36K Followers 24 Following Standard C++ Foundation, and ISO C++ standards committee - ISO/IEC JTC1/SC22/WG21
conda @condaproject
4K Followers 501 Following #conda has moved to 🐘 Mastodon: @[email protected] 🔗 LinkedIn: https://t.co/O1O5kWxg1X Join us! Conda is a Package & Environment manager
Wolfram @WolframResearch
69K Followers 332 Following Where computation meets knowledge. Creators of Mathematica, @Wolfram_Alpha, Wolfram Language. Founded in 1987 by @stephen_wolfram.
katahiromz🐜 @katahiromz
3K Followers 5K Following 画像カラスンポ: https://t.co/3qGCvFF1WK 万能プラカードジェネレータ: https://t.co/ajp49K3bLm クロスワードギバー: https://t.co/aG7R4RIk6s
3uTools @3utools
19K Followers 102 Following 3uTools is a free all-in-one tool, supplying diverse ringtones, wallpapers, apps for iOS users as well as supporting flash.
Diablo @Diablo
958K Followers 57 Following ESRB Rating: MATURE with Blood and Gore, Intense Violence, Language. #DiabloIV is available now. Welcome to Hell 🌹
Solid Project @ MIT @SolidMit
6K Followers 16 Following A platform for radically decentralized user-centric Linked Data applications. https://t.co/r6aqWo9zqI
Visual C++ @visualc
18K Followers 108 Following The Microsoft C++ team, working on @VisualStudio, @code, and #vcpkg.
Phoronix @phoronix
71K Followers 120 Following Founded by @MichaelLarabel in 2004, Phoronix is the largest #opensource news, #Linux hardware reviews & Linux PC/server/HPC performance benchmark site.
llvm.org @llvmorg
12K Followers 7 Following Open source project for compiler infrastructure, tools, and more. Here Be Dragons.
Microsoft @Microsoft
13.1M Followers 1K Following We're on a mission to empower every person and every organization on the planet to achieve more. Support: @MicrosoftHelps
Libera Chat @liberachat
3K Followers 33 Following
appfleet @appfleetcloud
1K Followers 4 Following Host your docker containers on the edge! Acquired by Cloudflare. Follow @jimaek for more content
Node.js @nodejs
913K Followers 599 Following The Node.js JavaScript Runtime. 🐢🚀 Need help with Node.js? We've got a repo for that: https://t.co/BaCSQqhNRO
Bootstrap @getbootstrap
518K Followers 21 Following Official account for Bootstrap, a toolkit providing simple and flexible HTML, CSS, and JS for popular UI components and interactions. Tweets by @mdo.
GCC - GNU Toolchain @gnutools
12K Followers 218 Following GCC, Binutils, GDB, GLIBC. The system compiler for the FOSS and Linux software ecosystem. Continually Improving.
midipix @mpixorg
55 Followers 10 Following midipix is a development environment that lets you create programs for Windows using the standard C and POSIX APIs. No compromises made, no shortcuts taken.
Open at Microsoft @OpenAtMicrosoft
87K Followers 9 Following All things #OpenSource at Microsoft https://t.co/4V3rben8iJ
Lantern @getlantern
42K Followers 765 Following The VPN that fights censorship. 12+ years helping millions access the open internet. Nonprofit. Since 2013.
Rust Foundation @rust_foundation
40K Followers 178 Following The nonprofit dedicated to stewarding the Rust programming lang & its community 🦀 bsky: https://t.co/pURKYFM3az Mastodon: rustfoundation
The Julia Language @JuliaLanguage
29K Followers 318 Following Julia is a high-level, dynamic programming language built for technical computing. Join the conversation at #JuliaLang @JuliaConOrg @JuliaInclusive
Fortran @fortranlang
4K Followers 0 Following Fortran programming language. Powering the world since 1956. Latest edition is Fortran 2023. The next one is Fortran 2028.
Java at Microsoft @JavaAtMicrosoft
16K Followers 281 Following Java usage at Microsoft spans from Azure to Minecraft, across LinkedIn to Visual Studio Code, and beyond! We use more Java than one can imagine.
Andreas Kling @awesomekling
65K Followers 2K Following building @ladybirdbrowser. recovering addict. husband of @katalinkult. uncle. gymnasium brother.
Niklas @nschoellhorn
206 Followers 110 Following Software developer, system administrator, freelancer, and aviation enthusiast.
Rust Language @rustlang
152K Followers 2 Following A programming language empowering everyone to build reliable and efficient software. ** This account is no longer active. Follow us on other platforms! **
Nim @nim_lang
5K Followers 384 Following
Haiku @haikuOS
6K Followers 11 Following An open-source operating system that specifically targets personal computing.
ReactOS @reactos
23K Followers 554 Following Discover a free and open source operating system able to run your favorite application and drivers compatible with Windows.
QGIS @qgis
63K Followers 63 Following Official account of QGIS - the user friendly Open Source Geographic Information System
Raspberry Pi @Raspberry_Pi
616K Followers 1K Following We make very small computers which you can buy from just $4. We are also the literal coolest. Be excellent to each other. Tech support: https://t.co/ZEBSfmuErK
Gitter @gitchat
5K Followers 649 Following The home of #community. Where #developers, #engineers & #scientists come to talk. For support, please get in touch on support at gitter.im
Google DeepMind @GoogleDeepMind
1.6M Followers 267 Following The engine room of @Google. Building AI safely and responsibly to solve the world’s most complex problems. Join us: https://t.co/jUHQA27iBL
OpenJDK @OpenJDK
41K Followers 0 Following The place to collaborate on an open-source implementation of the Java Platform, Standard Edition, and related projects · 🐘 https://t.co/UtKelUz9Xh
Eclipse Adoptium @adoptium
10K Followers 408 Following The Adoptium Working Group promotes and supports high-quality runtimes and associated technology for use across the @Java ecosystem. Part of @EclipseFdn
GitHub Status @githubstatus
119K Followers 1 Following
Red Language @red_lang
2K Followers 159 Following The full-stack programming language, fighting back against complexity.
The V Programming Lan... @v_language
12K Followers 7 Following Simple, fast, safe, compiled language for developing maintainable software.You might like
