Extended Mind Transformers (EMTs) are a new approach to working with very large contexts and external data sources developed by @KlettPhoebe, @thomasahle, Normal's AI team. Inspired by the Extended Mind Thesis, we modify Multihead Attention to directly query a vector database.

Our method outperforms Retrieval Augmented Generation, RAG, on long-range retrieval tasks. Where RAG only does one query to the vector database per prompt, EMTs do one query for every layer in the transformer. This is a bit slower, but results in much better performance.

@NormalComputing @MaxAifer @KlettPhoebe @thomasahle This is one of the most interesting techniques that I have seen

@NormalComputing @KlettPhoebe @thomasahle Cool, will give this a read

@NormalComputing @KlettPhoebe @thomasahle That's awesome. Any GitHub link for the implementation too?

@NormalComputing @KlettPhoebe @thomasahle @memdotai mem it