Fluffy @fluffystack

@huggingface @virattt @WeAviate @cohere 🧵12/ Key Takeaways (2/3): C. Rewritten query + raw user query provided better results when used with a reranker, but at the cost of increased latency (~900ms). P.S. Not sure why just using the keywords fared poorly with the reranker in the mix. @Nils_Reimers thoughts?

@huggingface @virattt @WeAviate @cohere 🧵11/ Key Takeaways (1/3): A: Re-rankers substantially enhance context retrieval for intricate tasks such as mapping queries to code explanations. B. We've only scratched the surface with embeddings. So much so that it got us toying with the idea of training one for code-centric…

@huggingface @virattt 🧵10/ Fully committed we expanded our retrieval techniques to include Semantic, BM25, and a Hybrid search. And finally decided to throw a reranker into the mix, i.e. top 5 results from @WeAviate against top 100 re-ranked using @cohere's reranker.

@huggingface @virattt 🧵9/ (continued) • Keyword Array: Combining keywords from the rewritten query. • Rewritten Query: Rewritten query + Keywords • Hybrid Approach: Rewritten query + Raw user query

@huggingface @virattt 🧵8/ I couldn't resist running an experiment to compare various query term strategies: • Raw User Query: Directly from the GitHub issue. • Baseline: Solely 'SeamlessM4Tv2'. • Baseline_2: The specific function name "SeamlessM4Tv2ForSpeechToText" from the raw user query.

@huggingface 🧵7/ We then decided to rewrite the user query. Using GPT-3.5, we restructured the user query & extracted key words from the raw user query. Thanks @virattt!

@huggingface 🧵6/ This prompted us to check if there was something wrong with how we were querying our weaviate db. We simplified our search to solely "SeamlessM4Tv2,". The results were notably precise.

@huggingface 🧵5/ Take this: A straightforward search—lifting the user's query directly from the issue, code name drop and all—unveiled our top 5 file contenders. Task captured? Absolutely. But related to model used by the user? Far from it.

🧵4/ A pattern emerged: @huggingface's dev team meticulously organizes the repo with a clear nomenclature. Our metric? The presence of "seamless_m4t" within the filepaths of the Top 5 results, ranked by similarity scores. Not a perfect measure, but a practical proxy.

🧵3/ We randomly landed on issue #29139 in the transformers repo, tackling a Speech To Text query with the SeamlessM4Tv2 model. Our objective? Sift through the sea of files to pinpoint those crucial to this issue. The question then arises—what makes a file "relevant"?

🧵 2/ Objective - Our pipeline, akin to a developer, must identify the most relevant code files (using their explanations) in response to a user query — files Fluffy uses to provide answers.

🧵1/ At Fluffy, testing our "Talk to a Repo" feature revealed a challenge: our retrieval pipelines, designed for smaller open-source repositories, struggle with larger behemoths like Transformers.

@omarsar0 @paperswithcode @w_conviction @amanrsanger @cursor_ai @rauchg @mutableai @Wattenberger @GitHubNext @simonw 🔗 We're truly grateful for all the insights shared by folks building in public. What tips or resources have been game-changers for your projects? Eager to learn from your experiences! 🌟

@omarsar0 @paperswithcode @w_conviction @amanrsanger @cursor_ai @rauchg @mutableai @Wattenberger @GitHubNext 6/ 🧩 Delving into @simonw's insights on Symbex, LLMs for Code Gen/Explanation, and GitHub Issue Clustering has been interestubg. His articles have offered us valuable direction to bridge the gaps in our approach!

@omarsar0 @paperswithcode @w_conviction @amanrsanger @cursor_ai @rauchg @mutableai @Wattenberger 5/ 💡 @GitHubNext's CoPilot for Docs & @Wattenberger's PenPal have inspired us to rethink Fluffy's design. We're committed to exploring and iterating on new forms to enhance the developer experience. We'd love to hear your thoughts on what works best for you!

@omarsar0 @paperswithcode @w_conviction @amanrsanger @cursor_ai @rauchg @mutableai 4/ 📊 "Not all files in a repo are equal" - a key insight from @Wattenberger's Repo Visualization work: githubnext.com/projects/repo-…. It prompts us to ask: Which files are important, what do they tell us, and in what contexts?

@omarsar0 @paperswithcode @w_conviction @amanrsanger @cursor_ai 3/ 📝 Thanks to @rauchg for highlighting @mutableai's Auto Wiki - It programmatically creates Wikis for OS repos. Trying to reverse-engineer their approach by mapping codebases to their docs gave us a fresh perspective on tackling the source graph puzzle. 🔍

@omarsar0 @paperswithcode 2/ 🤔 Realization hit us - chatting with an OS Repo isn't as straightforward as with a PDF. This nudged us towards a source graph approach. 🧠 Big nods to @w_conviction for their initial ideas at conviction.com/startups. And to @amanrsanger for sharing his @cursor_ai journey…

1/ 📚 First up, understanding the constraints that generative models operate within. Hats off to @omarsar0 for the curated ML paper goldmine. When we want to go deeper, we turn to @paperswithcode. ML Papers: github.com/dair-ai/ML-Pap… -

🔄 Yesterday, we reflected on the feedback we got from our initial users @neodocs & the insights we derived from it. Today, I want to talk about the community insights that helped us navigate these challenges. Ready for a deep dive? 🧵👇 #BuildInPublic