Querying Billions of GitHub Events Using Modal and DuckDB
noreasontopanic.com/p/querying-bil…
Very cool demo of how to use Modal for fan-out parallelism
did some tuning and can now download the last 5 years of public @github events in ~15m
3.5 TB of data (50k+ json.gz files) big fanout made super easy using @modal 🎉, lots of parallel downloads + retries with exponential back-off
blog post and open-source repo coming soon!
did some tuning and can now download the last 5 years of public @github events in ~15m
3.5 TB of data (50k+ json.gz files) big fanout made super easy using @modal 🎉, lots of parallel downloads + retries with exponential back-off
blog post and open-source repo coming soon! https://t.co/ZMnz44uxK5
PSA if you haven't used python in the past 5 years it's a completely new language now
- uv solved the package management problems entirely
- mypy type checks have become actually useful
- cpython is now much faster
- polars + fastapi + ruff replaced pandas + flask + black
I'm rediscovering python after a few years, and between @astral_sh and @modal_labs, I'm completely blown away by how fun it is.
no more env/dep insanity, and practically instant remote code execution 🎉
7K Followers 2K FollowingIt’s like Zack with an R. Bay native 🇺🇸🇮🇳. Product lad turned partner @BainCapVC. Investing in AI, developers, infra, & security. Grew up @Atlassian
36 Followers 409 FollowingDJ | Producer | 1/2 of @TheChainsmokers 🎶 | Grammy Award Winner 🏆 | Singer/Songwriter | Maine-born, NYC-made 🌍 | Living for music & moments ✨
240 Followers 479 Following||OAU|| Ogun | Student Baker | Scrabble ❤️ | NASELS | Official Campus Ambassador at International MUN @imunofficial ✨🇺🇳 || Helping the growth @WayoWork
6K Followers 5K FollowingFOLLOWS YOU 🫵 https://t.co/F7MzDOTC1k
ML/AI, R&D eng, quant trading, ASR in noise, TTS
Open AGI, ASI…—Open AI computation for */acc 😇
It's more fun to compute 🥰
14K Followers 15K FollowingAustin Powered. Co-founder of OpenStack & OpenInfra Foundation. General Manager of AI & Infrastructure for the Linux Foundation. open source for fun & profit.
14K Followers 1K Following“All methods are sacred if they are internally necessary” (GP @amplifypartners, prev @canvasvc; Head of Data @Mattermark; @palantirtech; @c4ads)
419K Followers 2K Following@Shopify CEO by day, Dad in evening, hacker at night. Aspiring comprehensivist. (tweets auto delete eventually) retweet=noteworthy share, not endorsement
10K Followers 2K FollowingCartoonist, Engineer, PM, Partner @a16z investing in infra & AI
Prev Product lead @HashiCorp, Founding Eng/PM @Transposit. Eng @AppDynamics. Opinions = own.
13K Followers 5K FollowingLorin Hochstein
Student of complex systems failures, resilience eng, cognitive sys eng. Will talk your ear off about @LFISoftware.
@[email protected]
8K Followers 161 FollowingData warehouse for customer-facing and internal analytics, built in collab with @duckdblabs.
Join us at Small Data SF, Nov 4th & 5th. https://t.co/TSxedEMVEm
15K Followers 2K FollowingFounder @RillData, the fastest path to operational BI. Previously founded @Metamarkets, @DCVC, @CustomInk. Lapsed computational biologist.
21K Followers 36 FollowingDuckDB is an analytical in-process SQL database management system. "DuckDB" and the DuckDB logo are registered trademarks of the DuckDB Foundation.
33K Followers 507 FollowingFounder/Chairman @thumbtack & @athenago & https://t.co/TRiL1X4kl6
I always dreamed of having 48 hours in a day: https://t.co/GcEoLPP1ic
18K Followers 20 FollowingA high-throughput and memory-efficient inference and serving engine for LLMs. Join https://t.co/lxJ0SfX5pJ to discuss together with the community!
9K Followers 5K Followingceo @tembo • prev founder/ceo @astronomerio (yes, that astronomer) • I follow back all startup people • and I unfollow if you unfollow
4K Followers 1K FollowingBoris Babayan said "architecture, operating system and languages, compiler, it's only one project". Investing in improvements to the project @AmplifyPartners.
2K Followers 69 FollowingIndiehacker. #BullMQ - https://t.co/uzk4uaxk1r - https://t.co/ECOwuvxKA3.
Follow my journey in trying to build profitable businesses around Open Source.
1.8M Followers 91 FollowingOfficial account of Ray Dalio, founder of Bridgewater Associates, author of #1 New York Times bestseller 'Principles,' professional mistake maker
270K Followers 5K FollowingPosts about Business. Building SaaS for 20+ years. Started https://t.co/xrOkg6UWaz (2005), KISSmetrics (2008) & https://t.co/uA827W8uqv (2020). What are you building? 🙏
39K Followers 2K FollowingI cover SCOTUS @TheEconomist and teach @BHSEC. Brooklynite, @UMich pol science PhD, dad of three, runner. Views here are mine.