Scale Engineer @scale_engineer

@ParamSiddh You should check us out too 😊

11 Basic concepts every Backend Engineer Should Know👇

100 weeks of writing newsletter through @beehiiv Consistency is paying off

How did Instagram scale from 0 → 14M users in just over a year with only 3 engineers? - Built on Django for rapid iteration, iOS-only launch - AWS migration: EC2, S3, CloudFront, Elastic Load Balancing - Data strategy: PostgreSQL + NoSQL, caching with Redis/Memcached, sharding & replication - Media processing: JPEG compression, H.264 encoding, ML for moderation & tagging. Scaling further: - Explore System: Multi-stage ranking + Two-Tower Neural Network for personalized recommendations - Security & Monitoring: Encryption, 2FA, Sentry, Pingdom - AI-driven personalization: Bayesian optimization, CV tagging, engagement weighting Key lessons: Start simple → optimize later, cloud flexibility, caching & AI personalization = growth

keyboard_arrow_left Previous keyboard_arrow_right Next

It's Black Friday, traffic suddenly explodes 10x. Your app automatically adds more servers to handle the load. But just adding servers isn’t enough something else could break (like your database or backend). What extra strategy would keep your app running, even when traffic goes crazy?

Your friend starts streaming a fast-paced FPS on Discord. Suddenly, their WiFi drops from 100 Mbps to 5 Mbps. The stream doesn’t just stop Discord lowers quality and drops frames to keep it alive. If you were designing this, what would you sacrifice: clarity or smoothness?

Your storage layer is cracking. Cassandra struggles with multi-tenant workloads at massive scale. Billions of tweets must stay instantly available. Do you keep pushing and tuning Cassandra, or take the leap and design a custom storage system from scratch?

Pinterest didn’t start dreaming of 500M users. In 2010, 3 founders + 1 engineer ran Python scripts in a tiny apartment. Their first stack was simple: Python for app logic, NGINX as HTTP proxy, MySQL with a read replica, MongoDB for counters, and a basic task queue. Users flooded in, they added more tech, Redis, Cassandra, MBase, but that created new problems. By late 2011, the lesson was clear: fewer, reliable tools win. Core stack: - MySQL → reliable storage, sharded to scale horizontally - Redis → blazing fast in-memory data, feeds & follower graphs - Memcached → caches frequent data, reduces DB load, absorbs spikes - Scaling & architecture: - Monolith → microservices: Search, Feed, MySQL (sharding), PinLater (background tasks) - PinLater → queues tasks in MySQL, crash-resistant, auto-retries, priority management - Zookeeper → services auto-register, better load balancing, elasticity Data pipelines: Kafka → Secor → S3 → real-time capture of pins, likes, follows, errors → structured analytics Lessons & wisdom: Log everything from day 1 Pick boring, proven tech Don’t start with complexity Monitor what matters Simplify before optimizing Pinterest proves: speed first, simplicity wins, and reliable tools scale best. Read full article: hw.glich.co/p/how-pinteres…

Imagine running a massive library with millions of books. Visitors walk in, ask for a book, and expect instant results. To handle this, you need a powerful search system. That’s exactly what Elasticsearch does for data... Think of Elasticsearch as Google for your own data. It’s a distributed, horizontally scalable search engine that stores data as JSON documents. Core flow: Data → Indices → split into Shards → distributed across Nodes → form a Cluster Inverted Index → makes search 🔥 fast Key concepts to remember: Document → JSON object Index → collection of docs Shard → index split Replica → redundancy Cluster/Node → group of servers Used for: logs, metrics, app & website search, analytics. Trusted by Netflix, eBay, Walmart

Ever heard of Serverless and thought: "Wait… no servers?" Don’t worry, you’re not alone. Let’s break down what Serverless actually means, how it works, and why it’s reshaping modern apps. First things first: Serverless ≠ No Servers Your code still runs on servers. The difference? You don’t manage them. The cloud provider handles provisioning, scaling & maintenance. You just focus on writing code. Non-Serverless example (AWS EC2): • You write backend code (say Node.js/Python/Go). • To run it, you spin up an EC2 instance (a virtual machine). • You pick RAM, storage, OS, install dependencies, monitor usage. • You’re billed per hour, even if no traffic. You manage everything. Serverless example (AWS Lambda): • You upload your code. • AWS decides RAM, OS, scaling, deployment. • Code runs only when triggered (API request, event, etc). • Billed per invocation (e.g. $0.001/request). No idle costs. No server headaches. Billing difference: • Non-serverless: Pay 24×7 (even at 0 traffic). • Serverless: Pay only when code runs. If your app has zero traffic, you pay zero, but if you have high traffic, the cost can skyrocket too. Scaling: • Traditional servers → you manually configure Auto-Scaling Groups. • Serverless → scales automatically. Got 1000 users suddenly? Your function runs in parallel across multiple instances, no setup needed. In a nutshell: Serverless = run code without worrying about servers. It’s event-driven, scales automatically, and saves costs. Perfect for modern apps where traffic can be unpredictable.

Ever wondered how your password stays safe online…

Rohit Lakhotia 🐝 @rohitlakh

a year ago

Ever wondered how your password stays safe online… …or how you can trust that a downloaded file isn’t tampered with? The answer is Hashing: a powerful technique that turns data into unique codes. Let’s break it down 👇

3 5 12 1K 3

How is it even possible?

Rohit Lakhotia 🐝 @rohitlakh

a year ago

Ever made a video call straight from your browser and thought: "Wait, how is this possible without an app or plugin?" That’s the magic of WebRTC: real-time peer-to-peer communication on the web. Let’s break it down 🧵

3 4 11 1K 5

How can you log into an app using Google or Facebook without sharing your password? Or how apps fetch your data securely without asking for your credentials? That’s thanks to OAuth. Let’s break it down 🧵

What is TikTok's secret sauce? Read the article

Rohit Lakhotia 🐝 @rohitlakh

a year ago

Ever wondered how TikTok delivers millions of personalized videos smoothly every day? It’s not magic: it's a mix of microservices, ML, databases, and CDNs working in perfect sync. Let’s peek into TikTok’s tech stack

2 6 11 949 12

How Reddit handles 91 million active users daily? Find out

Rohit Lakhotia 🐝 @rohitlakh

a year ago

Reddit has 504M+ users with 91M daily actives. That’s an insane scale. But with billions of posts containing images, GIFs & videos, Reddit faced a tough challenge: Efficiently storing & serving media metadata at scale. Here’s how they solved it 👇

2 2 8 768 6

What's behind the BBC's architecture!

Rohit Lakhotia 🐝 @rohitlakh

a year ago

How does the BBC handle millions of visitors during breaking news when traffic can TRIPLE in seconds? The secret: Serverless computing (AWS Lambda). Here’s how the world’s oldest broadcaster scaled for the internet age 🧵

3 3 6 730 3

Twitter stays online when millions are tweeting! How??

Rohit Lakhotia 🐝 @rohitlakh

a year ago

Twitter stays online even when millions are tweeting every second! Maintaining high availability (HA) at this scale is no joke. Here’s how Twitter does it, with Kubernetes 🧵

1 4 7 723 4

Why you buy more when you are in Amazon. Find out!

Rohit Lakhotia 🐝 @rohitlakh

a year ago

Ever wondered how Amazon seems to know exactly what you want, sometimes before you do? From books to gadgets, its recommendation engine is scarily accurate. Let’s break down how it works 🧵

4 4 6 910 5

The birth of Instagram.

Rohit Lakhotia 🐝 @rohitlakh

a year ago

How did Instagram scale from 0 to 14 million users in just over a year with only 3 engineers? From simple photo-sharing app to billion-user platform, here's the technical journey behind one of tech's greatest scaling stories 🧵

2 5 20 3K 17