Ethan @EthanWH

I've been using 𝐀𝐩𝐚𝐜𝐡𝐞 𝐊𝐚𝐟𝐤𝐚 for years now and I absolutely love it. Let me explain the message/event flow in simple terms. Give it a read. 👇 𝐃𝐨 𝐲𝐨𝐮 𝐤𝐧𝐨𝐰 𝐰𝐡𝐚𝐭? 𝐀𝐩𝐚𝐜𝐡𝐞 𝐊𝐚𝐟𝐤𝐚 𝐰𝐚𝐬 𝐛𝐨𝐫𝐧 𝐨𝐮𝐭 𝐨𝐟 𝐚 𝐩𝐫𝐨𝐛𝐥𝐞𝐦. 😉 LinkedIn engineers => faced difficulties in tracking website metrics, activity streams and other operational data. A team of engineers => led by @jaykreps , @nehanarkhede and @junrao started developing a distributed publish-subscribe messaging system that could handle high-throughput, low-latency data streams. This system eventually became Apache Kafka. It was open sourced in early 2011. The name 'Kafka' was chosen by Jay Kreps. He named the system after the famous author 'Franz Kafka'. 😊 Kreps was an admirer of Franz Kafka's writing and found the name fitting for a system that dealt with the flow of information. It's written in Java and Scala. Later they founded => 'Confluent (@confluentinc)' (a company) in 2014 to provide commercial support and additional tools for Kafka users. 📌 𝐋𝐞𝐭'𝐬 𝐮𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝 𝐭𝐡𝐞 𝐛𝐚𝐬𝐢𝐜 𝐟𝐥𝐨𝐰. [1.] Producer sends a message ◾ An application acts as a producer, creating a message with data (payload) and optional key. ◾ The producer connects to a broker in the Kafka cluster and identifies the target topic. ◾ Kafka uses a partitioner to determine which partition within the topic should receive the message. This ensures load balancing and parallel processing. ◾ The message is delivered to the leader replica of the chosen partition. [2.] Message storage and replication ◾ The leader replica appends the message to its log segment. ◾ The message receives a unique offset, serving as its position within the log. ◾ The leader replicates the message to follower replicas for fault tolerance. [3.] Consumer fetches messages ◾ An application acts as a consumer, joining a consumer group. ◾ Consumers within the same group share offsets and coordinate consumption. ◾ Each consumer fetches messages from its assigned partitions based on its committed offset. ◾ The consumer receives batches of messages and processes them. [4.] Acknowledging consumption ◾ Once processing is complete, the consumer commits its new offset. ◾ This tells Kafka which messages have been successfully consumed. ◾ Kafka tracks committed offsets for each consumer in the group. [*.] Flow continues ◾ Producers continue sending messages and consumers keep fetching and processing them based on their latest offsets. ◾ This cycle ensures ordered delivery and reliable consumption even with failures or restarts. Remember, 👉 Message flow is asynchronous. Producers don't wait for consumers to process messages. 👉 Consumers can lag behind producers if processing is slow. 👉 Kafka offers mechanisms for handling failures and ensuring at-least-once or exactly-once delivery semantics. Topics => Partitions =>Log Segments (Data is actually stored in log segments) Follow @techNmak

4 107 559 50K 715

View Details

Ethan @EthanWH

a year ago

A ver si el próximo año la retransmisión no es una mierda, que hay estar ajustando el volumen manualmente 6 veces por canción #EurovisionRTVE

0 0 3 823 0

View Details

Neo Kim @systemdesignone

a year ago

If you want to become good at system design (not kidding), learn these case studies:

13 211 2K 161K 4K

View Details

Sahn Lam @sahnlam

2 years ago

Useful Free APIs for Developers

12 335 2K 182K 3K

View Details

Ethan @EthanWH

2 years ago

@xtralife_es #NS2xtralife

0 0 1 22 0

View Details

xtralife @xtralife_es

2 years ago

🎉Celebramos esa generación con un #SORTEO de #NintendoSwitch2🎉 Para participar: ➡️Sigue a @xtralife_es 🔁Haz RT a esta publicación 💬Comenta con #NS2xtralife en esta publicación 📅Hasta 14/04/2025

6K 5K 3K 389K 210

View Details

Ethan @EthanWH

2 years ago

@mnfpayasa94 Es coña no? XDDDDD

0 0 0 16 0

View Details

Ethan @EthanWH

2 years ago

@ErJeremias @8BitDo Estás deseando eh…..😝

0 0 1 27 0

View Details

Mark Paluch 🦋 👨‍💻,🎹&🚑 @mp911de

2 years ago

Spring Data 2025.0.0-M1 released #valentines 💝💐 spring.io/blog/2025/02/1…

0 6 11 3K 1

View Details

Ethan @EthanWH

2 years ago

@W17LY0 ESPAÑA 🇪🇸 🐂 🥘

0 0 1 16 0

View Details

Tech Fusionist | Kushal Gangil @techyoutbe

2 years ago

How does DNS work? (Simplified)

5 100 638 58K 730

View Details

Ethan @EthanWH

2 years ago

@W17LY0 En tres días están en Walapop a 600€ el par jajaja

0 0 1 16 0

View Details

Ethan @EthanWH

2 years ago

@JmpMotion Diosssssssss

0 0 0 17 0

View Details

Ethan @EthanWH

2 years ago

@JmpMotion Que?!

0 0 1 27 0

View Details

Ethan @EthanWH

2 years ago

@VivianVvj No te olvides de que los platelmintos hacen esgrima de penes y el ganador le derrama el semen al otro en la piel el cual será absorbido y automáticamente fecundado