Data Management Services Devs @bpldevelopment

What is Kafka? Apache Kafka is an open source distributed streaming platform designed for building real-time data pipelines and streaming applications. Kafka operates as a 𝗱𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗽𝘂𝗯-𝘀𝘂𝗯 𝗺𝗲𝘀𝘀𝗮𝗴𝗶𝗻𝗴 𝘀𝘆𝘀𝘁𝗲𝗺. Allowing applications to publish and subscribe to real-time or near-real-time data feeds. The 𝗵𝗶𝗴𝗵 𝘁𝗵𝗿𝗼𝘂𝗴𝗵𝗽𝘂𝘁, 𝘀𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆, 𝗳𝗮𝘂𝗹𝘁-𝘁𝗼𝗹𝗲𝗿𝗮𝗻𝗰𝗲, 𝗱𝘂𝗿𝗮𝗯𝗶𝗹𝗶𝘁𝘆, and 𝗲𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺 𝗞𝗮𝗳𝗸𝗮 𝗽𝗿𝗼𝘃𝗶𝗱𝗲𝘀 have made it a very 𝗽𝗼𝗽𝘂𝗹𝗮𝗿 𝗰𝗵𝗼𝗶𝗰𝗲 𝗳𝗼𝗿 use cases where 𝗿𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗱𝗮𝘁𝗮 𝗳𝗲𝗲𝗱𝘀 are required. 𝗧𝗵𝗲 𝗸𝗲𝘆 𝗰𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀 𝗼𝗳 𝗞𝗮𝗳𝗸𝗮 𝗶𝗻𝗰𝗹𝘂𝗱𝗲: 🔸 𝗣𝗿𝗼𝗱𝘂𝗰𝗲𝗿 Producers publish (write) messages to a Kafka topic. 🔸 𝗖𝗼𝗻𝘀𝘂𝗺𝗲𝗿 Consumers/subscribers subscribe to topics and process (read) the feed of published messages. 🔸 𝗕𝗿𝗼𝗸𝗲𝗿 Brokers are Kafka servers that store data and serve clients. Multiple brokers form a cluster. 🔸 𝗧𝗼𝗽𝗶𝗰 Topics are a feed name or message category to which messages are published by producers. 🔸 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻 Messages are organized into topics, which can then be further divided into partitions. This increases parallelism and scalability allowing consumers to read different partitions at the same time. 𝗞𝗮𝗳𝗸𝗮 𝗰𝗮𝗻 𝗯𝗲 𝘂𝘀𝗲𝗱 𝗳𝗼𝗿 𝗺𝗮𝗻𝘆 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲𝘀. A few of the more common 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲𝘀 𝗶𝗻𝗰𝗹𝘂𝗱𝗲: 🔹 𝗔𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗶𝗻𝗴 𝗱𝗮𝘁𝗮 𝗳𝗿𝗼𝗺 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝘀𝗼𝘂𝗿𝗰𝗲𝘀: Whether it's ETL pipelines, data lakes, or log aggregation; ingesting, organizing, transforming, and distributing data are core functions of Kafka. 🔹 𝗦𝘁𝗿𝗲𝗮𝗺 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Kafka can be used to build real-time analytics applications. 🔹 𝗘𝘃𝗲𝗻𝘁 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: systems or applications that rely on processing events in real-time such as IoT devices. 🔹 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴: Kafka is well suited for storing logs and metrics, allowing for real-time monitoring and alerts. Designed for fault tolerance, high throughput, durability, and scalability, Kafka serves as an intermediary service for handling large data. While Kafka provides many advantages for building real-time data pipelines and streaming applications, it's not without its drawbacks. The most noteworthy being the layers of complexity it adds to the system architecture. In saying that, if real-time data feed is required, Kafka is a popular choice.

Great leadership starts with self-awareness

I've been building microservice systems for 10 years. Here's the biggest lesson I've learned (painfully, so you don't have to!). It has nothing (in essence) to do with the old microservices v. monolith debate.

Did you miss my new keynote on Product Thinking at #Agile2022? Well it's now online! I wrote this as a follow up to Escaping the Build Trap, and what I've seen happening lately as more companies are maturing in #prodmgmt. Tell me what you think! vimeo.com/737906332

Turns out I have a lot to say about this! I expanded my replies into a more detailed blog entry here: simonwillison.net/2022/Oct/1/sof…

Gergely Orosz @GergelyOrosz

4 years ago

A few software engineering practices which many teams will also consider "best practices." What other useful engineering practices would you mention? (Yes, getting back to working on @EngGuidebook!)

92 251 2K 0 1K

How did REST end up meaning the opposite of REST? 🤔htmx.org/essays/how-did…

When Steve Jobs returned to Apple in 1997, he spoke at WWDC. An audience member insulted him. Here’s how he responded:

Satirical emulation of an 8080 microprocessor using a containerised microservices-based architecture running on Kubernetes with frontends for a CP/M test harness blog.davetcode.co.uk/post/21st-cent…

SQL patterns for data science pipelines ploomber.io/blog/sql/

Writing is an increasingly important skill in tech. Poor writing can hamper career progression, above a certain level. I wrote about tactics for more clear, more frequent, and more confident writing. Read it here: blog.pragmaticengineer.com/becoming-a-bet…

Bloomberg announced today that its Data License content is now available on AWS, enabling clients globally to receive content cloud-natively and dramatically reducing data on-boarding time. bloom.bg/3GuObug

Pleased to announce that Bloomberg has won Best Buy-Side Regulatory Data Solution Provider at A-Team Group's RegTech Insight Awards USA. @RegTechInsight bloom.bg/32R6aN6

New Stripe Press site: press.stripe.com. Team did an amazing job.

Terminology in the #API space can be a bit confusing. This is why @JRyanAPI of @Axway helps with "API Terminology Explained: Gateways, Portals, Catalogs, Marketplaces" and walks us through these concepts and their history and perspective for the future. youtube.com/watch?v=JPkd1C…

SEC now has an api for company filings w/ public documentation : ) sec.gov/edgar/sec-api-…

What's the Difference Between REST and RESTful? | Nordic APIs | b.mamund.com/3vpALKl "Below we’ll be taking a closer look at the origins, and the recommended usage, of REST and RESTful." @ArtCopywriter #API360

The Developer Advocate Iceberg. Learn more: dev.to/wassimchegham/… #devrel #DEVCommunity

Accessibility is Hard: Before You Get Toast-happy, Think About Everybody! by Attila Vágó hmh.engineering/accessibility-…

I just posted an article I really didn't want to have to write, but it's needed: svpg.com/product-vs-pro…

@BarryOSullivan As well as Campbell's Law - the system being measured will distort and corrupt to game the metric - and a host of ecological fallacies set up through the KPIs and analytics used to make sense of them