Cloud Based Integration & Messaging. Data Processing & Streaming (aka Data Pipeline). Open Data Hub

1. Message Queue in Kubernetes. Event-driven Messaging. Real-Time Data Streaming
2. RPC vs Messaging
3. Tibco Business Works BWCE
4. Message Brokers
   1. ActiveMQ message broker
   2. RabbitMQ message broker
   3. Redis message broker
   4. Apache Camel message broker
      1. Apache Camel K
   5. KubeMQ message broker
   6. Google Cloud Platform Pub/Sub
   7. JMS Message Queue vs. Apache Kafka
5. Cloud Based Integration. Integration Platform-as-a-Service (iPaaS) solutions
   1. Red Hat Fuse and Red Hat Fuse Online
   2. Syndesis open source integration platform
6. Debezium open source distributed platform for Change Data Capture (CDC) software design pattern
7. Red Hat Integration service registry and Apicurio
8. Data Mesh
9. Data Processing (aka Streaming Data, Data Pipeline or Big Data Pipeline)
   1. Apache Kafka
      1. Kafka Tools
      2. Strimzi kubernetes operator for apache kafka
      3. Apache Kafka Desktop Clients
   2. AWS Kinesis
   3. MQTT
   4. Banzai Cloud Supertubes (Cloud Native Kafka implementation)
   5. Confluent Cloud (Apache Kafka Re-engineered for the Cloud)
   6. Redpanda (kafka alternative). A modern streaming platform for mission critical workloads
      1. KsqlDB
   7. Apache Pulsar
   8. Apache Flink
   9. Hazelcast JET
   10. Postgress as message queue
10. Workflow Engines
    1. Zeebe
    2. Apache Airflow
    3. Couler
11. Red Hat AMQ (ActiveMQ Artemis broker and Apache Kafka)
    1. Red Hat AMQ Broker (ActiveMQ Artemis)
    2. Red Hat AMQ Streams
    3. Slides of Red Hat AMQ Streams
12. Open Data Hub AI-as-a-Service (AIaaS) platform
13. Integration Platform as a Solution (iPaaS). Platforms for collecting, storing and routing customer event data
    1. IpaaS Vendors
14. eBooks
15. Related
16. Questions and Answers
17. Videos
18. Tweets

Message Queue in Kubernetes. Event-driven Messaging. Real-Time Data Streaming

• Wikipedia: Message Broker
• Wikipedia: Event-driven messaging
• Wikipedia: Streaming Data
• nginx.com: Event-Driven Data Management for Microservices 🌟
• dzone: Event-Driven Architecture as a Strategy Event-driven architecture provides five key benefits to modern application architecture: scalability, resilience, agility, data sharing, and cloud enabling.
• infoq.com: From Monolith to Event-Driven: Finding Seams in Your Future Architecture
• wikipedia: Enterprise service bus
• thenewstack.io: The Rise of the Event Streaming Database 🌟
• cncf.io: The need for Kubernetes Native Messaging Platform in Hybrid Cloud Environment
• wiprodigital.com: A Guide to Enterprise Event-Driven Architecture
• medium: Introduction to Event-Driven Architecture 🌟 The essential concepts that every developer should know
• ibm.com: Event-driven cloud-native applications (microservices) The event backbone is being part of the microservices mesh, providing the publish-and-subscribe communication between microservices and enabling the support of loosely coupled event-driven microservices.
• stackoverflow.blog: How event-driven architecture solves modern web app problems 🌟 In this article, we’ll discuss some of the problems driving innovation in modern web development. Then we’ll dive into the basics of event-driven architecture (EDA), which tries to address these problems by thinking about back-end architecture in a novel way.
• sebalopezz.medium.com: Monolith to Microservices + Event-Driven Architecture 🌟
• confluent.io: Event-Driven Microservices Architecture (white paper) 🌟 Microservices are an architectural pattern that structures an application as a collection of small, loosely coupled services that operate together to achieve a common goal. Because they work independently, they can be added, removed, or upgraded without interfering with other applications. While there are numerous benefits to microservices architecture, like easier deployment and testing, improved productivity, flexibility, and scalability, they also pose a few disadvantages, as independently run microservices require a seamless method of communication to operate as one larger application. Event-driven microservices allow for real-time microservices communication, enabling data to be consumed in the form of events before they’re even requested. In this white paper, we’ll cover how event-driven microservices work, presenting a sample currency exchange platform to illustrate the design and architecture of an application composed of event-driven microservices using Apache Kafka® and Confluent Platform. We also discuss other aspects of microservices architectures, such as team structure, continuous delivery, deployment, and testing. Lastly, we discuss how Apache Kafka and Confluent Platform enable and extend core principles of microservices, including decoupling, separation of concerns, agility, and real-time streaming of event data.
• redhat.com: Event-driven architecture: Understanding the essential benefits 🌟 Event-driven architectures bring significant benefits when managing many endpoints, but it also has its complexities to be aware of.
• medium: Introduction to Message Queues 🌟
• headspring.com: Is Kafka or RabbitMQ the right messaging tool for you?
• dzone: Why Pub/Sub Isn’t Enough for Modern Apps Pub/Sub is the most appropriate way of architecting the delivery side of an event-driven architecture (especially for the web).
• baeldung.com: Pub-Sub vs. Message Queues 🌟
• engineering.atspotify.com: Spotify’s Event Delivery – The Road to the Cloud (Part I)
• medium: Monolithic to Microservices Architecture with Patterns & Best Practices 🌟
• infoq.com: Turning Microservices Inside-Out
• towardsdatascience.com: Architecture for High-Throughput Low-Latency Big Data Pipeline on Cloud 🌟 Scalable and efficient data pipelines are as important for the success of analytics, data science, and machine learning as reliable supply lines are for winning a war.
• dzone: RESTful Applications in An Event-Driven Architecture Hybrid architecture with both RESTful and event-driven services.
• developers.redhat.com: Distributed transaction patterns for microservices compared
• thenewstack.io: The Rise of Event-Driven Architecture
• jinwookim928.medium.com: Why Not Event Driven Architecture?

• thenewstack.io: Streaming Data and the Modern Real-Time Data Stack

  	Modern Data Stack	Modern Real-Time Data Stack
  Language	SQL	SQL
  Deployment	Cloud-native	Cloud-native
  Data Ops	Complex batch transformations every 15 minutes, hourly or daily	Simple incremental transformations every second
  Insights	Monthly, Weekly or Daily	Instantly
  Cost	Affordable at massive scale	Affordable at massive scale and speed

• blog.direktiv.io: Event driven orchestration with Knative (part 1)

• blog.direktiv.io: Redefining event-driven orchestration for automation & applications
• pub.towardsai.net: Deep Dive into Event-Driven architecture | Gul Ershad
• developer.com: An Introduction to Event Driven Microservices
• dzone.com: What Are Microservices and The Event Aggregator Pattern? 🌟 Learn about the Event Aggregator pattern for organizing event communication in a microservices architecture with an example.
• ibm.com: Event-driven cloud-native applications (microservices) The event backbone is being part of the microservices mesh, providing the publish-and-subscribe communication between microservices and enabling the support of loosely coupled event-driven microservices.
• irfanyusanif.medium.com: Best practices to communicate between microservices
• swapnil-chougule.medium.com: Rapid Feature Engineering through SQL
• blog.twitter.com: Processing billions of events in real time at Twitter
• codeopinion.com: Event Sourcing vs Event Driven Architecture
• thenewstack.io: The Path to Getting the Full Data Stack on Kubernetes
• medium.com/tinyclues-vision: 4 Design Principles for Robust Data Pipelines Design Principles for traditional Software Engineering quickly fail when working with large and diverse sets of data — a new way of thinking about this difference.
• medium.com/fiverr-engineering: How to Share Data Between Microservices on High Scale Several approaches suitable for a scalable system considering tradeoffs between availability and consistency
• medium.com/codex: Microservices Communication — Queues Topics and Streams In this article, I will speak about types of messages broker that are famous in microservices communication.
• blog.bitsrc.io: Why Microservices Should use Event Sourcing 🌟 6 Reasons where you need to use Event Sourcing in Micro-services
• emirayhan.medium.com: What is the difference Message Queue and Message Bus? 🌟
• verraes.net: DDD and Messaging Architectures 🌟 An overview of my different series on patterns in distributed systems. A good collection of Messaging Patterns
• medium.com/event-driven-utopia: Comparing Stateful Stream Processing and Streaming Databases How do these two technologies work? how do they differ, and when is the right time to use them?
• thenewstack.io: How to Get Started with Data Streaming With Kafka and associated tools, developers can create stream-processing pipelines that transform data for real-time applications.
• linkedin.com: How to Move From a “Wait for it…” Batch-Processing Culture to a “Get It Now” Real-Time Data Culture
• dzone: Resilient MultiCloud Messaging Messaging becomes an important technical option when operating solutions span clouds, hybrid deployments, and even inter-application and inter-process communication.
• juhache.substack.com: From Data Engineer to YAML Engineer

RPC vs Messaging

• particular.net: RPC vs. Messaging – which is faster?

Tibco Business Works BWCE

• medium.com/dev-jam: TIBCO Business Works vs. Apache Camel — A short Comparison 🌟
  • ESB stands for Enterprise Service Bus. It is an architecture pattern that enables disparate applications to connect seamlessly with each other. Under the hood, ESB uses an integration tool, more commonly known as middleware. Integration or Middleware tools have capabilities such as data transformation (such as XML to JSON), protocol transformation (like FTP to HTTP), content-based message routing and service orchestration. Many vendors converted this concept into an ESB product with standard connectors
  • In this blog, I will compare two such integration tools, one which I have worked extensively i.e TIBCO BW and the de facto open source integration framework Apache Camel. I choose open source as it has a bright future and becoming very popular among many enterprises. I did not choose Mule ESB because it is not completely open-source as its most vital components come under a licensed enterprise version.

Message Brokers

• Apache ActiveMQ
• Dzone: Introduction to Message Brokers. Part 1: Apache Kafka vs. RabbitMQ
• Dzone: Introduction to Message Brokers. Part 2: ActiveMQ vs. Redis Pub/Sub
• developers.redhat.com: Choosing the right asynchronous-messaging infrastructure for the job
• medium.com/@codeeverywhere: Redis vs Kafka vs RabbitMQ
• kai-waehner.de: When to use Apache Camel vs. Apache Kafka? 🌟 Should I use Apache Camel or Apache Kafka for my next integration project? The question is very valid and comes up regularly. This blog post explores both open-source frameworks and explains the difference between application integration and event streaming. The comparison discusses when to use Kafka or Camel, when to combine them, when not to use them at all. A decision tree shows how you can quickly qualify out one for the other.

ActiveMQ message broker

• ActiveMQ 5.x “classic”
• ActiveMQ Artemis Apache ActiveMQ is a subproject of Apache ActiveMQ. It has been donated to the Apache Software Foundation in 2015. There were lots of changes in project names in the past. The Artemis project first started as JBoss Messaging and got renamed to HornetQ in August 2009.
• Apache Artemis JMeter Running the ActiveMQ Artemis JMeter Performance Testing Examples.
• developers.redhat.com: Implementing Apache ActiveMQ-style broker meshes with Apache Artemis

RabbitMQ message broker

• K8s prevent queue worker Pod from being killed during deployment How to prevent a Kubernetes (like RabbitMQ) queue worker Pod from being killed during deployment while handling a message?
• medium.com: RabbitMQ vs. Kafka An architect’s dilemma
• blog.rabbitmq.com: First Application With RabbitMQ Streams
• geshan.com.np: How to use RabbitMQ and Node.js with Docker and Docker-compose
• salaboy.com: Event-Driven applications with CloudEvents on Kubernetes
• medium.com/@paolo.gazzola: How to deploy a high available and fault tolerant RabbitMQ service in an on-premise Kubernetes multi-node cluster environment

Redis message broker

• Redis
• Redis Pub/sub
• betterprogramming.pub: The Perfect Message Queue Solution Based on the Redis Stream Type

Apache Camel message broker

• Apache Camel Camel is an Open Source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data. In version 3 we use <5MB memory, including the JVM. Also reflection free, low GC, super modular, native compilation friendly.
• Quora.com: What’s the difference between Apache Camel and Kafka?
• dzone: Hybrid multi-cloud event mesh architectural design Building the event mesh with Camel
• developers.redhat.com: Integrating systems with Apache Camel and Quarkus on Red Hat OpenShift

Apache Camel K

• Apache Camel K is a lightweight cloud-integration platform that runs natively on Kubernetes. Based on the famous Apache Camel, Camel K is designed and optimized for serverless and microservices architectures.
• developers.redhat.com: Six reasons to love Camel K
• developers.redhat.com: Extending Kafka connectivity with Apache Camel Kafka connectors
• developers.redhat.com: Design event-driven integrations with Kamelets and Camel K
• thenewstack.io: Camel K Brings Apache Camel to Kubernetes for Event-Driven Architectures
• github.com/osa-ora/camel-k-samples

KubeMQ message broker

• KubeMQ.io: Kubernetes Native Message Queue Broker
• devops.com: Best of 2019: Implementing Message Queue in Kubernetes
• kubemq.io: Kafka VS KubeMQ 🌟
• github.com/kubemq-io/kubemq-community 🌟 KubeMQ community version is now available as an open-source project!
• dzone: KubeMQ: A Modern Alternative to Kafka This article introduces a modern, Kubernetes-native message queue called KubeMQ, to show how organizations trying to implement Kafka on Kubernetes can benefit from it.

Google Cloud Platform Pub/Sub

• Google Cloud Platform Pub/Sub
• A generic framework of concurrent consumers for Google Cloud Platform Pub/Sub 🌟 An example shows how to publish messages to Pub/Sub and build a service to consume the messages concurrently using the Python multiprocessing module

JMS Message Queue vs. Apache Kafka

• kai-waehner.de: Comparison: JMS Message Queue vs. Apache Kafka

Cloud Based Integration. Integration Platform-as-a-Service (iPaaS) solutions

• Wikipedia: Cloud Based Integration (iPaaS)
• Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.
• Integration platform as a service (iPaaS) is a set of automated tools for connecting software applications that are deployed in different environments. iPaaS is often used by large business-to-business (B2B) enterprises that need to integrate on-premises applications and data with cloud applications and data.
• blog.axway.com: What is iPaaS?
• ibm.com: iPaaS (Integration-Platform-as-a-Service): iPaaS is a cloud-based solution that simplifies application integration across on-premises and cloud environments, to help you accelerate innovation and lower your integration and operations costs.

Red Hat Fuse and Red Hat Fuse Online

• Red Hat Fuse
• Red Hat Fuse Online

Syndesis open source integration platform

• Syndesis open source integration platform (OpenSource Project for Red Hat Fuse Online)
• developers.redhat.com: Low-code microservices orchestration with Syndesis

Debezium open source distributed platform for Change Data Capture (CDC) software design pattern

• Change Data Capture, or CDC, is a well-established software design pattern for a system that monitors and captures the changes in data so that other software can respond to those changes. CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus. Applications can read these change event streams and access these change events in the order in which they occurred.
• Debezium: Stream changes from your database
• developers.redhat.com: Decoupling microservices with Apache Camel and Debezium
• A good explanation of how to avoid distributed transactions using outbox pattern: Transaction Log Tailing With Debezium
• developers.redhat.com: Capture database changes with Debezium Apache Kafka connectors
• developers.redhat.com: Change data capture for microservices without writing any code
• debezium.io: Lessons Learned from Running Debezium with PostgreSQL on Amazon RDS
• info.crunchydata.com: PostgreSQL Change Data Capture With Debezium
• medium.com: Stream Your Database into Kafka with Debezium An Introduction and Experience Report. Insightful post by David Hettler of comsysto about their usage of Debezium, touching on many details like outbox pattern, Avro schemas, Postgres on RDS etc.
• noti.st: Change Data Capture with Flink SQL and Debezium 🌟
• vladmihalcea.com: A beginner’s guide to CDC (Change Data Capture)
• shopify.engineering: Capturing Every Change From Shopify’s Sharded Monolith
• developers.redhat.com: Db2 and Oracle connectors coming to Debezium 1.4 GA
• medium: Change Data Capture — Using Debezium
• daily.dev: Building a fault-tolerant event-driven architecture with Google Cloud, Pulumi and Debezium
• pradeepdaniel.medium.com: Creating an ETL data pipeline to sync data to Snowflake using Kafka and Debezium Setting up a real-time data pipeline from scratch to sync data from transactional databases to Snowflake cloud warehouse.
• medium: A Visual Introduction to Debezium 🌟 A story-based introduction to understanding what Debezium is, how it is made of, and how it works in a real-world scenario
• debezium.io: Using Debezium to Create a Data Lake with Apache Iceberg
• developers.redhat.com: Improve your Kafka Connect builds of Debezium. A Kubernetes native way of building Kafka Connect images in Debezium 1.6
• conduktor.io: Capturing MySQL database changes using Debezium, Kafka, and Conduktor Using Kafka Connect in Conduktor and specifically how to use Debezium to monitor the changes in a MySQL database.

Red Hat Integration service registry and Apicurio

• Red Hat Integration service registry
• Apicurio Registry An API/Schema registry - stores APIs and Schemas.
• Event streaming and data federation: A citizen integrator’s story
• redhat.com: Using a schema registry to ensure data consistency between microservices Make interservice communication easier by using a schema registry.

Data Mesh

• martinfowler.com: Data Mesh Principles and Logical Architecture
• infoq.com: Data Mesh Principles and Logical Architecture Defined
• martinfowler.com: How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh
• towardsdatascience.com: Data Domains and Data Products Practical guidance from the field
• mrpaulandrew.com: BUILDING A DATA MESH ARCHITECTURE IN AZURE – PART 2

Data Processing (aka Streaming Data, Data Pipeline or Big Data Pipeline)

• Awesome Streaming A curated list of awesome streaming (stream processing) frameworks, applications, readings and other resources.
• cloudblog.withgoogle.com: Turn any Dataflow pipeline into a reusable template
• thenewstack.io: Part 1: The Evolution of Data Pipeline Architecture
• eng.uber.com: Uber’s Journey Toward Better Data Culture From First Principles
• satishchandragupta.com: Scalable Efficient Big Data Pipeline Architecture
• openshift.com: How to Orchestrate Data Pipelines with Applications Deployed on OpenShift

Apache Kafka

• Apache Kafka
• developers.redhat.com: Using secrets in Kafka Connect configuration
• developers.redhat.com: Capture database changes with Debezium Apache Kafka connectors
• Awesome Kafka
• Single Message Transformations - The Swiss Army Knife of Kafka Connect
• medium: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake Replacing Apache Airflow with Debezium.
• medium: Apache Kafka Startup Guide: System Design Architectures: Notification System, Web Activity Tracker, ELT Pipeline, Storage System 🌟
• medium: Getting Started With Kafka on OpenShift
• containerjournal.com: Red Hat Platform Brings Kafka Closer to Kubernetes
• lightbend.com: Monitor Kafka Consumer Group Latency with Kafka Lag Exporter
• AKHQ (previously known as KafkaHQ) 🌟 Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more…
• banzaicloud.com: Kafka Schema Registry on Kubernetes the declarative way
• Build a simple cloud-native change data capture pipeline
• banzaicloud.com: Bulletproof Kafka, and the tale of an Amazon outage=
• confluent.fr: Infrastructure Modernization with Google Anthos and Apache Kafka
• confluent.io: Apache Kafka DevOps with Kubernetes and GitOps
• Build a data streaming pipeline using Kafka Streams and Quarkus
• levelup.gitconnected.com: Kafka for Engineers 🌟 Here are things about Kafka that you need to understand as a software engineer.
• confluent.io: How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka
• banzaicloud.com: Kafka on Kubernetes - using etcd 🌟
• softwareengineeringdaily.com: Kafka Applications with Tim Berglund (podcast) 🌟
• medium: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake
• infoq.com: Building a SQL Database Audit System using Kafka, MongoDB and Maxwell’s Daemon
• tecmint: How to Install Apache Kafka in CentOS/RHEL 7
• medium: Processing guarantees in Kafka “Duplicates and lost messages are due not only to features of the messaging systems, but in the design of producer and consumer applications as well.” One of the best posts on processing guarantees in kafka.
• davidxiang.com: Kafka As A Database? Yes Or No
• medium: How Pinterest runs Kafka at scale
• medium: Google Pub/Sub Lite for Kafka Users
• medium: 4 Microservices Caching Patterns at Wix
• Confluent.io: Intro to Apache Kafka: How Kafka Works 🌟
• levelup.gitconnected.com: Kafka for Engineers
• medium: Microservices in Rust with Kafka
• medium: Apache Kafka in a Nutshell 🌟 Architecture, Use Cases, and a Getting Started guide — rolled into one
• confluent.io: Simplifying Apache Kafka Multi-Cluster Management Using Control Center and Cluster Registry
• kai-waehner.de: App Modernization and Hybrid Cloud Architectures with Apache Kafka
• kai-waehner.de: Apache Kafka and MQTT (Part 1 of 5) – Overview and Comparison
• medium: Solutions to Communication Problems in Microservices using Apache Kafka and Kafka Lens
• kafka-tutorials.confluent.io 🌟
  • kafka-tutorials.confluent.io: How to join a stream and a lookup table 🌟 If I have events in a Kafka topic and a table of reference data (aka a lookup table), how can I join each event in the stream to a piece of data in the table based on a common key?
• confluent.io: DevOps for Apache Kafka with Kubernetes and GitOps 🌟
• dzone.com: Microservices, Event-Driven Architecture and Kafka 🌟
• medium: Understanding Kafka Topic Partitions Everything in Kafka is modeled around partitions. They rule Kafka’s storage, scalability, replication, and message movement.
• kafka-tutorials.confluent.io: How to count messages in a Kafka topic
• confluent.io: Apache Kafka Made Simple: A First Glimpse of a Kafka Without ZooKeeper 🌟
• piotrminkowski.com: Knative Eventing with Kafka and Quarkus
• blog.cloudera.com: Scalability of Kafka Messaging using Consumer Groups
• thenewstack.io: Beyond the Quickstart: Running Apache Kafka as a Service on Kubernetes
• towardsdatascience.com: You Can Replace Kafka with a Database
• Handling Retries in Kafka: If You’re Using Kafka With Your Microservices, You’re Probably Handling Retries Wrong
• confluent.io: What’s New in Apache Kafka 2.8
• devclass.com: Apache Kafka 2.8.0 previews life without ZooKeeper
• instaclustr.com: Apache Kafka Architecture: A Complete Guide 🌟
• youtube playlist: Kafka Connect Tutorials | Kafka Connect 101: REST API 🌟 KafkaConnect uses a REST API to expose its management capabilities. tlberglund demonstrates many of the key functions available using the REST API, including creating connectors, viewing their status, and accessing troubleshooting information.
• developers.redhat.com: Event-driven APIs and schema governance for Apache Kafka: Get ready for Kafka Summit Europe 2021
• developers.redhat.com: Building resilient event-driven architectures with Apache Kafka
• tech.ebayinc.com: Resiliency and Disaster Recovery with Kafka
• newrelic.com: Effective Strategies for Kafka Topic Partitioning 🌟
• gentlydownthe.stream A children’s book about Apache Kafka.
• confluent.io: Apache Kafka Made Simple: A First Glimpse of a Kafka Without ZooKeeper
• dzone: Event-Driven APIs and Schema Governance for Apache Kafka As a developer, I’m always excited to attend so many great sessions addressing critical challenges in the Apache Kafka ecosystem like how changes to event-driven APIs are leading developers to focus on contract-first development for Kafka.
• phoenixnap.com: How to Set Up and Run Kafka on Kubernetes 🌟
• piotrminkowski.com: Knative Eventing with Quarkus, Kafka and Camel
• itnext.io: Configuring Kafka Sources and Sinks declaratively in Kubernetes using Knative This solves the complexity in work flow of compiling JARs and uploading them to a Kafka connect cluster. Using Knative it can be possible to leverage the Kubernetes cluster and define Kafka sources and sinks with Kubernetes objects.
• strimzi.io: Kafka upgrade improvements
• redhat.com: Introducing Red Hat OpenShift Streams for Apache Kafka
• developers.redhat.com: Getting started with Red Hat OpenShift Streams for Apache Kafka
• developers.redhat.com: Managing the API life cycle in an event-driven architecture: A practical approach 🌟
• baeldung.com: List Active Brokers in a Kafka Cluster Using Shell Commands 🌟
• developers.redhat.com: How to secure Apache Kafka schemas with Red Hat Integration Service Registry 2.0
• mercurytfs.blogspot.com: Colas Kafka
• grafana.com: Get comprehensive monitoring for your Apache Kafka ecosystem instances quickly with Grafana Cloud
• dzone: Next-Gen Data Pipes With Spark, Kafka and k8s 🌟 This article examines the architecture patterns and provides some sample code for the readers to implement in their own environment.
• confluent.io: Making Apache Kafka Serverless: Lessons From Confluent Cloud
• developer.confluent.io 🌟🌟 over ten hours of FREE video courses with hands-on exercises, 50+ event streaming patterns, deep-dive articles on Kafka’s internals, and a ton more.
• itnext.io: Sending Messages to Kafka in Kubernetes
• cloudhut.dev: Running Apache Kafka on Kubernetes successfully A comparison for different installation methods for running Kafka in Kubernetes
• developers.redhat.com: The outbox pattern with Apache Kafka and Debezium 🌟
• towardsdatascience.com: Overview of UI Tools for Monitoring and Management of Apache Kafka Clusters
• analyticsindiamag.com: How Uber is Leveraging Apache Kafka For More Than 300 Micro Services
• itnext.io: Securely Decoupling Kubernetes-based Applications on Amazon EKS using Kafka with SASL/SCRAM Securely decoupling Go-based microservices on Amazon EKS using Amazon MSK with IRSA, SASL/SCRAM, and data encryption
• medium: Running Kafka in Kubernetes, Part 1: Why we migrated our Kafka clusters to Kubernetes At Wise, we chose to migrate our Apache Kafka clusters, previously running on Amazon Web Services (AWS) EC2 instances, into a multi-cluster Kubernetes setup. This article is the first part of a two-part series aiming to outline the motivations behind this choice and the challenges we faced.
• betterprogramming.pub: How to Handle Duplicate Messages and Message Ordering in Kafka Dealing with the challenges faced when using Apache Kafka
• medium: Optimizing Kafka Streams Apps on Kubernetes by Splitting Topologies
• blog.twitter.com: Processing billions of events in real time at Twitter
• inder-devops.medium.com: Kafka- Best practices & Lessons Learned | By Inder
• datadoghq.com: Monitoring Kafka performance metrics
• blog.workwell.io: How to manage your Kafka consumers from the producer
• slack.engineering: Building Self-driving Kafka clusters using open source components
• adam-kotwasinski.medium.com: Kafka mesh filter in Envoy
• conduktor.io: What is Apache Kafka? Learn about Apache Kafka and its ecosystem in 20 minutes.
• medium.com/airwallex-engineering: Kafka Streams: Iterative Development and Blue-Green Deployment Blue-Green Deployment for Kafka Stream Applications.
• redhat.com: How we use Apache Kafka to improve event-driven architecture performance When designing your event-driven architecture, consider these ways to configure Kafka to improve performance.
• dev.to: Running Kafka on kubernetes for local development
• medium.com/udemy-engineering: Introducing Hot and Cold Retries on Apache Kafka An overview of how to build a fault-tolerant event delivery system by using non-blocking retries of Apache Kafka® in Udemy Payments Team
• conduktor.io/kafka: Learn Apache Kafka like never before Conduktor Kafkademy is the quickest, easiest and most effective way for you to learn Apache Kafka for free.
• developers.redhat.com: Which is better: A single Kafka cluster to rule them all, or many?
• medium.com/dna-technology: Why we dropped event sourcing with Kafka Streams when given a second chance
• kai-waehner.de: When NOT to use Apache Kafka?
• betterprogramming.pub: Everything You Need To Know About Kafka 🌟 A simple guide for beginners
• learnk8s.io/kafka-ha-kubernetes: Designing and testing a highly available Kafka cluster on Kubernetes 🌟 Learn how to design a Kafka cluster to achieve high availability using standard kubernetes resources and test how it tolerates maintenance and total node failures
• blog.developer.adobe.com: Exploring Kafka Producer’s Internals 🌟 This is the first part of a series where we explore Kafka client’s internals. This post focuses on the Kafka Producer.
• medium.com/altitudehq: Kafka retries and maintaining the order of retry events 🌟
• medium.com/cloudnesil: Kafka Streams State Store at Scale Managing Interactive Queries on multiple application instances Alternative solutions and their comparisons.
• towardsdev.com: Performance Testing Your Kubernetes Kafka Cluster In this article, you will learn how to benchmark Apache Kafka instances running on Kubernetes against parallel loads.
• medium.com/@hardiktaneja_99752: Lessons after running Kafka in production 🌟 TLDR: Have atleast as many paritions (P) as max number of consumers(C), but number of partitions should always be multiple of number of consumers. P = n * C, n =1,2,3…
• betterprogramming.pub: Monitoring Kafka Applications — Implementing Healthchecks and Tracking Lag For KafkaConsumer, Streams, Spring-Kafka, Kafka-Connect
• thenewstack.io: LinkedIn Layered Architecture Minimizes Kafka Scaling Issues With Kafka, too many data producers can cause issues, as can having too many data consumers. Here’s how LinkedIn separated the resources to alleviate exhaustion.
• developers.redhat.com: End-to-end field-level encryption for Apache Kafka Connect
• blog.datumo.io: Setting up Kafka on Kubernetes - an easy way In this article, you will learn how to create a basic deployment for Kafka (with Zookeeper) on Kubernetes
• linkedin.com: Kafka Cluster Setup on Kubernetes
• medium.com/wix-engineering: Troubleshooting Kafka for 2000 Microservices at Wix Debugging microservices’ event driven communication is not trivial, especially when it’s based on event streaming platforms like Apache Kafka. While scale grows more easily and the distributed system becomes more decoupled and resilient, the operational aspects become much harder.
• engineering.grab.com: Zero trust with Kafka
• freecodecamp.org: The Apache Kafka Handbook – How to Get Started Using Kafka 🌟 Apache Kafka is an open source event streaming tool that transports tons of data w/ low latency. This link covers its core concepts, how to use its CLI, & how to install + build a project with it.
• medium.com/@rramiz.rraza: Kafka metrics monitoring with Prometheus and Grafana 🌟
• rogulski.it: Consume Kafka events with Knative service and FastAPI on kubernetes 🌟 In this article, you will learn how to build a fully scalable, event-driven and easy-to-maintain system using Python (FastAPI), Kafka, and Knative
• piotrminkowski.com: Concurrency with Kafka and Spring Boot
• thenewstack.io: Kafka on Kubernetes: Should You Adopt a Managed Solution? A look at the various factors to consider when deciding whether to deploy Kafka yourself or to purchase a managed solution
• thelinuxnotes.com: How to deploy Kafka in Kubernetes with Helm chart + kafdrop

Kafka Tools

• Kafdrop – Kafka Web UI 🌟
• redpanda-data/kowl Kowl is a Web UI for Apache Kafka that allows exploring messages, consumers, configurations and more with a focus on a good UI & UX.
• KLoadGen - Kafka + (Avro/Json Schema) Load Generator 🌟 KLoadGen is kafka load generator plugin for jmeter designed to work with AVRO and JSON schema. It allows sending kafka messages with a structure defined as an AVRO Schema or a Json Schema. It connects to the Scheme Registry Server, retrieve the subject to send and generate a random message every time.
• dev.to: Learn how to use Kafkacat – the most versatile Kafka CLI client 🌟
• github.com/lensesio/fast-data-dev (Lenses Box) Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors. A apachekafka docker image that actually works without zookeeper. If you don’t want do deal with docker-compose this one is for you.
• dzone: Visualize your Apache Kafka Streams using the Quarkus Dev UI Visualize your Apache Kafka Streams using the Quarkus Dev UI
• medium: Solutions to Communication Problems in Microservices using Apache Kafka and Kafka Lens
• github.com/sauljabin/kaskade kaskade is a text user interface for kafka, which allows you to interact and consume topics from your terminal in style!

Strimzi kubernetes operator for apache kafka

• strimzi.io
• developers.redhat.com: how easy to deploy and configure a Kafka Connect on Kubernetes through strimziio operator and use secrets
• developers.redhat.com: Introduction to Strimzi: Apache Kafka on Kubernetes (KubeCon Europe 2020) 🌟
• strimzi.io: Optimizing Kafka producers
• strimzi.io: Optimizing Kafka consumers 🌟
• strimzi.io: Optimizing Kafka producers 🌟
• pepy.tech/project/strimzi-kafka-cli 🌟 - pypi.org/project/strimzi-kafka-cli
• strimzi/kafka-kubernetes-config-provider: Kubernetes Configuration Provider for Apache Kafka Apache Kafka supports pluggable configuration providers which can load configuration data from external sources. The configuration providers in this repo can be used to load data from Kubernetes Secrets and Config Maps. It can be used in all Kafka components and does not depend on the other Strimzi components. So you could, for example, use it with your producer or consumer applications even if you don’t use the Strimzi operators to provide your Kafka cluster. One of the example use-cases is to load certificates or JAAS configuration from Kubernetes Secrets.
• strimzi.io: Using Kubernetes Configuration Provider to load data from Secrets and Config Maps
• strimzi.io: Using HTTP Bridge as a Kubernetes sidecar
• strimzi.io: Using Open Policy Agent with Strimzi and Apache Kafka
• strimzi/strimzi-canary This repository contains the Strimzi canary tool implementation. It acts as an indicator of whether Kafka clusters are operating correctly. This is achieved by creating a canary topic and periodically producing and consuming events on the topic and getting metrics out of these exchanges.
• medium: Mastering Apache Kafka on Kubernetes — Strimzi K8s operator
• medium.com/@ahmed.farhan: Kafka Setup in Kubernetes Using Strimzi K8s operator — Part 2
• medium.com/adaltas: Operating Kafka in Kubernetes with Strimzi

Apache Kafka Desktop Clients

• conduktor.io 🌟 Apache Kafka Desktop Client. We created Conduktor, the all-in-one friendly interface to work with the Kafka ecosystem. Develop and manage Apache Kafka with confidence.

AWS Kinesis

• AWS Kinesis
• softkraft.co: WS Kinesis vs Kafka comparison: Which is right for you? 🌟

MQTT

• mqtt.org MQTT: The Standard for IoT Messaging
• developers.redhat.com: Deploying the Mosquitto MQTT message broker on Red Hat OpenShift, Part 1
  • developers.redhat.com: Deploying the Mosquitto MQTT message broker on Red Hat OpenShift, Part 2

Banzai Cloud Supertubes (Cloud Native Kafka implementation)

• Banzai Cloud
• Banzai Kafka Operator
• The benefits of integrating Apache Kafka with Istio

Confluent Cloud (Apache Kafka Re-engineered for the Cloud)

• confluent.io The Complete Event Streaming Platform for Apache Kafka.
• Focus on building apps and not managing clusters with a scalable, resilient and secure event streaming platform. Event streaming with Kafka made simple on AWS, Azure and GCP clouds.
• mongodb.com: DaaS with MongoDB and Confluent
• confluent.io: Confluent and Microsoft Announce Strategic Alliance
• confluent.io: Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana

Redpanda (kafka alternative). A modern streaming platform for mission critical workloads

• Redpanda 🌟 is a Kafka® compatible event streaming platform. No Zookeeper, no JVM, and no code changes required. Use all your favorite open source tooling - 10x faster.
• hub.docker.com/r/vectorized/redpanda Easy Docker experience to use VectorizedIO Redpanda in a container. Streaming platform for mission critical workloads, Kafka compatible, no Zookeeper, no JVM, no code changes required - 10x faster.
• Redpanda is now Free & Source Available
• softwareengineeringdaily.com: Redpanda: Kafka Alternative with Alexander Gallego 🌟

KsqlDB

• ksqlDB The event streaming database purpose-built for stream processing applications.
• Kafka Streams and ksqlDB Compared – How to Choose

Apache Pulsar

• Apache Pulsar is an open-source distributed pub-sub messaging system originally created at Yahoo and now part of the Apache Software Foundation
• Pulsar vs Kafka – Comparison and Myths Explored

Apache Flink

• Apache Flink Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
• How to set up Apache Flink on Kubernetes for real time data processing
• flink.apache.org: How to natively deploy Flink on Kubernetes with High-Availability (HA)

Hazelcast JET

• Hazelcast JET Open-Source Distributed Stream Processing
• devops.com: Hazelcast Simplifies Streaming for Extremely Fast Event Processing in IoT, Edge and Cloud Environments

Postgress as message queue

• dagster.io: Postgres: a better message queue than Kafka?

Workflow Engines

• wikipedia: Workflow Engine

Zeebe

• Zeebe workflow engine
• infoq.com: Event Streams and Workflow Engines – Kafka and Zeebe 🌟
• Orchestration Made Easy with Zeebe and Kafka

Apache Airflow

• towardsdatascience.com: A journey to Airflow on Kubernetes
• dzone: Apache Airflow Architecture on OpenShift
• redhat.com: Monitoring Apache Airflow using Prometheus
• towardsdatascience.com: Apache Airflow for containerized data-pipelines Are you having problems running tasks with a different version of Python on Airflow? In this article, I explain how to solve this issue.
• Apache Airflow official helm chart 🌟
• youtube: Airflow Helm Chart : Quick Start For Beginners in 10mins
• snowflake.com: Migrating Airflow from Amazon EC2 to Kubernetes
• dev.to: Get started with Apache Airflow
• betterprogramming.pub: Running Airflow Using Kubernetes Executor and Kubernetes Pod Operator with Istio There are many obstacles when you deploy Airflow with the Kubernetes executor, Pod Operator and Istio since the Istio proxy sidecar makes the worker pod hang and run forever. In this article, you will learn how to overcome that.
• towardsdatascience.com: Apache Airflow Architecture 🌟 A deep dive into Apache Airflow architecture and how it orchestrates workflows
• airflow.apache.org: KubernetesPodOperator 🌟🌟🌟
  • The KubernetesPodOperator allows you to create and run Pods on a Kubernetes cluster.
  • The KubernetesPodOperator uses the Kubernetes API to launch a pod in a Kubernetes cluster. By supplying an image URL and a command with optional arguments, the operator uses the Kube Python Client to generate a Kubernetes API request that dynamically launches those individual pods.
  • The KubernetesPodOperator enables task-level resource configuration and is optimal for custom Python dependencies that are not available through the public PyPI repository. It also allows users to supply a template YAML file using the pod_template_file parameter. Ultimately, it allows Airflow to act a job orchestrator - no matter the language those jobs are written in.
• dataengineeringcentral.substack.com: Why is everyone trying to kill Airflow? 🌟 Apache Airflow has been the ruler of Data Engineering orchestration for years, is the end in sight?
• airflow.apache.org: Add Owner Links to DAG You can set the owner_links argument on your DAG object, which will make the owner a clickable link in the main DAGs view page instead of a search filter.
• docs.astronomer.io: Dynamically generating DAGs in Airflow How to load DAGs from YAML files in Airflow dynamically?
• blog.devgenius.io: Send information from Databricks to Airflow
• medium.com/apache-airflow: Passing Data Between Tasks with the KubernetesPodOperator in Apache Airflow 🌟 TL;DR: Use the @task.kubernetes decorator!
• medium.com/@piyush_74867: Apache Airflow on Kubernetes at scale — a peak under the hood
• medium.com/@alfahreiza: Building an ELT Pipeline: From CSV to BigQuery using dbt
• medium.com/apache-airflow: What we learned after running Airflow on Kubernetes for 2 years In this case study, you will learn how the team at Teya scaled Airflow on Kubernetes to 5000 daily tasks

Couler

• Couler Couler aims to provide a unified interface for constructing and managing workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

Red Hat AMQ (ActiveMQ Artemis broker and Apache Kafka)

• Red Hat AMQ overview
• Red Hat AMQ = AMQ Broker (Apache ActiveMQ Artemis) + AMQ Streams (Apache Kafka)

Red Hat AMQ Broker (ActiveMQ Artemis)

• Apache ActiveMQ Artemis broker
• developers.redhat.com: JDBC Master-Slave Persistence setup with Activemq using Postgresql database
• developers.redhat.com: Connecting external clients to Red Hat AMQ Broker on Red Hat OpenShift

Red Hat AMQ Streams

• Understanding Red Hat AMQ Streams components for OpenShift and Kubernetes 🌟
• Red Hat AMQ streams (kafka): Simplify Apache Kafka on Red Hat OpenShift
• Set up Red Hat AMQ Streams custom certificates on OpenShift
• speakerdeck.com: Apache Kafka with Red Hat AMQ Streams 🌟
• HTTP-based Kafka messaging with Red Hat AMQ Streams
• blog.jromanmartin.io: How to upgrade Strimzi Operator using the CLI

Product	Also Known As	Components	URL
Red Hat AMQ 6	JBoss AMQ 6	Apache ActiveMQ	Ref
Red Hat AMQ 7	JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite	AMQ Broker + AMQ Streams	Ref
Red Hat AMQ 7	JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite	JBoss AMQ 7 (Broker) + Apache Kafka	Ref
Red Hat AMQ 7	JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite	Apache ActiveMQ Artemis + Apache Kafka	Ref

Slides of Red Hat AMQ Streams

Click to expand!

Open Data Hub AI-as-a-Service (AIaaS) platform

• Open Data Hub
• Open Data Hub 0.6 brings component updates and Kubeflow architecture
• A development roadmap for Open Data Hub

Integration Platform as a Solution (iPaaS). Platforms for collecting, storing and routing customer event data

• quandarycg.com: Everything You Need To Know About System Integration (And IPaaS) 🌟
• blog.hubspot.com: The 22 Best iPaaS Vendors for Any Budget

IpaaS Vendors

• rudderstack.com iPaaS - opensource.com: Stream event data with rudderstack
• Mulesoft
• etc

eBooks

• O’Really: Streaming data

Related

• Service meshes to the rescue: Load balancing and scaling long-lived connections in Kubernetes 🌟 Kubernetes doesn’t load balance long-lived connections, some Pods might receive more requests than others, In case you are using HTTP/2, gRPC, RSockets, AMQP. Any work around?

Questions and Answers

• adambien.blog - 75^th airhacks.tv Questions and Answers: Kafka, JAX-RS, MicroProfile, JSON-B, GSON, JWT, VSC, NetBeans, Java Fullstack “Kafka vs. JAX-RS / RPC, thoughts about APIs, JSON-B vs. GSON, Path.of over Paths.get, Java Records, MicroProfile JWT, beginners vs. expert content, best Java fullstack, code coverage, NetBeans in 2020, Visual Studio Setup for Java, screencast configuration, ReactJS / Angular over JSF?, JSON-P vs. JSON-B, security code scanning”

Videos

Click to expand!

Tweets

Click to expand!

Pub-Sub ≠ Partitioning ≠ Multiplexing pic.twitter.com/0ZVaH9Mxvr

— Clemens Vasters 🇪🇺☁📨 (@clemensv) July 28, 2020

We are excited to announce that KubeMQ community version is now available as an open-source project!

The community version supports all messaging patterns, connectors, bridges, and run in production. Give us a star on Github if you like our project!https://t.co/0ufRQ5bhCE

— KubeMQ (@KubeMq) September 10, 2021

How do we design a system using the 𝐞𝐯𝐞𝐧𝐭 𝐬𝐨𝐮𝐫𝐜𝐢𝐧𝐠 paradigm? How is it different from normal system design? What are the benefits? We will talk about it in this post. pic.twitter.com/PhKNDDCmMv

— Alex Xu (@alexxubyte) June 23, 2022

Push notifications are a very popular feature for many applications.

This is how to design a scalable push notification service: ↓ {1/13} pic.twitter.com/BWsaCKSrnr

— Fernando 🇮🇹🇨🇭 (@Franc0Fernand0) October 1, 2022

Redis is commonly known as a key-value server, but actually is also a messaging server.

This is how Redis Pub/Sub works and when it's a good choice: {1/10} ↓ pic.twitter.com/Mj9o7HQCOi

— Fernando 🇮🇹🇨🇭 (@Franc0Fernand0) October 29, 2022