Apache Kafka examples

The following examples will help you understand how to use the different Apache Kafka components for Knative.

Prerequisites

All examples require:

  • A Kubernetes cluster with
    • Knative Eventing v0.9+
    • Knative Serving v0.9+
  • An Apache Kafka cluster

Setting up Apache Kafka

If you want to run the Apache Kafka cluster on Kubernetes, the simplest option is to install it by using Strimzi.

  1. Create a namespace for your Apache Kafka installation, like kafka:
    kubectl create namespace kafka
    
  2. Install the Strimzi operator, like:
    curl -L "https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.16.2/strimzi-cluster-operator-0.16.2.yaml" \
      | sed 's/namespace: .*/namespace: kafka/' \
      | kubectl -n kafka apply -f -
    
  3. Describe the size of your Apache Kafka installation in kafka.yaml, like:
    apiVersion: kafka.strimzi.io/v1beta1
    kind: Kafka
    metadata:
      name: my-cluster
    spec:
      kafka:
        version: 2.4.0
        replicas: 1
        listeners:
          plain: {}
          tls: {}
        config:
          offsets.topic.replication.factor: 1
          transaction.state.log.replication.factor: 1
          transaction.state.log.min.isr: 1
          log.message.format.version: "2.4"
        storage:
          type: ephemeral
      zookeeper:
        replicas: 3
        storage:
          type: ephemeral
      entityOperator:
        topicOperator: {}
        userOperator: {}
    
  4. Deploy the Apache Kafka cluster
    $ kubectl apply -n kafka -f kafka.yaml
    

This will install a small, non-production, cluster of Apache Kafka. To verify your installation, check if the pods for Strimzi are all up, in the kafka namespace:

$ kubectl get pods -n kafka
NAME                                          READY   STATUS    RESTARTS   AGE
my-cluster-entity-operator-65995cf856-ld2zp   3/3     Running   0          102s
my-cluster-kafka-0                            2/2     Running   0          2m8s
my-cluster-zookeeper-0                        2/2     Running   0          2m39s
my-cluster-zookeeper-1                        2/2     Running   0          2m49s
my-cluster-zookeeper-2                        2/2     Running   0          2m59s
strimzi-cluster-operator-77555d4b69-sbrt4     1/1     Running   0          3m14s

NOTE: For production ready installs check Strimzi.

Installation script

If you want to install the latest version of Strimzi, in just one step, we have a script for your convenience, which does exactly the same steps that are listed above:

$ ./kafka_setup.sh

Examples of Apache Kafka and Knative

A number of different examples, showing the KafkaSource, KafkaChannel and KafkaBinding can be found here: