# Installing Kafka Clusters with Helm Charts: A Step-by-Step Guide

Apache Kafka is the backbone of modern data streaming, and deploying it on Kubernetes ensures scalability and resilience. In this tutorial, we will set up a Kafka cluster in KRaft mode (without Zookeeper) using a custom Helm Chart.

By the end of this guide, you will have a running Kafka cluster defined as code, ready to handle your streaming data.

### Prerequisites

Before we dive in, make sure you have the following tools installed and configured:

1. Kubernetes Cluster: A running cluster (Minikube, Kind, or a cloud provider like GKE/EKS).
    
2. kubectl: The Kubernetes command-line tool.
    
3. Helm: The package manager for Kubernetes.
    

### Step 1: Installing Helm

If you haven't installed Helm yet, here is how you can do it on Linux/macOS.

For macOS (using Homebrew):

```bash
brew install helm
```

For Linux (using Script):

```bash
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
```

Verify the installation:

```bash
helm version
```

### Step 2: Initialize the Helm Chart

Let's create the directory structure for our chart. Run the following command to generate a boilerplate chart:

```bash
helm create kafka-chart
```

This creates a folder named `kafka-chart`. Since we want to build our own logic, clean up the default templates:

```bash
rm -rf kafka-chart/templates/*
rm kafka-chart/values.yaml
```

Now we have a clean slate to add our configuration files.

### Step 3: Configuration Files

We need to define our Chart metadata and default values.

1. Chart Definition (`Chart.yaml`)
    

Open `kafka-chart/Chart.yaml` and replace its content with the following to define our application info:

```yaml
apiVersion: v2
name: kafka-chart
description: A Helm chart for deploying Kafka with KRaft mode
type: application
version: 0.1.0
appVersion: "1.0"
```

2. Default Values (`values.yaml`)
    

Create a new `kafka-chart/values.yaml`. This file serves as the single source of truth for our configuration (replicas, image, storage, etc.).

```yaml
replicaCount: 3

service:
  name: kafka-svc
  port: 9092

image:
  repository: doughgle/kafka-kraft
  tag: latest
  pullPolicy: IfNotPresent

pdb:
  minAvailable: 2

storage:
  size: 1Gi

kafka:
  clusterId: "oh-sxaDRTcyAr6pFRbXyzA"
  replicationFactor: 3
  minInSyncReplicas: 2
  shareDir: /mnt/kafka

namespace: default
```

### Step 4: Creating Kubernetes Templates

Now, let's create the actual Kubernetes resources inside the `kafka-chart/templates/ directory`.

1. Headless Service (`templates/services.yaml`)
    

We use a Headless Service (`clusterIP: None`) because Kafka brokers need stable network identities.

```yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.service.name }}
  labels:
    app: kafka-app
spec:
  clusterIP: None
  ports:
    - name: '9092'
      port: {{ .Values.service.port }}
      protocol: TCP
      targetPort: {{ .Values.service.port }}
  selector:
    app: kafka-app
```

2. Pod Disruption Budget (`templates/pdb.yaml`)
    

To ensure high availability during voluntary disruptions (like node upgrades), we define a PDB.

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
spec:
  minAvailable: {{ .Values.pdb.minAvailable }}
  selector:
    matchLabels:
      app: kafka-app
```

3. StatefulSet (`templates/statefulset.yaml`)
    

The StatefulSet manages the deployment and scaling of the Kafka pods. It handles the storage volume claims and passes necessary environment variables for the KRaft mode.

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  labels:
    app: kafka-app
spec:
  serviceName: {{ .Values.service.name }}
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: kafka-app
  template:
    metadata:
      labels:
        app: kafka-app
    spec:
      containers:
        - name: kafka-container
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - containerPort: 9092
            - containerPort: 9093
          env:
            - name: REPLICAS
              value: "{{ .Values.replicaCount }}"
            - name: SERVICE
              value: "{{ .Values.service.name }}"
            - name: NAMESPACE
              value: "{{ .Values.namespace }}"
            - name: SHARE_DIR
              value: "{{ .Values.kafka.shareDir }}"
            - name: CLUSTER_ID
              value: "{{ .Values.kafka.clusterId }}"
            - name: DEFAULT_REPLICATION_FACTOR
              value: "{{ .Values.kafka.replicationFactor }}"
            - name: DEFAULT_MIN_INSYNC_REPLICAS
              value: "{{ .Values.kafka.minInSyncReplicas }}"
          volumeMounts:
            - name: data
              mountPath: {{ .Values.kafka.shareDir }}
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: {{ .Values.storage.size }}
```

### Step 5: Deploying the Chart

With all files in place, we can now install our Kafka cluster.

1. Dry Run (Optional):
    
    It's good practice to verify what will be generated before applying it.
    
    ```bash
    helm install kafka-release ./kafka-chart --dry-run --debug
    ```
    
2. Install the Chart:
    
    Run the following command to deploy:
    
    ```bash
    helm install kafka-release ./kafka-chart
    ```
    

### Step 6: Verification

Once installed, check the status of your pods:

```bash
kubectl get pods -w
```

You should see 3 pods (`kafka-0, kafka-1, kafka-2`) transitioning to the Running state.

To verify the service:

```bash
kubectl get svc
```

You have now successfully deployed a Kafka cluster using Helm! This setup uses the KRaft mode, removing the dependency on Zookeeper and simplifying the architecture.

Happy Coding! 🚀