Environment used in this guide

• 3 nodes: ceph1 (192.168.122.222), ceph2 (192.168.122.121), ceph3 (192.168.122.89)

• OS: Ubuntu 24.04 LTS

• Container runtime: Docker

• Ceph release: Squid (19.2.3), deployed via cephadm

• Each node has one raw disk (/dev/vdb, 15 GiB) dedicated to an OSD

Table of Contents

1. Core Concepts

2. Cluster Architecture

3. The Data Path: Pool, PG, CRUSH

4. Data Protection

5. Cluster Deployment

6. Object Storage (RGW / S3)

7. Block Storage (RBD)

8. The Demo Application

9. Storage Write Flow

10. Application Source Code

11. Code Path Trace (request → persisted)

12. Command Reference

13. Troubleshooting Notes

1. Core Concepts

Ceph is a distributed, decentralized storage system that solves the three inherent limitations of a single disk:

• Capacity — a single disk has a ceiling; Ceph distributes data across many disks.

• Durability — disks fail; Ceph keeps multiple copies on independent disks.

• Throughput — a single disk serializes I/O; Ceph parallelizes across disks.

Unlike traditional SAN/NAS systems, Ceph has no central controller. There is no single point of failure and no central bottleneck. This is achieved through RADOS and the CRUSH algorithm (below).

RADOS

At its lowest layer, Ceph stores everything as objects (a flat namespace of uniquely-named binary blobs with metadata). This layer is called RADOS (Reliable Autonomic Distributed Object Store). Every higher-level interface — block, file, and object gateway — is built on top of RADOS and ultimately translates its operations into RADOS object reads/writes.

2. Cluster Architecture

A Ceph cluster is composed of daemons, each with a distinct role:

Quorum (why MONs are odd-numbered)

Monitors must reach a majority (quorum) to agree on cluster state. With 3 MONs, majority is 2, tolerating 1 failure. An even count (e.g. 4) provides no extra fault tolerance over 3 and increases split-brain risk, so an odd number is always used.

MON vs MGR redundancy

• MONs run all-active because they must vote to agree (quorum).

• MGRs run active/standby because the role is single-instance; the standby takes over on failure. Redundancy here is for failover, not consensus.

Neither MON nor MGR sits in the data path — clients fetch maps once, then talk directly to OSDs. This is why they never become a bottleneck.

3. The Data Path: Pool, PG, CRUSH

Ceph does not maintain a central lookup table of object locations. Instead it computes locations. This makes the cluster infinitely scalable (no central bottleneck) and self-adjusting.

The write path has two computed stages:

object name --(hash mod pg_num)--> PG --(CRUSH + cluster map)--> [OSD, OSD, OSD]

1. Object → PG: the object name is hashed and taken modulo pg_num to select a Placement Group. PGs are a middle layer that groups millions of objects into a manageable number of buckets — recovery, replication, and rebalancing all operate at PG granularity.

2. PG → OSDs: CRUSH takes the PG id plus the cluster's CRUSH map (physical topology) and returns the list of OSDs that should hold the PG. CRUSH operates on PGs, not individual objects.

CRUSH and failure domains

Because the CRUSH map encodes physical topology (which OSD is in which host, rack, row), CRUSH can place replicas across distinct failure domains. Example: with host as the failure domain, a 3-replica object is placed on OSDs in three different hosts, so losing an entire host still leaves two valid copies.

Primary OSD

The first OSD in the CRUSH output is the Primary. Clients write to and read from the Primary; the Primary coordinates replication to the Secondaries and only acknowledges the write once all copies are persisted. CRUSH distributes the Primary role across OSDs to spread coordination load evenly.

4. Data Protection

Protection is configured per pool, not cluster-wide. Two methods:

Key replication parameters:

• size — number of replicas (default 3).

• min_size — minimum replicas required to continue serving writes (default 2). Below this, the pool blocks writes to protect data integrity.

5. Cluster Deployment

The general deployment pattern is: bootstrap one node → add remaining nodes → add OSDs. This holds for a 3-node lab or a 100-node cluster.

5.1 Install cephadm (on ceph1)

apt update
apt install -y cephadm

5.2 Bootstrap the cluster (on ceph1)

This creates the first MON and MGR. Note the --image override pointing to a mirror — the default quay.io registry may be unreachable in some networks.

cephadm --docker --image quay.m.daocloud.io/ceph/ceph:v19.2.3 bootstrap \
  --mon-ip 192.168.122.222 \
  --cluster-network 192.168.122.0/24 \
  --allow-fqdn-hostname

Bootstrap output includes the dashboard URL, admin user, and a one-time password. Save these. All daemons run as Docker containers.

Set the mirror as the default base image so newly added nodes pull from it too:

ceph config set mgr mgr/cephadm/container_image_base quay.m.daocloud.io/ceph/ceph

5.3 Install the ceph CLI on the host (on ceph1)

cephadm add-repo --release squid
cephadm install ceph-common
ceph -s   # cluster status; expect HEALTH_WARN until OSDs exist

5.4 Add the other nodes

cephadm manages nodes over SSH using its own key (/etc/ceph/ceph.pub).

Distribute the key (enter each node's root password when prompted):

ssh-copy-id -f -i /etc/ceph/ceph.pub root@192.168.122.121
ssh-copy-id -f -i /etc/ceph/ceph.pub root@192.168.122.89

Ubuntu 24.04 note: the cluster SSH key is RSA (SHA-1), which recent OpenSSH rejects by default. On each new node, allow RSA-SHA2:

echo "PubkeyAcceptedAlgorithms +ssh-rsa,rsa-sha2-256,rsa-sha2-512" \
  > /etc/ssh/sshd_config.d/99-ceph-rsa.conf
systemctl restart ssh

Add the hosts:

ceph orch host add ceph2 192.168.122.121
ceph orch host add ceph3 192.168.122.89
ceph orch host ls

cephadm automatically distributes MONs to reach a 3-node quorum and adds a standby MGR.

5.5 (Optional) Remove the monitoring stack

For a lab, the Prometheus/Grafana/Alertmanager/node-exporter stack is optional and its images may be blocked. Removing it keeps HEALTH clean and the nodes light:

ceph orch rm grafana
ceph orch rm prometheus
ceph orch rm alertmanager
ceph orch rm node-exporter

5.6 Create OSDs

Convert every available raw disk into an OSD:

ceph orch apply osd --all-available-devices

This also sets a standing policy: any future raw disk is auto-consumed as an OSD. After the OSDs come up, the cluster reaches HEALTH_OK:

ceph -s
ceph osd tree     # shows root -> host -> osd topology (the CRUSH map)

Expected state: 3 osds: 3 up, 3 in, mon: 3 daemons (quorum), mgr: active + standby.

6. Object Storage (RGW / S3)

6.1 Deploy the RGW daemon (on ceph1)

RGW is a Ceph daemon; it runs on the cluster nodes, not on the application host. Applications are remote S3 clients.

ceph orch apply rgw myrgw --placement="ceph1" --port=8000
ceph orch ps --daemon-type rgw   # wait for STATUS = running

RGW auto-creates several internal pools (.rgw.root, default.rgw.meta, default.rgw.log, default.rgw.buckets.data, etc.). This is expected.

6.2 Create an S3 user (on ceph1)

The user is a machine identity (access key + secret key), analogous to AWS IAM credentials. Create one per application; use meaningful UIDs.

radosgw-admin user create --uid=myapp --display-name="My App"

Record the access_key and secret_key from the JSON output. The user's op_mask of read, write, delete already permits normal bucket/object CRUD.

6.3 Client model

• User/bucket ownership: keys belong to the user, not a bucket. A user can own many buckets.

• Who does what: admin creates the user (server side, radosgw-admin); the application creates buckets and reads/writes objects (client side, S3 protocol).

7. Block Storage (RBD)

RBD exposes a virtual block device. Unlike RGW, RBD needs no gateway daemon — the client (kernel driver) talks directly to the OSDs after computing placement via CRUSH.

7.1 Create pool and image (on ceph1)

ceph osd pool create rbd-pool 32
rbd pool init rbd-pool
rbd create rbd-pool/myimage --size 1024   # 1 GiB
rbd info rbd-pool/myimage

rbd info shows the image is split into 256 × 4 MiB objects. These objects are thin-provisioned — only written when data actually lands.

7.2 Map, format, mount

rbd map rbd-pool/myimage      # -> /dev/rbd0
mkfs.ext4 /dev/rbd0
mkdir -p /mnt/myrbd
mount /dev/rbd0 /mnt/myrbd

The OS now treats /dev/rbd0 as an ordinary disk. Any write is transparently split into RADOS objects and replicated across the three nodes.

Note: map/mount are not persistent across reboots by default. For persistence, configure /etc/ceph/rbdmap and /etc/fstab.

7.3 Verify replication

Every RBD data object is replicated exactly like a rados-written object:

rados -p rbd-pool ls | head
ceph osd map rbd-pool <object-name>
# -> up ([1,2,0], p1)  = three OSDs on three hosts, Primary = osd.1

8. The Demo Application

A Flask app (in Docker) that stores uploads via both paths, side by side, to make the difference concrete.

Two paths compared

Key insight: in the RBD path the app has no Ceph-specific code — it writes to /data like any folder. The Docker volume mount -/mnt/myrbd:/data is what places that folder on Ceph. RBD attaches at the system level; S3 attaches at the code level.

Configuration (environment variables)

Credentials are injected at runtime, never hardcoded:

environment:
  S3_ENDPOINT: "http://192.168.122.222:8000"
  S3_ACCESS_KEY: "<access_key>"
  S3_SECRET_KEY: "<secret_key>"
  S3_BUCKET: "myapp-bucket"
  DISK_PATH: "/data"
volumes:
  - /mnt/myrbd:/data

Run

cd ceph-storage-app
docker compose up --build -d
docker compose ps
# open http://192.168.122.222:5000

Verify (from the cluster)

# S3 objects landed in RADOS:
rados -p default.rgw.buckets.data ls
# RBD files landed on the mounted device:
ls -lh /mnt/myrbd

9. Storage Write Flow

S3 path (upload → persisted)

1. Browser — user submits the upload form (HTTP to app on :5000).

2. Flask app — receives the file; boto3 wraps it in the S3 protocol with the access key and sends it to RGW.

3. RGW (:8000) — authenticates the key, converts the file into RADOS object(s).

4. Placement computed — object name is hashed to a PG; CRUSH maps the PG to three OSDs. No central lookup.

5. Primary OSD — writes replica 1.

6. Replication — the Primary copies to the two Secondary OSDs (because the pool has size=3), on three distinct hosts.

7. Ack — the Primary waits until all three replicas are persisted, then acknowledges.

8. Success — the ack propagates back through RGW → app → browser.

RBD path (difference)

The RBD path skips steps 1–3 (no browser round-trip semantics, no RGW, no boto3). The app writes to /data; the RBD kernel driver computes placement (step 4) directly and writes to the OSDs. From step 4 onward, both paths are identical — same hash, same CRUSH, same 3-replica placement. This is the practical proof that "everything is RADOS underneath."

10. Application Source Code

Complete, runnable source of the demo app. Directory layout:

ceph-storage-app/
├── app.py                 # Flask app, both storage backends
├── templates/
│   └── index.html         # upload UI
├── requirements.txt       # flask + boto3
├── Dockerfile             # container image
└── docker-compose.yml     # env vars (S3 creds) + RBD volume mount

10.1 app.py

import os
import io
from datetime import datetime
from flask import Flask, request, render_template, redirect, url_for, send_file, flash
import boto3
from botocore.client import Config

app = Flask(__name__)
app.secret_key = "ceph-demo-secret"

# --- Path 1: S3 / RGW ---
S3_ENDPOINT = os.environ.get("S3_ENDPOINT", "http://192.168.122.222:8000")
S3_ACCESS_KEY = os.environ.get("S3_ACCESS_KEY", "")
S3_SECRET_KEY = os.environ.get("S3_SECRET_KEY", "")
S3_BUCKET = os.environ.get("S3_BUCKET", "myapp-bucket")

# --- Path 2: local disk (backed by RBD) ---
DISK_PATH = os.environ.get("DISK_PATH", "/data")


def get_s3_client():
    """Build an S3 client pointed at RGW (endpoint_url), not AWS."""
    return boto3.client(
        "s3",
        endpoint_url=S3_ENDPOINT,
        aws_access_key_id=S3_ACCESS_KEY,
        aws_secret_access_key=S3_SECRET_KEY,
        config=Config(signature_version="s3v4"),
        region_name="us-east-1",
    )


def ensure_bucket():
    """Create the bucket if it does not exist (the app does this itself)."""
    s3 = get_s3_client()
    try:
        existing = [b["Name"] for b in s3.list_buckets().get("Buckets", [])]
        if S3_BUCKET not in existing:
            s3.create_bucket(Bucket=S3_BUCKET)
    except Exception as e:
        print(f"[warn] could not ensure bucket: {e}")


@app.route("/")
def index():
    """Home page: list files from both backends."""
    s3_files, s3_error = [], None
    try:
        s3 = get_s3_client()
        resp = s3.list_objects_v2(Bucket=S3_BUCKET)
        for obj in resp.get("Contents", []):
            s3_files.append({
                "name": obj["Key"],
                "size": obj["Size"],
                "modified": obj["LastModified"].strftime("%Y-%m-%d %H:%M"),
            })
    except Exception as e:
        s3_error = str(e)

    disk_files, disk_error = [], None
    try:
        os.makedirs(DISK_PATH, exist_ok=True)
        for name in sorted(os.listdir(DISK_PATH)):
            full = os.path.join(DISK_PATH, name)
            if os.path.isfile(full):
                st = os.stat(full)
                disk_files.append({
                    "name": name,
                    "size": st.st_size,
                    "modified": datetime.fromtimestamp(st.st_mtime).strftime("%Y-%m-%d %H:%M"),
                })
    except Exception as e:
        disk_error = str(e)

    return render_template(
        "index.html",
        s3_files=s3_files, s3_error=s3_error, s3_bucket=S3_BUCKET, s3_endpoint=S3_ENDPOINT,
        disk_files=disk_files, disk_error=disk_error, disk_path=DISK_PATH,
    )


@app.route("/upload", methods=["POST"])
def upload():
    """Store the uploaded file in S3 or on disk, based on user choice."""
    backend = request.form.get("backend")
    f = request.files.get("file")
    if not f or f.filename == "":
        flash("No file selected.")
        return redirect(url_for("index"))

    if backend == "s3":
        try:
            ensure_bucket()
            s3 = get_s3_client()
            s3.upload_fileobj(f, S3_BUCKET, f.filename)
            flash(f"File '{f.filename}' stored in S3 (RGW).")
        except Exception as e:
            flash(f"S3 upload error: {e}")
    elif backend == "disk":
        try:
            os.makedirs(DISK_PATH, exist_ok=True)
            f.save(os.path.join(DISK_PATH, f.filename))
            flash(f"File '{f.filename}' stored on disk (RBD).")
        except Exception as e:
            flash(f"Disk write error: {e}")
    return redirect(url_for("index"))


@app.route("/download/<backend>/<path:name>")
def download(backend, name):
    """Download a file from the selected backend."""
    if backend == "s3":
        s3 = get_s3_client()
        buf = io.BytesIO()
        s3.download_fileobj(S3_BUCKET, name, buf)
        buf.seek(0)
        return send_file(buf, as_attachment=True, download_name=name)
    elif backend == "disk":
        return send_file(os.path.join(DISK_PATH, name), as_attachment=True, download_name=name)
    return redirect(url_for("index"))


@app.route("/health")
def health():
    return {"status": "ok"}, 200


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000, debug=True)

10.2 templates/index.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Ceph Storage App</title>
  <style>
    body { font-family: sans-serif; max-width: 900px; margin: 30px auto; padding: 0 16px; }
    .cols { display: flex; gap: 20px; flex-wrap: wrap; }
    .col { flex: 1; min-width: 320px; border: 1px solid #ddd; border-radius: 10px; padding: 16px; }
    table { width: 100%; border-collapse: collapse; font-size: 13px; }
    th, td { text-align: left; padding: 6px; border-bottom: 1px solid #eee; }
    .upload-box { border: 1px solid #ddd; border-radius: 10px; padding: 16px; margin-bottom: 20px; }
    button { background: #185FA5; color: #fff; border: none; padding: 8px 16px; border-radius: 6px; cursor: pointer; }
    .flash { background: #FAEEDA; border: 1px solid #EF9F27; padding: 10px; border-radius: 6px; margin-bottom: 16px; }
  </style>
</head>
<body>
  <h1>Ceph Storage App</h1>
  {% with messages = get_flashed_messages() %}
    {% if messages %}{% for m in messages %}<div class="flash">{{ m }}</div>{% endfor %}{% endif %}
  {% endwith %}

  <div class="upload-box">
    <form action="/upload" method="post" enctype="multipart/form-data">
      <input type="file" name="file" required>
      <div>
        <label><input type="radio" name="backend" value="s3" checked> Store in S3 (RGW)</label>
        <label><input type="radio" name="backend" value="disk"> Store on disk (RBD)</label>
      </div>
      <button type="submit">Upload</button>
    </form>
  </div>

  <div class="cols">
    <div class="col">
      <h2>S3 / RGW</h2>
      <div>bucket: {{ s3_bucket }} — endpoint: {{ s3_endpoint }}</div>
      {% if s3_error %}<div>error: {{ s3_error }}</div>{% endif %}
      <table>
        <tr><th>name</th><th>size</th><th>modified</th><th></th></tr>
        {% for f in s3_files %}
        <tr><td>{{ f.name }}</td><td>{{ f.size }} B</td><td>{{ f.modified }}</td>
        <td><a href="/download/s3/{{ f.name }}">download</a></td></tr>
        {% endfor %}
      </table>
    </div>
    <div class="col">
      <h2>Disk / RBD</h2>
      <div>mount: {{ disk_path }}</div>
      {% if disk_error %}<div>error: {{ disk_error }}</div>{% endif %}
      <table>
        <tr><th>name</th><th>size</th><th>modified</th><th></th></tr>
        {% for f in disk_files %}
        <tr><td>{{ f.name }}</td><td>{{ f.size }} B</td><td>{{ f.modified }}</td>
        <td><a href="/download/disk/{{ f.name }}">download</a></td></tr>
        {% endfor %}
      </table>
    </div>
  </div>
</body>
</html>

10.3 requirements.txt

flask==3.0.3
boto3==1.34.144

10.4 Dockerfile

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .
COPY templates/ templates/

EXPOSE 5000

CMD ["python", "app.py"]

10.5 docker-compose.yml

services:
  storage-app:
    build: .
    container_name: ceph-storage-app
    ports:
      - "5000:5000"
    environment:
      # --- S3 / RGW path ---
      S3_ENDPOINT: "http://192.168.122.222:8000"
      S3_ACCESS_KEY: "<access_key>"
      S3_SECRET_KEY: "<secret_key>"
      S3_BUCKET: "myapp-bucket"
      # --- disk / RBD path ---
      DISK_PATH: "/data"
    volumes:
      # /data inside the container maps to the RBD mount on the host.
      - /mnt/myrbd:/data
    restart: unless-stopped

11. Code Path Trace (request → persisted)

This section follows the exact code executed for each storage path, from the HTTP request to the bytes landing on OSDs.

11.1 S3 path — line by line

Trigger: browser POSTs the form to /upload with backend=s3.

1. upload() handler runs (app.py).

   • backend = request.form.get("backend") → "s3".

   • f = request.files.get("file") → the uploaded file object (in memory).

   • Empty-file guard: if no file, flash a message and redirect.

2. ensure_bucket() is called.

   • Builds a client via get_s3_client().

   • s3.list_buckets() → RGW returns existing buckets.

   • If myapp-bucket is missing, s3.create_bucket(...) → RGW creates it (this itself becomes RADOS metadata objects in the RGW pools).

3. get_s3_client() constructs the boto3 client.

   • endpoint_url=S3_ENDPOINT → points boto3 at RGW (http://…:8000), not AWS.

   • aws_access_key_id / aws_secret_access_key → the RGW user's keys.

   • signature_version="s3v4" → request signing scheme RGW expects.

4. s3.upload_fileobj(f, S3_BUCKET, f.filename) — the actual write.

   • boto3 streams the file body to RGW as an S3 PUT (multipart if large), signing the request with the secret key.

   • Leaves the app process here. Everything below is server-side Ceph.

5. RGW (radosgw on :8000) receives the PUT.

   • Verifies the signature/credentials (authentication).

   • Splits the object into RADOS objects and issues writes to the default.rgw.buckets.data pool.

6. RADOS placement (per object).

   • Object name → hash mod pg_num → PG.

   • CRUSH(PG, cluster map) → three OSDs on three distinct hosts.

7. Replication.

   • The write goes to the Primary OSD; the Primary replicates to two Secondaries.

   • Primary acks only after all three copies are persisted (size=3).

8. Return trip.

   • RGW returns 200 OK to boto3 → upload_fileobj returns → the handler runs flash("File '…' stored in S3 (RGW).") and redirects to /.

   • index() then calls s3.list_objects_v2(...), and the file appears in the S3 column.

Where to observe it on the cluster:

rados -p default.rgw.buckets.data ls
# object name is: <bucket-id>.<shard>_<original-filename>
ceph osd map default.rgw.buckets.data "<object-name>"   # -> up ([x,y,z])

11.2 RBD path — line by line

Trigger: browser POSTs the form to /upload with backend=disk.

1. upload() handler runs.

   • backend → "disk".

   • os.makedirs(DISK_PATH, exist_ok=True) ensures /data exists.

2. f.save(os.path.join(DISK_PATH, f.filename)) — the actual write.

   • This is an ordinary POSIX file write to /data/<filename>.

   • No boto3, no RGW, no Ceph-specific code in the app.

3. Docker volume mapping.

   • /data in the container is bind-mounted to /mnt/myrbd on the host (docker-compose.yml: - /mnt/myrbd:/data).

   • So the write actually lands on the host's /mnt/myrbd.

4. ext4 → RBD block device.

   • /mnt/myrbd is an ext4 filesystem on /dev/rbd0.

   • The ext4 layer turns the file write into block writes on /dev/rbd0.

5. RBD kernel driver → RADOS.

   • The krbd driver maps block offsets to RADOS objects (rbd_data.<id>.<seq>) in rbd-pool.

   • For each such object: hash → PG → CRUSH → three OSDs (identical to step 6–7 of the S3 path).

6. Persisted.

   • Once the OSDs ack, the block write completes; ext4 and the app see success.

   • index() lists /data with os.listdir, and the file appears in the disk column.

Where to observe it on the cluster:

ls -lh /mnt/myrbd
rados -p rbd-pool ls | grep rbd_data | head
ceph osd map rbd-pool "<object-name>"   # -> up ([x,y,z])

11.3 The convergence point

Both paths differ only in the entry gateway into RADOS:

• S3 path enters through RGW (application-level, via boto3 code).

• RBD path enters through the krbd driver (system-level, via a file write).

From the moment an object reaches RADOS (hash → PG → CRUSH → 3 replicas), the two paths are byte-for-byte identical in how data is placed, replicated, and persisted. This is the concrete demonstration that every Ceph interface is a facade over the same RADOS object store.

12. Command Reference

Status

ceph -s                      # cluster summary (run this constantly)
ceph health detail           # explain any WARN/ERR
ceph df                      # capacity by pool
ceph osd tree                # CRUSH topology: root -> host -> osd
ceph osd df                  # usage + PG count per OSD

Orchestration

ceph orch host ls
ceph orch host add <name> <ip>
ceph orch ps [--daemon-type osd|mon|mgr|rgw]
ceph orch ls
ceph orch device ls [--refresh]
ceph orch apply osd --all-available-devices

Pools

ceph osd pool create <name> <pg_num>
ceph osd pool ls
ceph osd pool get <name> size
ceph osd pool get <name> min_size
ceph osd pool application enable <name> <rbd|rgw|cephfs|rados>

Objects (rados / placement)

rados -p <pool> put <obj> <file>
rados -p <pool> ls
ceph osd map <pool> <obj>    # shows PG and OSD placement for an object

RBD

rbd pool init <pool>
rbd create <pool>/<image> --size <MB>
rbd info <pool>/<image>
rbd map <pool>/<image>
rbd showmapped
rbd du <pool>/<image>

RGW / S3 admin

ceph orch apply rgw <name> --placement="<host>" --port=<port>
radosgw-admin user create --uid=<uid> --display-name="<name>"
radosgw-admin user info --uid=<uid>
radosgw-admin key create --uid=<uid> --key-type=s3 --gen-access-key --gen-secret

13. Troubleshooting Notes

403 Forbidden pulling quay.io/ceph/ceph Registry unreachable. Use a mirror via --image quay.m.daocloud.io/ceph/ceph:<tag> at bootstrap and set mgr/cephadm/container_image_base to the mirror.

ssh-copy-id / cephadm cannot SSH to nodes (Ubuntu 24.04) The RSA (SHA-1) cluster key is rejected. Add PubkeyAcceptedAlgorithms +ssh-rsa,... under /etc/ssh/sshd_config.d/ on the target nodes and restart ssh.

HEALTH_WARN: OSD count 0 < osd_pool_default_size 3 No OSDs yet. Expected before ceph orch apply osd.

HEALTH_WARN: pool(s) do not have an application enabled Run ceph osd pool application enable <pool> <rados|rbd|rgw|cephfs>.

MON_CLOCK_SKEW: clock skew detected Time drift between MONs. Re-sync (systemctl restart systemd-timesyncd) and, if needed, restart the MON (ceph orch daemon restart mon.<host>). Clears after the next health check.

RBD not present after reboot map/mount are not persistent by default. Re-run rbd map + mount, or configure /etc/ceph/rbdmap and /etc/fstab.

Dashboard/RGW not reachable from a laptop The 192.168.122.x addresses are private to the cluster network. Access from a host on that network, or use an SSH tunnel.

Summary

Starting from three bare Ubuntu nodes, this guide built a fault-tolerant Ceph cluster (3-MON quorum, active/standby MGR, 3 OSDs), exposed both object (RGW/S3) and block (RBD) storage, and connected a containerized application that writes through both. Regardless of the interface used, all data converges to the same RADOS layer, is placed by CRUSH across three failure domains, and is stored as three replicas — delivering scalability, durability, and no single point of failure.

By the end of this guide, you will have a running Kafka cluster defined as code, ready to handle your streaming data.

Prerequisites

Before we dive in, make sure you have the following tools installed and configured:

1. Kubernetes Cluster: A running cluster (Minikube, Kind, or a cloud provider like GKE/EKS).

2. kubectl: The Kubernetes command-line tool.

3. Helm: The package manager for Kubernetes.

Step 1: Installing Helm

If you haven't installed Helm yet, here is how you can do it on Linux/macOS.

For macOS (using Homebrew):

brew install helm

For Linux (using Script):

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Verify the installation:

helm version

Step 2: Initialize the Helm Chart

Let's create the directory structure for our chart. Run the following command to generate a boilerplate chart:

helm create kafka-chart

This creates a folder named kafka-chart. Since we want to build our own logic, clean up the default templates:

rm -rf kafka-chart/templates/*
rm kafka-chart/values.yaml

Now we have a clean slate to add our configuration files.

Step 3: Configuration Files

We need to define our Chart metadata and default values.

1. Chart Definition (Chart.yaml)

Open kafka-chart/Chart.yaml and replace its content with the following to define our application info:

apiVersion: v2
name: kafka-chart
description: A Helm chart for deploying Kafka with KRaft mode
type: application
version: 0.1.0
appVersion: "1.0"

2. Default Values (values.yaml)

Create a new kafka-chart/values.yaml. This file serves as the single source of truth for our configuration (replicas, image, storage, etc.).

replicaCount: 3

service:
  name: kafka-svc
  port: 9092

image:
  repository: doughgle/kafka-kraft
  tag: latest
  pullPolicy: IfNotPresent

pdb:
  minAvailable: 2

storage:
  size: 1Gi

kafka:
  clusterId: "oh-sxaDRTcyAr6pFRbXyzA"
  replicationFactor: 3
  minInSyncReplicas: 2
  shareDir: /mnt/kafka

namespace: default

Step 4: Creating Kubernetes Templates

Now, let's create the actual Kubernetes resources inside the kafka-chart/templates/ directory.

1. Headless Service (templates/services.yaml)

We use a Headless Service (clusterIP: None) because Kafka brokers need stable network identities.

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.service.name }}
  labels:
    app: kafka-app
spec:
  clusterIP: None
  ports:
    - name: '9092'
      port: {{ .Values.service.port }}
      protocol: TCP
      targetPort: {{ .Values.service.port }}
  selector:
    app: kafka-app

2. Pod Disruption Budget (templates/pdb.yaml)

To ensure high availability during voluntary disruptions (like node upgrades), we define a PDB.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
spec:
  minAvailable: {{ .Values.pdb.minAvailable }}
  selector:
    matchLabels:
      app: kafka-app

3. StatefulSet (templates/statefulset.yaml)

The StatefulSet manages the deployment and scaling of the Kafka pods. It handles the storage volume claims and passes necessary environment variables for the KRaft mode.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  labels:
    app: kafka-app
spec:
  serviceName: {{ .Values.service.name }}
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: kafka-app
  template:
    metadata:
      labels:
        app: kafka-app
    spec:
      containers:
        - name: kafka-container
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - containerPort: 9092
            - containerPort: 9093
          env:
            - name: REPLICAS
              value: "{{ .Values.replicaCount }}"
            - name: SERVICE
              value: "{{ .Values.service.name }}"
            - name: NAMESPACE
              value: "{{ .Values.namespace }}"
            - name: SHARE_DIR
              value: "{{ .Values.kafka.shareDir }}"
            - name: CLUSTER_ID
              value: "{{ .Values.kafka.clusterId }}"
            - name: DEFAULT_REPLICATION_FACTOR
              value: "{{ .Values.kafka.replicationFactor }}"
            - name: DEFAULT_MIN_INSYNC_REPLICAS
              value: "{{ .Values.kafka.minInSyncReplicas }}"
          volumeMounts:
            - name: data
              mountPath: {{ .Values.kafka.shareDir }}
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: {{ .Values.storage.size }}

Step 5: Deploying the Chart

With all files in place, we can now install our Kafka cluster.

1. Dry Run (Optional):

   It's good practice to verify what will be generated before applying it.

    helm install kafka-release ./kafka-chart --dry-run --debug
   

2. Install the Chart:

   Run the following command to deploy:

    helm install kafka-release ./kafka-chart
   

Step 6: Verification

Once installed, check the status of your pods:

kubectl get pods -w

You should see 3 pods (kafka-0, kafka-1, kafka-2) transitioning to the Running state.

To verify the service:

kubectl get svc

You have now successfully deployed a Kafka cluster using Helm! This setup uses the KRaft mode, removing the dependency on Zookeeper and simplifying the architecture.

Happy Coding! 🚀