# Ceph Storage Cluster — From Concepts to a Working Application A complete, hands-on reference covering Ceph architecture, a 3-node cluster deployment with `cephadm`, object (RGW/S3) and block (RBD) storage, and a containerized application that consumes both. **Environment used in this guide** * 3 nodes: `ceph1` (192.168.122.222), `ceph2` (192.168.122.121), `ceph3` (192.168.122.89) * OS: Ubuntu 24.04 LTS * Container runtime: Docker * Ceph release: Squid (19.2.3), deployed via `cephadm` * Each node has one raw disk (`/dev/vdb`, 15 GiB) dedicated to an OSD * * * ## Table of Contents 1. [Core Concepts](#1-core-concepts) 2. [Cluster Architecture](#2-cluster-architecture) 3. [The Data Path: Pool, PG, CRUSH](#3-the-data-path-pool-pg-crush) 4. [Data Protection](#4-data-protection) 5. [Cluster Deployment](#5-cluster-deployment) 6. [Object Storage (RGW / S3)](#6-object-storage-rgw--s3) 7. [Block Storage (RBD)](#7-block-storage-rbd) 8. [The Demo Application](#8-the-demo-application) 9. [Storage Write Flow](#9-storage-write-flow) 10. [Application Source Code](#10-application-source-code) 11. [Code Path Trace (request → persisted)](#11-code-path-trace-request--persisted) 12. [Command Reference](#12-command-reference) 13. [Troubleshooting Notes](#13-troubleshooting-notes) * * * ## 1\. Core Concepts Ceph is a distributed, decentralized storage system that solves the three inherent limitations of a single disk: * **Capacity** — a single disk has a ceiling; Ceph distributes data across many disks. * **Durability** — disks fail; Ceph keeps multiple copies on independent disks. * **Throughput** — a single disk serializes I/O; Ceph parallelizes across disks. Unlike traditional SAN/NAS systems, Ceph has **no central controller**. There is no single point of failure and no central bottleneck. This is achieved through RADOS and the CRUSH algorithm (below). ### RADOS At its lowest layer, Ceph stores everything as **objects** (a flat namespace of uniquely-named binary blobs with metadata). This layer is called **RADOS** (*Reliable Autonomic Distributed Object Store*). Every higher-level interface — block, file, and object gateway — is built on top of RADOS and ultimately translates its operations into RADOS object reads/writes. * * * ## 2\. Cluster Architecture A Ceph cluster is composed of daemons, each with a distinct role: | Daemon | Process | Role | Recommended count | | --- | --- | --- | --- | | Monitor | `ceph-mon` | Holds cluster maps (source of truth), authentication. Requires quorum. | 3 (odd) | | Manager | `ceph-mgr` | Metrics, dashboard, orchestration. Active/standby. | 2 | | OSD | `ceph-osd` | Stores object data; handles replication, recovery, rebalancing. | 3+ (one per disk) | | RGW | `ceph-radosgw` | S3/Swift-compatible object gateway. Optional. | as needed | | MDS | `ceph-mds` | CephFS metadata. Optional. | as needed | ### Quorum (why MONs are odd-numbered) Monitors must reach a **majority (quorum)** to agree on cluster state. With 3 MONs, majority is 2, tolerating 1 failure. An even count (e.g. 4) provides no extra fault tolerance over 3 and increases split-brain risk, so an **odd number** is always used. ### MON vs MGR redundancy * **MONs** run all-active because they must *vote* to agree (quorum). * **MGRs** run active/standby because the role is single-instance; the standby takes over on failure. Redundancy here is for *failover*, not *consensus*. Neither MON nor MGR sits in the data path — clients fetch maps once, then talk directly to OSDs. This is why they never become a bottleneck. * * * ## 3\. The Data Path: Pool, PG, CRUSH Ceph does **not** maintain a central lookup table of object locations. Instead it **computes** locations. This makes the cluster infinitely scalable (no central bottleneck) and self-adjusting. The write path has two computed stages: ```plaintext object name --(hash mod pg_num)--> PG --(CRUSH + cluster map)--> [OSD, OSD, OSD] ``` 1. **Object → PG:** the object name is hashed and taken modulo `pg_num` to select a Placement Group. PGs are a middle layer that groups millions of objects into a manageable number of buckets — recovery, replication, and rebalancing all operate at PG granularity. 2. **PG → OSDs:** CRUSH takes the PG id plus the cluster's CRUSH map (physical topology) and returns the list of OSDs that should hold the PG. CRUSH operates on **PGs, not individual objects**. ### CRUSH and failure domains Because the CRUSH map encodes physical topology (which OSD is in which host, rack, row), CRUSH can place replicas across **distinct failure domains**. Example: with `host` as the failure domain, a 3-replica object is placed on OSDs in three different hosts, so losing an entire host still leaves two valid copies. ### Primary OSD The first OSD in the CRUSH output is the **Primary**. Clients write to and read from the Primary; the Primary coordinates replication to the Secondaries and only acknowledges the write once all copies are persisted. CRUSH distributes the Primary role across OSDs to spread coordination load evenly. * * * ## 4\. Data Protection Protection is configured **per pool**, not cluster-wide. Two methods: | Method | Mechanism | Space efficiency | Recovery cost | Use case | | --- | --- | --- | --- | --- | | Replication | N full copies (`size`, default 3) | Low (~33% at size 3) | Low (plain copy) | Hot data, latency-sensitive | | Erasure Coding | k data + m parity chunks | High (e.g. ~67% at k=4,m=2) | High (math rebuild) | Cold/archival, large volumes | Key replication parameters: * `size` — number of replicas (default 3). * `min_size` — minimum replicas required to continue serving writes (default 2). Below this, the pool blocks writes to protect data integrity. * * * ## 5\. Cluster Deployment The general deployment pattern is: **bootstrap one node → add remaining nodes → add OSDs**. This holds for a 3-node lab or a 100-node cluster. ### 5.1 Install cephadm (on ceph1) ```bash apt update apt install -y cephadm ``` ### 5.2 Bootstrap the cluster (on ceph1) This creates the first MON and MGR. Note the `--image` override pointing to a mirror — the default `quay.io` registry may be unreachable in some networks. ```bash cephadm --docker --image quay.m.daocloud.io/ceph/ceph:v19.2.3 bootstrap \ --mon-ip 192.168.122.222 \ --cluster-network 192.168.122.0/24 \ --allow-fqdn-hostname ``` Bootstrap output includes the dashboard URL, admin user, and a one-time password. Save these. All daemons run as Docker containers. Set the mirror as the default base image so newly added nodes pull from it too: ```bash ceph config set mgr mgr/cephadm/container_image_base quay.m.daocloud.io/ceph/ceph ``` ### 5.3 Install the ceph CLI on the host (on ceph1) ```bash cephadm add-repo --release squid cephadm install ceph-common ceph -s # cluster status; expect HEALTH_WARN until OSDs exist ``` ### 5.4 Add the other nodes cephadm manages nodes over SSH using its own key (`/etc/ceph/ceph.pub`). Distribute the key (enter each node's root password when prompted): ```bash ssh-copy-id -f -i /etc/ceph/ceph.pub root@192.168.122.121 ssh-copy-id -f -i /etc/ceph/ceph.pub root@192.168.122.89 ``` > **Ubuntu 24.04 note:** the cluster SSH key is RSA (SHA-1), which recent OpenSSH rejects by default. On each new node, allow RSA-SHA2: > > ```bash > echo "PubkeyAcceptedAlgorithms +ssh-rsa,rsa-sha2-256,rsa-sha2-512" \ > > /etc/ssh/sshd_config.d/99-ceph-rsa.conf > systemctl restart ssh > ``` Add the hosts: ```bash ceph orch host add ceph2 192.168.122.121 ceph orch host add ceph3 192.168.122.89 ceph orch host ls ``` cephadm automatically distributes MONs to reach a 3-node quorum and adds a standby MGR. ### 5.5 (Optional) Remove the monitoring stack For a lab, the Prometheus/Grafana/Alertmanager/node-exporter stack is optional and its images may be blocked. Removing it keeps `HEALTH` clean and the nodes light: ```bash ceph orch rm grafana ceph orch rm prometheus ceph orch rm alertmanager ceph orch rm node-exporter ``` ### 5.6 Create OSDs Convert every available raw disk into an OSD: ```bash ceph orch apply osd --all-available-devices ``` This also sets a standing policy: any future raw disk is auto-consumed as an OSD. After the OSDs come up, the cluster reaches `HEALTH_OK`: ```bash ceph -s ceph osd tree # shows root -> host -> osd topology (the CRUSH map) ``` Expected state: `3 osds: 3 up, 3 in`, `mon: 3 daemons (quorum)`, `mgr: active + standby`. * * * ## 6\. Object Storage (RGW / S3) ### 6.1 Deploy the RGW daemon (on ceph1) RGW is a Ceph daemon; it runs **on the cluster nodes**, not on the application host. Applications are remote S3 clients. ```bash ceph orch apply rgw myrgw --placement="ceph1" --port=8000 ceph orch ps --daemon-type rgw # wait for STATUS = running ``` RGW auto-creates several internal pools (`.rgw.root`, `default.rgw.meta`, `default.rgw.log`, `default.rgw.buckets.data`, etc.). This is expected. ### 6.2 Create an S3 user (on ceph1) The user is a **machine identity** (access key + secret key), analogous to AWS IAM credentials. Create one per application; use meaningful UIDs. ```bash radosgw-admin user create --uid=myapp --display-name="My App" ``` Record the `access_key` and `secret_key` from the JSON output. The user's `op_mask` of `read, write, delete` already permits normal bucket/object CRUD. ### 6.3 Client model * **User/bucket ownership:** keys belong to the *user*, not a bucket. A user can own many buckets. * **Who does what:** admin creates the user (server side, `radosgw-admin`); the application creates buckets and reads/writes objects (client side, S3 protocol). * * * ## 7\. Block Storage (RBD) RBD exposes a virtual block device. Unlike RGW, RBD needs **no gateway daemon** — the client (kernel driver) talks directly to the OSDs after computing placement via CRUSH. ### 7.1 Create pool and image (on ceph1) ```bash ceph osd pool create rbd-pool 32 rbd pool init rbd-pool rbd create rbd-pool/myimage --size 1024 # 1 GiB rbd info rbd-pool/myimage ``` `rbd info` shows the image is split into 256 × 4 MiB objects. These objects are thin-provisioned — only written when data actually lands. ### 7.2 Map, format, mount ```bash rbd map rbd-pool/myimage # -> /dev/rbd0 mkfs.ext4 /dev/rbd0 mkdir -p /mnt/myrbd mount /dev/rbd0 /mnt/myrbd ``` The OS now treats `/dev/rbd0` as an ordinary disk. Any write is transparently split into RADOS objects and replicated across the three nodes. > **Note:** `map`/`mount` are not persistent across reboots by default. For persistence, configure `/etc/ceph/rbdmap` and `/etc/fstab`. ### 7.3 Verify replication Every RBD data object is replicated exactly like a `rados`\-written object: ```bash rados -p rbd-pool ls | head ceph osd map rbd-pool # -> up ([1,2,0], p1) = three OSDs on three hosts, Primary = osd.1 ``` * * * ## 8\. The Demo Application A Flask app (in Docker) that stores uploads via **both** paths, side by side, to make the difference concrete. ### Two paths compared | | S3 / RGW path | Disk / RBD path | | --- | --- | --- | | How the app connects | in **code** (boto3) | via a **mounted** folder | | App aware of Ceph? | yes (endpoint + keys) | no (just a folder) | | Gateway daemon | RGW (`radosgw`) | none | | Best for | files, objects, backups, sharing | databases, app disks, volumes | Key insight: in the RBD path the app has **no Ceph-specific code** — it writes to `/data` like any folder. The Docker volume mount `-/mnt/myrbd:/data` is what places that folder on Ceph. RBD attaches at the *system* level; S3 attaches at the *code* level. ### Configuration (environment variables) Credentials are injected at runtime, never hardcoded: ```yaml environment: S3_ENDPOINT: "http://192.168.122.222:8000" S3_ACCESS_KEY: "" S3_SECRET_KEY: "" S3_BUCKET: "myapp-bucket" DISK_PATH: "/data" volumes: - /mnt/myrbd:/data ``` ### Run ```bash cd ceph-storage-app docker compose up --build -d docker compose ps # open http://192.168.122.222:5000 ``` ### Verify (from the cluster) ```bash # S3 objects landed in RADOS: rados -p default.rgw.buckets.data ls # RBD files landed on the mounted device: ls -lh /mnt/myrbd ``` * * * ## 9\. Storage Write Flow ### S3 path (upload → persisted) 1. **Browser** — user submits the upload form (HTTP to app on :5000). 2. **Flask app** — receives the file; boto3 wraps it in the S3 protocol with the access key and sends it to RGW. 3. **RGW (:8000)** — authenticates the key, converts the file into RADOS object(s). 4. **Placement computed** — object name is hashed to a PG; CRUSH maps the PG to three OSDs. No central lookup. 5. **Primary OSD** — writes replica 1. 6. **Replication** — the Primary copies to the two Secondary OSDs (because the pool has `size=3`), on three distinct hosts. 7. **Ack** — the Primary waits until all three replicas are persisted, then acknowledges. 8. **Success** — the ack propagates back through RGW → app → browser. ### RBD path (difference) The RBD path **skips steps 1–3** (no browser round-trip semantics, no RGW, no boto3). The app writes to `/data`; the RBD kernel driver computes placement (step 4) directly and writes to the OSDs. **From step 4 onward, both paths are identical** — same hash, same CRUSH, same 3-replica placement. This is the practical proof that "everything is RADOS underneath." | Stage | S3 path | RBD path | | --- | --- | --- | | App write mechanism | boto3 code | ordinary file write | | Gateway daemon | RGW | none | | Hash → PG → CRUSH → 3 replicas | identical | identical | * * * ## 10\. Application Source Code Complete, runnable source of the demo app. Directory layout: ```plaintext ceph-storage-app/ ├── app.py # Flask app, both storage backends ├── templates/ │ └── index.html # upload UI ├── requirements.txt # flask + boto3 ├── Dockerfile # container image └── docker-compose.yml # env vars (S3 creds) + RBD volume mount ``` ### 10.1 `app.py` ```python import os import io from datetime import datetime from flask import Flask, request, render_template, redirect, url_for, send_file, flash import boto3 from botocore.client import Config app = Flask(__name__) app.secret_key = "ceph-demo-secret" # --- Path 1: S3 / RGW --- S3_ENDPOINT = os.environ.get("S3_ENDPOINT", "http://192.168.122.222:8000") S3_ACCESS_KEY = os.environ.get("S3_ACCESS_KEY", "") S3_SECRET_KEY = os.environ.get("S3_SECRET_KEY", "") S3_BUCKET = os.environ.get("S3_BUCKET", "myapp-bucket") # --- Path 2: local disk (backed by RBD) --- DISK_PATH = os.environ.get("DISK_PATH", "/data") def get_s3_client(): """Build an S3 client pointed at RGW (endpoint_url), not AWS.""" return boto3.client( "s3", endpoint_url=S3_ENDPOINT, aws_access_key_id=S3_ACCESS_KEY, aws_secret_access_key=S3_SECRET_KEY, config=Config(signature_version="s3v4"), region_name="us-east-1", ) def ensure_bucket(): """Create the bucket if it does not exist (the app does this itself).""" s3 = get_s3_client() try: existing = [b["Name"] for b in s3.list_buckets().get("Buckets", [])] if S3_BUCKET not in existing: s3.create_bucket(Bucket=S3_BUCKET) except Exception as e: print(f"[warn] could not ensure bucket: {e}") @app.route("/") def index(): """Home page: list files from both backends.""" s3_files, s3_error = [], None try: s3 = get_s3_client() resp = s3.list_objects_v2(Bucket=S3_BUCKET) for obj in resp.get("Contents", []): s3_files.append({ "name": obj["Key"], "size": obj["Size"], "modified": obj["LastModified"].strftime("%Y-%m-%d %H:%M"), }) except Exception as e: s3_error = str(e) disk_files, disk_error = [], None try: os.makedirs(DISK_PATH, exist_ok=True) for name in sorted(os.listdir(DISK_PATH)): full = os.path.join(DISK_PATH, name) if os.path.isfile(full): st = os.stat(full) disk_files.append({ "name": name, "size": st.st_size, "modified": datetime.fromtimestamp(st.st_mtime).strftime("%Y-%m-%d %H:%M"), }) except Exception as e: disk_error = str(e) return render_template( "index.html", s3_files=s3_files, s3_error=s3_error, s3_bucket=S3_BUCKET, s3_endpoint=S3_ENDPOINT, disk_files=disk_files, disk_error=disk_error, disk_path=DISK_PATH, ) @app.route("/upload", methods=["POST"]) def upload(): """Store the uploaded file in S3 or on disk, based on user choice.""" backend = request.form.get("backend") f = request.files.get("file") if not f or f.filename == "": flash("No file selected.") return redirect(url_for("index")) if backend == "s3": try: ensure_bucket() s3 = get_s3_client() s3.upload_fileobj(f, S3_BUCKET, f.filename) flash(f"File '{f.filename}' stored in S3 (RGW).") except Exception as e: flash(f"S3 upload error: {e}") elif backend == "disk": try: os.makedirs(DISK_PATH, exist_ok=True) f.save(os.path.join(DISK_PATH, f.filename)) flash(f"File '{f.filename}' stored on disk (RBD).") except Exception as e: flash(f"Disk write error: {e}") return redirect(url_for("index")) @app.route("/download//") def download(backend, name): """Download a file from the selected backend.""" if backend == "s3": s3 = get_s3_client() buf = io.BytesIO() s3.download_fileobj(S3_BUCKET, name, buf) buf.seek(0) return send_file(buf, as_attachment=True, download_name=name) elif backend == "disk": return send_file(os.path.join(DISK_PATH, name), as_attachment=True, download_name=name) return redirect(url_for("index")) @app.route("/health") def health(): return {"status": "ok"}, 200 if __name__ == "__main__": app.run(host="0.0.0.0", port=5000, debug=True) ``` ### 10.2 `templates/index.html` ```html Ceph Storage App

Ceph Storage App

{% with messages = get_flashed_messages() %} {% if messages %}{% for m in messages %}
{{ m }}
{% endfor %}{% endif %} {% endwith %}

S3 / RGW

bucket: {{ s3_bucket }} — endpoint: {{ s3_endpoint }}
{% if s3_error %}
error: {{ s3_error }}
{% endif %} {% for f in s3_files %} {% endfor %}
namesizemodified
{{ f.name }}{{ f.size }} B{{ f.modified }} download

Disk / RBD

mount: {{ disk_path }}
{% if disk_error %}
error: {{ disk_error }}
{% endif %} {% for f in disk_files %} {% endfor %}
namesizemodified
{{ f.name }}{{ f.size }} B{{ f.modified }} download
``` ### 10.3 `requirements.txt` ```plaintext flask==3.0.3 boto3==1.34.144 ``` ### 10.4 `Dockerfile` ```dockerfile FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app.py . COPY templates/ templates/ EXPOSE 5000 CMD ["python", "app.py"] ``` ### 10.5 `docker-compose.yml` ```yaml services: storage-app: build: . container_name: ceph-storage-app ports: - "5000:5000" environment: # --- S3 / RGW path --- S3_ENDPOINT: "http://192.168.122.222:8000" S3_ACCESS_KEY: "" S3_SECRET_KEY: "" S3_BUCKET: "myapp-bucket" # --- disk / RBD path --- DISK_PATH: "/data" volumes: # /data inside the container maps to the RBD mount on the host. - /mnt/myrbd:/data restart: unless-stopped ``` * * * ## 11\. Code Path Trace (request → persisted) This section follows the **exact code** executed for each storage path, from the HTTP request to the bytes landing on OSDs. ### 11.1 S3 path — line by line **Trigger:** browser POSTs the form to `/upload` with `backend=s3`. 1. `upload()` **handler runs** (`app.py`). * `backend = request.form.get("backend")` → `"s3"`. * `f = request.files.get("file")` → the uploaded file object (in memory). * Empty-file guard: if no file, flash a message and redirect. 2. `ensure_bucket()` is called. * Builds a client via `get_s3_client()`. * `s3.list_buckets()` → RGW returns existing buckets. * If `myapp-bucket` is missing, `s3.create_bucket(...)` → RGW creates it (this itself becomes RADOS metadata objects in the RGW pools). 3. `get_s3_client()` constructs the boto3 client. * `endpoint_url=S3_ENDPOINT` → points boto3 at RGW (`http://…:8000`), not AWS. * `aws_access_key_id` / `aws_secret_access_key` → the RGW user's keys. * `signature_version="s3v4"` → request signing scheme RGW expects. 4. `s3.upload_fileobj(f, S3_BUCKET, f.filename)` — the actual write. * boto3 streams the file body to RGW as an S3 `PUT` (multipart if large), signing the request with the secret key. * **Leaves the app process here.** Everything below is server-side Ceph. 5. **RGW (**`radosgw` **on :8000)** receives the PUT. * Verifies the signature/credentials (authentication). * Splits the object into RADOS objects and issues writes to the `default.rgw.buckets.data` pool. 6. **RADOS placement** (per object). * Object name → hash mod `pg_num` → PG. * CRUSH(PG, cluster map) → three OSDs on three distinct hosts. 7. **Replication.** * The write goes to the Primary OSD; the Primary replicates to two Secondaries. * Primary acks only after all three copies are persisted (`size=3`). 8. **Return trip.** * RGW returns `200 OK` to boto3 → `upload_fileobj` returns → the handler runs `flash("File '…' stored in S3 (RGW).")` and redirects to `/`. * `index()` then calls `s3.list_objects_v2(...)`, and the file appears in the S3 column. **Where to observe it on the cluster:** ```bash rados -p default.rgw.buckets.data ls # object name is: ._ ceph osd map default.rgw.buckets.data "" # -> up ([x,y,z]) ``` ### 11.2 RBD path — line by line **Trigger:** browser POSTs the form to `/upload` with `backend=disk`. 1. `upload()` **handler runs.** * `backend` → `"disk"`. * `os.makedirs(DISK_PATH, exist_ok=True)` ensures `/data` exists. 2. `f.save(os.path.join(DISK_PATH, f.filename))` — the actual write. * This is an ordinary POSIX file write to `/data/`. * **No boto3, no RGW, no Ceph-specific code in the app.** 3. **Docker volume mapping.** * `/data` in the container is bind-mounted to `/mnt/myrbd` on the host (`docker-compose.yml`: `- /mnt/myrbd:/data`). * So the write actually lands on the host's `/mnt/myrbd`. 4. **ext4 → RBD block device.** * `/mnt/myrbd` is an ext4 filesystem on `/dev/rbd0`. * The ext4 layer turns the file write into block writes on `/dev/rbd0`. 5. **RBD kernel driver → RADOS.** * The `krbd` driver maps block offsets to RADOS objects (`rbd_data..`) in `rbd-pool`. * For each such object: hash → PG → CRUSH → three OSDs (identical to step 6–7 of the S3 path). 6. **Persisted.** * Once the OSDs ack, the block write completes; ext4 and the app see success. * `index()` lists `/data` with `os.listdir`, and the file appears in the disk column. **Where to observe it on the cluster:** ```bash ls -lh /mnt/myrbd rados -p rbd-pool ls | grep rbd_data | head ceph osd map rbd-pool "" # -> up ([x,y,z]) ``` ### 11.3 The convergence point Both paths differ only in the **entry gateway** into RADOS: * S3 path enters through **RGW** (application-level, via boto3 code). * RBD path enters through the **krbd driver** (system-level, via a file write). From the moment an object reaches RADOS (hash → PG → CRUSH → 3 replicas), the two paths are **byte-for-byte identical** in how data is placed, replicated, and persisted. This is the concrete demonstration that every Ceph interface is a facade over the same RADOS object store. * * * ## 12\. Command Reference ### Status ```bash ceph -s # cluster summary (run this constantly) ceph health detail # explain any WARN/ERR ceph df # capacity by pool ceph osd tree # CRUSH topology: root -> host -> osd ceph osd df # usage + PG count per OSD ``` ### Orchestration ```bash ceph orch host ls ceph orch host add ceph orch ps [--daemon-type osd|mon|mgr|rgw] ceph orch ls ceph orch device ls [--refresh] ceph orch apply osd --all-available-devices ``` ### Pools ```bash ceph osd pool create ceph osd pool ls ceph osd pool get size ceph osd pool get min_size ceph osd pool application enable ``` ### Objects (rados / placement) ```bash rados -p put rados -p ls ceph osd map # shows PG and OSD placement for an object ``` ### RBD ```bash rbd pool init rbd create / --size rbd info / rbd map / rbd showmapped rbd du / ``` ### RGW / S3 admin ```bash ceph orch apply rgw --placement="" --port= radosgw-admin user create --uid= --display-name="" radosgw-admin user info --uid= radosgw-admin key create --uid= --key-type=s3 --gen-access-key --gen-secret ``` * * * ## 13\. Troubleshooting Notes `403 Forbidden` **pulling** `quay.io/ceph/ceph` Registry unreachable. Use a mirror via `--image quay.m.daocloud.io/ceph/ceph:` at bootstrap and set `mgr/cephadm/container_image_base` to the mirror. `ssh-copy-id` **/ cephadm cannot SSH to nodes (Ubuntu 24.04)** The RSA (SHA-1) cluster key is rejected. Add `PubkeyAcceptedAlgorithms +ssh-rsa,...` under `/etc/ssh/sshd_config.d/` on the target nodes and restart ssh. `HEALTH_WARN: OSD count 0 < osd_pool_default_size 3` No OSDs yet. Expected before `ceph orch apply osd`. `HEALTH_WARN: pool(s) do not have an application enabled` Run `ceph osd pool application enable `. `MON_CLOCK_SKEW: clock skew detected` Time drift between MONs. Re-sync (`systemctl restart systemd-timesyncd`) and, if needed, restart the MON (`ceph orch daemon restart mon.`). Clears after the next health check. **RBD not present after reboot** `map`/`mount` are not persistent by default. Re-run `rbd map` + `mount`, or configure `/etc/ceph/rbdmap` and `/etc/fstab`. **Dashboard/RGW not reachable from a laptop** The `192.168.122.x` addresses are private to the cluster network. Access from a host on that network, or use an SSH tunnel. * * * ## Summary Starting from three bare Ubuntu nodes, this guide built a fault-tolerant Ceph cluster (3-MON quorum, active/standby MGR, 3 OSDs), exposed both object (RGW/S3) and block (RBD) storage, and connected a containerized application that writes through both. Regardless of the interface used, all data converges to the same RADOS layer, is placed by CRUSH across three failure domains, and is stored as three replicas — delivering scalability, durability, and no single point of failure.