# Highly Available Kubernetes Monitoring: Prometheus + VictoriaMetrics

A step-by-step guide to building a monitoring stack with **no single point of failure**, using Prometheus for metric collection and VictoriaMetrics for long-term storage.

* * *

## Table of Contents

*   [Why This Architecture](#why-this-architecture)
    
*   [Architecture Overview](#architecture-overview)
    
*   [Prerequisites](#prerequisites)
    
*   [Step 1 — Create the Namespace](#step-1--create-the-namespace)
    
*   [Step 2 — Deploy VictoriaMetrics (HA Pair)](#step-2--deploy-victoriametrics-ha-pair)
    
*   [Step 3 — Create a Read Service](#step-3--create-a-read-service)
    
*   [Step 4 — Deploy kube-prometheus-stack](#step-4--deploy-kube-prometheus-stack)
    
*   [Step 5 — Verify remote\_write](#step-5--verify-remote_write)
    
*   [Step 6 — Configure Grafana](#step-6--configure-grafana)
    
*   [Step 7 — Failure Testing](#step-7--failure-testing)
    
*   [Troubleshooting](#troubleshooting)
    
*   [Design Decisions](#design-decisions)
    

* * *

## Why This Architecture

Prometheus is excellent at scraping metrics, but it has two limitations that matter in production:

**It was never designed for long-term storage.** Prometheus stores data on local disk. If you need a year of history, the disk fills up, queries slow down, and you are one PVC failure away from losing everything.

**It has no built-in replication.** Unlike most databases, you cannot configure a primary and a replica. The official answer to high availability is "run two identical instances" — which creates a new problem: each instance has slightly different data, and gaps appear whenever one restarts.

This guide solves both problems:

| Problem | Solution |
| --- | --- |
| Scraping stops if Prometheus dies | Run 2 Prometheus replicas |
| Each replica has gaps and different timestamps | VictoriaMetrics deduplicates on query |
| No long-term retention | VictoriaMetrics stores 12+ months |
| Storage layer becomes the new SPOF | Run 2 VictoriaMetrics instances, write to both |
| Duplicate alerts from 2 Prometheus replicas | Alertmanager in clustered mode |

* * *

**Key insight:** each Prometheus replica writes to *both* VictoriaMetrics instances. This means both storage instances hold a complete, identical copy of the data. Losing either one changes nothing.

This is different from load-balancing the writes, which would split the data and leave each instance with half a dataset.

### Component Responsibilities

| Component | Role | Replicas |
| --- | --- | --- |
| Prometheus | Scrapes targets, evaluates alert rules, forwards to storage | 2 |
| VictoriaMetrics | Long-term time-series storage, query engine | 2 |
| Alertmanager | Groups, deduplicates and routes alert notifications | 2 (clustered) |
| Grafana | Visualization | 1 |
| node-exporter | Host-level metrics (CPU, RAM, disk, network) | 1 per node |
| kube-state-metrics | Kubernetes object state as metrics | 1 |

Prometheus keeps only a short local retention window (6 hours). It acts as a **collector and forwarder**, not as the system of record. All durable data lives in VictoriaMetrics.

* * *

## Prerequisites

*   A Kubernetes cluster (this guide was tested on v1.35 with 3 control-plane nodes)
    
*   `kubectl` configured and working
    
*   `helm` v3
    
*   A working StorageClass that supports `ReadWriteOnce`
    
*   At least ~4 GiB of memory available across the cluster for the monitoring stack
    

Verify your storage:

```bash
kubectl get storageclass
```

* * *

## Step 1 — Create the Namespace

```bash
kubectl create namespace monitoring
```

Everything in this guide lives in `monitoring`. Using a dedicated namespace makes RBAC, network policies and resource quotas far easier later.

* * *

## Step 2 — Deploy VictoriaMetrics (HA Pair)

Add the chart repository:

```bash
helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update
helm search repo vm/victoria-metrics-single --versions | head -3
```

Create the values file:

```yaml
# vm-values.yaml
server:
  replicaCount: 2

  retentionPeriod: 12

  persistentVolume:
    enabled: true
    size: 20Gi

  extraArgs:
    dedup.minScrapeInterval: 30s

  resources:
    requests:
      cpu: 100m
      memory: 512Mi
    limits:
      memory: 2Gi

  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: server
          topologyKey: kubernetes.io/hostname

  statefulSet:
    enabled: true
```

### What each setting does

`replicaCount: 2` — deploys two *independent* VictoriaMetrics instances. They do not talk to each other and do not replicate between themselves. Redundancy comes from Prometheus writing the same data to both.

`retentionPeriod: 12` — twelve months of retention. The unit is months. Adjust based on your compliance or billing requirements.

`dedup.minScrapeInterval: 30s` — when two Prometheus replicas send the same sample, VictoriaMetrics keeps only one at query time. This value **must match** your Prometheus `scrapeInterval`. If they disagree, deduplication silently fails and you get doubled series.

`podAntiAffinity` — forces the two instances onto different nodes. Without this, Kubernetes may schedule both on the same node, which defeats the entire purpose of running two.

`statefulSet.enabled: true` — gives each pod a stable DNS name. This is essential, because Prometheus needs to address each instance individually rather than through a load-balanced Service.

Install:

```bash
helm install victoria-metrics vm/victoria-metrics-single \
  -n monitoring \
  -f vm-values.yaml
```

Verify:

```bash
kubectl -n monitoring get pods -l app=server -o wide
kubectl -n monitoring get pvc
```

You should see two pods, `victoria-metrics-victoria-metrics-single-server-0` and `-1`, **on different nodes**, each with its own PVC.

### Record the stable DNS names

With a StatefulSet plus a headless Service, each pod is addressable at:

```plaintext
<pod-name>.<headless-service-name>.<namespace>.svc:8428
```

For this installation:

```plaintext
victoria-metrics-victoria-metrics-single-server-0.victoria-metrics-victoria-metrics-single-server.monitoring.svc:8428
victoria-metrics-victoria-metrics-single-server-1.victoria-metrics-victoria-metrics-single-server.monitoring.svc:8428
```

Confirm the exact names:

```bash
kubectl -n monitoring get svc
kubectl -n monitoring get pods -l app=server -o name
```

* * *

## Step 3 — Create a Read Service

The headless Service is correct for writes, where we address each pod individually. For **reads**, we want a single endpoint that load-balances across both instances.

Because both instances hold identical data, it does not matter which one answers a query.

```yaml
# vm-read-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vm-read
  namespace: monitoring
spec:
  type: ClusterIP
  selector:
    app: server
  ports:
    - name: http
      port: 8428
      targetPort: 8428
```

```bash
kubectl apply -f vm-read-service.yaml
```

Verify the selector matched both pods:

```bash
kubectl -n monitoring get endpoints vm-read
```

You should see two IP addresses. If you see zero, your label selector does not match — inspect the actual pod labels:

```bash
kubectl -n monitoring get pods --show-labels | grep victoria
```

> **Production note:** a plain Service round-robins between the two instances without health-aware failover for slow-but-alive backends. For stricter guarantees, place [vmauth](https://docs.victoriametrics.com/vmauth/) in front. It understands VictoriaMetrics semantics and can retry against the second backend when the first fails.

* * *

## Step 4 — Deploy kube-prometheus-stack

```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
```

Create the values file:

```yaml
# prom-values.yaml
prometheus:
  prometheusSpec:
    replicas: 2


    retention: 6h
    retentionSize: 8GB

    scrapeInterval: 30s
    evaluationInterval: 30s


    replicaExternalLabelName: "replica"
    prometheusExternalLabelName: "prometheus"

    remoteWrite:
      - url: http://victoria-metrics-victoria-metrics-single-server-0.victoria-metrics-victoria-metrics-single-server.monitoring.svc:8428/api/v1/write
        queueConfig:
          capacity: 10000
          maxShards: 30
          minShards: 1
          maxSamplesPerSend: 2000
          batchSendDeadline: 5s
      - url: http://victoria-metrics-victoria-metrics-single-server-1.victoria-metrics-victoria-metrics-single-server.monitoring.svc:8428/api/v1/write
        queueConfig:
          capacity: 10000
          maxShards: 30
          minShards: 1
          maxSamplesPerSend: 2000
          batchSendDeadline: 5s

 
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    probeSelectorNilUsesHelmValues: false

    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

    resources:
      requests:
        cpu: 200m
        memory: 1Gi
      limits:
        memory: 3Gi

    podAntiAffinity: "hard"

alertmanager:
  alertmanagerSpec:
    replicas: 2
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 2Gi
    resources:
      requests:
        cpu: 50m
        memory: 128Mi
      limits:
        memory: 256Mi
    podAntiAffinity: "hard"

grafana:
  enabled: true
  adminPassword: "CHANGE_ME"
  persistence:
    enabled: true
    size: 5Gi
  additionalDataSources:
    - name: VictoriaMetrics
      type: prometheus
      access: proxy
      url: http://vm-read.monitoring.svc:8428
      isDefault: true
      jsonData:
        timeInterval: 30s


kubeEtcd:
  enabled: false
```

### The critical settings explained

`replicas: 2` — two Prometheus instances scrape every target independently. If one dies, the other keeps collecting. No gap.

`retention: 6h` — this is deliberately short. Prometheus is a forwarder here, not a database. Keeping the local window small means each replica stays light on memory and disk.

`replicaExternalLabelName: "replica"` — attaches a label identifying which replica produced each sample. VictoriaMetrics strips this label during deduplication. Without it, deduplication cannot work, because the two series would look genuinely different.

**Two** `remoteWrite` **entries** — this is the heart of the HA design. Each Prometheus replica pushes every sample to *both* VictoriaMetrics instances. Combined with 2 Prometheus replicas, every sample takes four paths into storage. Any single failure on either side is invisible.

`queueConfig` — controls the in-memory buffer for remote writes. If a VictoriaMetrics instance becomes briefly unavailable, Prometheus buffers samples and replays them on reconnect. Increase `capacity` if you expect longer outages.

`serviceMonitorSelectorNilUsesHelmValues: false` — by default, Prometheus only discovers ServiceMonitors carrying this chart's release label. Setting this to `false` makes it pick up ServiceMonitors from any namespace, which is what you want for your own applications.

`podAntiAffinity: "hard"` — same reasoning as with VictoriaMetrics: the two replicas must not share a node.

`alertmanager.replicas: 2` — Alertmanager forms a gossip cluster automatically when replicas > 1. Cluster members coordinate so that each alert is delivered **once**, not once per replica. Skipping this is the most common mistake when making Prometheus HA: you end up with duplicate notifications for every alert.

Install:

```bash
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring \
  -f prom-values.yaml
```

This takes a few minutes — the chart installs a large set of CRDs.

Watch the rollout:

```bash
kubectl -n monitoring get pods -w
```

* * *

## Step 5 — Verify remote\_write

This is the step people skip, and it is the step that catches misconfiguration.

### 5.1 — Check from the Prometheus side

```bash
kubectl -n monitoring port-forward prometheus-kube-prometheus-stack-prometheus-0 9090:9090
```

Open `http://localhost:9090` and run:

```promql
prometheus_remote_storage_samples_total
```

You should see two series — one per remote\_write endpoint — with steadily increasing counters.

Now check for failures:

```promql
rate(prometheus_remote_storage_samples_failed_total[5m])
```

This should be `0`. Anything above zero means samples are being rejected; check the Prometheus logs.

Check the send queue is not backing up:

```promql
prometheus_remote_storage_pending_samples
```

A persistently growing value means VictoriaMetrics cannot keep up, or the network path is congested.

### 5.2 — Check both VictoriaMetrics instances independently

Query each instance directly and confirm both hold data.

```bash
kubectl -n monitoring port-forward \
  victoria-metrics-victoria-metrics-single-server-0 8428:8428
```

Open `http://localhost:8428/vmui` and run:

```promql
count(up)
```

Note the number. Stop the port-forward, then repeat for instance `-1` on a different local port:

```bash
kubectl -n monitoring port-forward \
  victoria-metrics-victoria-metrics-single-server-1 8429:8428
```

Open `http://localhost:8429/vmui` and run the same query.

**Both instances must return a similar count.** If one returns significantly fewer series, writes are not reaching it — re-check the second `remoteWrite` URL for typos.

### 5.3 — Confirm deduplication is active

Still in VMUI, run:

```promql
count by (replica) (up)
```

Two distinct `replica` values confirm both Prometheus instances are writing.

Then run:

```promql
count(up)
```

If deduplication is working, this returns roughly the number of scrape targets — not double it.

Confirm the flag actually reached the process:

```bash
kubectl -n monitoring exec victoria-metrics-victoria-metrics-single-server-0 -- \
  cat /proc/1/cmdline | tr '\0' '\n' | grep dedup
```

Expected output:

```plaintext
-dedup.minScrapeInterval=30s
```

If this prints nothing, the `extraArgs` block did not apply and you will be storing duplicate series.

* * *

## Step 6 — Configure Grafana

Retrieve the admin password:

```bash
kubectl -n monitoring get secret kube-prometheus-stack-grafana \
  -o jsonpath="{.data.admin-password}" | base64 -d; echo
```

Port-forward:

```bash
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
```

Open `http://localhost:3000`.

### Data sources

If you used the `additionalDataSources` block from Step 4, `VictoriaMetrics` is already configured and set as default. Verify under **Connections → Data sources**.

Note that VictoriaMetrics is registered with `type: prometheus`. This is correct — VictoriaMetrics implements the Prometheus query API, so Grafana treats it identically and all PromQL works unchanged.

### Keep the Prometheus data source

Do not delete the default Prometheus data source. It remains useful for:

*   Comparing against VictoriaMetrics when debugging
    
*   Querying the last few minutes with minimal latency
    
*   Bundled dashboards that reference it explicitly
    

Just make sure **VictoriaMetrics is the default**, so new dashboards use long-term storage.

### Defining data sources as code

Configuring data sources through the UI means losing them on reinstall. Defining them in `prom-values.yaml` — as above — keeps them reproducible and version-controlled.

* * *

## Step 7 — Failure Testing

A high-availability setup you have not tested is a high-availability setup you do not have.

### Test 1 — Kill one Prometheus replica

```bash
kubectl -n monitoring delete pod prometheus-kube-prometheus-stack-prometheus-0
```

While it restarts, query in Grafana:

```promql
up
```

Data should continue flowing, with no visible gap. Replica `1` covered the window.

### Test 2 — Kill one VictoriaMetrics instance

```bash
kubectl -n monitoring delete pod victoria-metrics-victoria-metrics-single-server-0
```

Grafana should continue serving queries through `vm-read`, which routes to instance `1`.

Meanwhile, Prometheus buffers writes destined for instance `0`. Watch the buffer:

```promql
prometheus_remote_storage_pending_samples
```

When the pod returns, the buffer drains and instance `0` catches up automatically.

### Test 3 — Verify alert deduplication

Trigger a test alert and confirm you receive **one** notification, not two.

Check that Alertmanager formed a cluster:

```bash
kubectl -n monitoring port-forward svc/kube-prometheus-stack-alertmanager 9093:9093
```

Open `http://localhost:9093/#/status`. Under **Cluster Status**, you should see two peers with status `ready`.

If it shows only one peer, the cluster did not form and you will receive duplicate alerts.

* * *

## Troubleshooting

### Grafana shows no data

Work backwards through the chain:

```bash
# 1. Is the read Service backed by pods?
kubectl -n monitoring get endpoints vm-read

# 2. Does VictoriaMetrics have data? (use VMUI)
kubectl -n monitoring port-forward victoria-metrics-victoria-metrics-single-server-0 8428:8428

# 3. Is Prometheus actually sending?
kubectl -n monitoring logs prometheus-kube-prometheus-stack-prometheus-0 -c prometheus | grep -i remote
```

### Series counts are doubled

Deduplication is not working. Check three things:

1.  `dedup.minScrapeInterval` matches Prometheus `scrapeInterval` exactly
    
2.  The flag reached the process (`/proc/1/cmdline` check above)
    
3.  `replicaExternalLabelName` is set in the Prometheus spec
    

### One VictoriaMetrics instance lags behind

Normal briefly after a restart while the buffer drains. If it persists:

```bash
kubectl -n monitoring logs prometheus-kube-prometheus-stack-prometheus-0 -c prometheus \
  | grep -i "remote_write"
```

Look for connection refused or timeout errors against that specific URL.

### Both VictoriaMetrics pods on the same node

Anti-affinity did not apply. Check the actual pod labels and correct the `matchLabels` selector:

```bash
kubectl -n monitoring get pods -l app=server -o wide --show-labels
```

### Duplicate alert notifications

Alertmanager did not form a cluster. Check the status page (Test 3) and confirm `replicas: 2` is set under `alertmanagerSpec`, not at the chart's top level.

* * *

## Design Decisions

Worth recording in your own architecture documentation:

**Why Prometheus for scraping instead of vmagent?** Prometheus is the industry standard. Its ecosystem, documentation and operational knowledge are far more widespread. `vmagent` would consume fewer resources, but adopting it trades familiarity for efficiency.

**Why VictoriaMetrics instead of Thanos?** Both solve long-term retention and HA query deduplication. Thanos requires four components (sidecar, query, store gateway, compactor) and mandatory object storage. VictoriaMetrics achieves comparable results with a single binary and optional object storage, at substantially lower memory cost. On a small cluster, this difference is decisive.

**Why two single-node instances instead of VictoriaMetrics cluster mode?** Cluster mode (`vminsert` / `vmstorage` / `vmselect`) provides true storage-level replication and horizontal scalability, at the cost of three components to operate instead of one. For moderate metric volumes, two independent single-node instances fed identical data provide equivalent redundancy with far less operational complexity. Migrate to cluster mode when a single node can no longer hold the dataset.

**Why 6-hour Prometheus retention?** Prometheus is a collector, not the system of record. A short window keeps memory and disk footprint small on both replicas. All durable history lives in VictoriaMetrics.

* * *