Skip to main content
HashnodeAmir Kolahi

Command Palette

Search for a command to run...

Highly Available Kubernetes Monitoring: Prometheus + VictoriaMetrics

Updated
•13 min read•View as Markdown
A
Amir Kolahi

A step-by-step guide to building a monitoring stack with no single point of failure, using Prometheus for metric collection and VictoriaMetrics for long-term storage.

Table of Contents

Why This Architecture

Prometheus is excellent at scraping metrics, but it has two limitations that matter in production:

It was never designed for long-term storage. Prometheus stores data on local disk. If you need a year of history, the disk fills up, queries slow down, and you are one PVC failure away from losing everything.

It has no built-in replication. Unlike most databases, you cannot configure a primary and a replica. The official answer to high availability is "run two identical instances" — which creates a new problem: each instance has slightly different data, and gaps appear whenever one restarts.

This guide solves both problems:

Problem Solution
Scraping stops if Prometheus dies Run 2 Prometheus replicas
Each replica has gaps and different timestamps VictoriaMetrics deduplicates on query
No long-term retention VictoriaMetrics stores 12+ months
Storage layer becomes the new SPOF Run 2 VictoriaMetrics instances, write to both
Duplicate alerts from 2 Prometheus replicas Alertmanager in clustered mode

Key insight: each Prometheus replica writes to both VictoriaMetrics instances. This means both storage instances hold a complete, identical copy of the data. Losing either one changes nothing.

This is different from load-balancing the writes, which would split the data and leave each instance with half a dataset.

Component Responsibilities

Component Role Replicas
Prometheus Scrapes targets, evaluates alert rules, forwards to storage 2
VictoriaMetrics Long-term time-series storage, query engine 2
Alertmanager Groups, deduplicates and routes alert notifications 2 (clustered)
Grafana Visualization 1
node-exporter Host-level metrics (CPU, RAM, disk, network) 1 per node
kube-state-metrics Kubernetes object state as metrics 1

Prometheus keeps only a short local retention window (6 hours). It acts as a collector and forwarder, not as the system of record. All durable data lives in VictoriaMetrics.

Prerequisites

  • A Kubernetes cluster (this guide was tested on v1.35 with 3 control-plane nodes)

  • kubectl configured and working

  • helm v3

  • A working StorageClass that supports ReadWriteOnce

  • At least ~4 GiB of memory available across the cluster for the monitoring stack

Verify your storage:

kubectl get storageclass

Step 1 — Create the Namespace

kubectl create namespace monitoring

Everything in this guide lives in monitoring. Using a dedicated namespace makes RBAC, network policies and resource quotas far easier later.

Step 2 — Deploy VictoriaMetrics (HA Pair)

Add the chart repository:

helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update
helm search repo vm/victoria-metrics-single --versions | head -3

Create the values file:

# vm-values.yaml
server:
  replicaCount: 2

  retentionPeriod: 12

  persistentVolume:
    enabled: true
    size: 20Gi

  extraArgs:
    dedup.minScrapeInterval: 30s

  resources:
    requests:
      cpu: 100m
      memory: 512Mi
    limits:
      memory: 2Gi

  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: server
          topologyKey: kubernetes.io/hostname

  statefulSet:
    enabled: true

What each setting does

replicaCount: 2 — deploys two independent VictoriaMetrics instances. They do not talk to each other and do not replicate between themselves. Redundancy comes from Prometheus writing the same data to both.

retentionPeriod: 12 — twelve months of retention. The unit is months. Adjust based on your compliance or billing requirements.

dedup.minScrapeInterval: 30s — when two Prometheus replicas send the same sample, VictoriaMetrics keeps only one at query time. This value must match your Prometheus scrapeInterval. If they disagree, deduplication silently fails and you get doubled series.

podAntiAffinity — forces the two instances onto different nodes. Without this, Kubernetes may schedule both on the same node, which defeats the entire purpose of running two.

statefulSet.enabled: true — gives each pod a stable DNS name. This is essential, because Prometheus needs to address each instance individually rather than through a load-balanced Service.

Install:

helm install victoria-metrics vm/victoria-metrics-single \
  -n monitoring \
  -f vm-values.yaml

Verify:

kubectl -n monitoring get pods -l app=server -o wide
kubectl -n monitoring get pvc

You should see two pods, victoria-metrics-victoria-metrics-single-server-0 and -1, on different nodes, each with its own PVC.

Record the stable DNS names

With a StatefulSet plus a headless Service, each pod is addressable at:

<pod-name>.<headless-service-name>.<namespace>.svc:8428

For this installation:

victoria-metrics-victoria-metrics-single-server-0.victoria-metrics-victoria-metrics-single-server.monitoring.svc:8428
victoria-metrics-victoria-metrics-single-server-1.victoria-metrics-victoria-metrics-single-server.monitoring.svc:8428

Confirm the exact names:

kubectl -n monitoring get svc
kubectl -n monitoring get pods -l app=server -o name

Step 3 — Create a Read Service

The headless Service is correct for writes, where we address each pod individually. For reads, we want a single endpoint that load-balances across both instances.

Because both instances hold identical data, it does not matter which one answers a query.

# vm-read-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vm-read
  namespace: monitoring
spec:
  type: ClusterIP
  selector:
    app: server
  ports:
    - name: http
      port: 8428
      targetPort: 8428

kubectl apply -f vm-read-service.yaml

Verify the selector matched both pods:

kubectl -n monitoring get endpoints vm-read

You should see two IP addresses. If you see zero, your label selector does not match — inspect the actual pod labels:

kubectl -n monitoring get pods --show-labels | grep victoria

Production note: a plain Service round-robins between the two instances without health-aware failover for slow-but-alive backends. For stricter guarantees, place vmauth in front. It understands VictoriaMetrics semantics and can retry against the second backend when the first fails.

Step 4 — Deploy kube-prometheus-stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Create the values file:

# prom-values.yaml
prometheus:
  prometheusSpec:
    replicas: 2


    retention: 6h
    retentionSize: 8GB

    scrapeInterval: 30s
    evaluationInterval: 30s


    replicaExternalLabelName: "replica"
    prometheusExternalLabelName: "prometheus"

    remoteWrite:
      - url: http://victoria-metrics-victoria-metrics-single-server-0.victoria-metrics-victoria-metrics-single-server.monitoring.svc:8428/api/v1/write
        queueConfig:
          capacity: 10000
          maxShards: 30
          minShards: 1
          maxSamplesPerSend: 2000
          batchSendDeadline: 5s
      - url: http://victoria-metrics-victoria-metrics-single-server-1.victoria-metrics-victoria-metrics-single-server.monitoring.svc:8428/api/v1/write
        queueConfig:
          capacity: 10000
          maxShards: 30
          minShards: 1
          maxSamplesPerSend: 2000
          batchSendDeadline: 5s

 
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    probeSelectorNilUsesHelmValues: false

    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

    resources:
      requests:
        cpu: 200m
        memory: 1Gi
      limits:
        memory: 3Gi

    podAntiAffinity: "hard"

alertmanager:
  alertmanagerSpec:
    replicas: 2
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 2Gi
    resources:
      requests:
        cpu: 50m
        memory: 128Mi
      limits:
        memory: 256Mi
    podAntiAffinity: "hard"

grafana:
  enabled: true
  adminPassword: "CHANGE_ME"
  persistence:
    enabled: true
    size: 5Gi
  additionalDataSources:
    - name: VictoriaMetrics
      type: prometheus
      access: proxy
      url: http://vm-read.monitoring.svc:8428
      isDefault: true
      jsonData:
        timeInterval: 30s


kubeEtcd:
  enabled: false

The critical settings explained

replicas: 2 — two Prometheus instances scrape every target independently. If one dies, the other keeps collecting. No gap.

retention: 6h — this is deliberately short. Prometheus is a forwarder here, not a database. Keeping the local window small means each replica stays light on memory and disk.

replicaExternalLabelName: "replica" — attaches a label identifying which replica produced each sample. VictoriaMetrics strips this label during deduplication. Without it, deduplication cannot work, because the two series would look genuinely different.

Two remoteWrite entries — this is the heart of the HA design. Each Prometheus replica pushes every sample to both VictoriaMetrics instances. Combined with 2 Prometheus replicas, every sample takes four paths into storage. Any single failure on either side is invisible.

queueConfig — controls the in-memory buffer for remote writes. If a VictoriaMetrics instance becomes briefly unavailable, Prometheus buffers samples and replays them on reconnect. Increase capacity if you expect longer outages.

serviceMonitorSelectorNilUsesHelmValues: false — by default, Prometheus only discovers ServiceMonitors carrying this chart's release label. Setting this to false makes it pick up ServiceMonitors from any namespace, which is what you want for your own applications.

podAntiAffinity: "hard" — same reasoning as with VictoriaMetrics: the two replicas must not share a node.

alertmanager.replicas: 2 — Alertmanager forms a gossip cluster automatically when replicas > 1. Cluster members coordinate so that each alert is delivered once, not once per replica. Skipping this is the most common mistake when making Prometheus HA: you end up with duplicate notifications for every alert.

Install:

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring \
  -f prom-values.yaml

This takes a few minutes — the chart installs a large set of CRDs.

Watch the rollout:

kubectl -n monitoring get pods -w

Step 5 — Verify remote_write

This is the step people skip, and it is the step that catches misconfiguration.

5.1 — Check from the Prometheus side

kubectl -n monitoring port-forward prometheus-kube-prometheus-stack-prometheus-0 9090:9090

Open http://localhost:9090 and run:

prometheus_remote_storage_samples_total

You should see two series — one per remote_write endpoint — with steadily increasing counters.

Now check for failures:

rate(prometheus_remote_storage_samples_failed_total[5m])

This should be 0. Anything above zero means samples are being rejected; check the Prometheus logs.

Check the send queue is not backing up:

prometheus_remote_storage_pending_samples

A persistently growing value means VictoriaMetrics cannot keep up, or the network path is congested.

5.2 — Check both VictoriaMetrics instances independently

Query each instance directly and confirm both hold data.

kubectl -n monitoring port-forward \
  victoria-metrics-victoria-metrics-single-server-0 8428:8428

Open http://localhost:8428/vmui and run:

count(up)

Note the number. Stop the port-forward, then repeat for instance -1 on a different local port:

kubectl -n monitoring port-forward \
  victoria-metrics-victoria-metrics-single-server-1 8429:8428

Open http://localhost:8429/vmui and run the same query.

Both instances must return a similar count. If one returns significantly fewer series, writes are not reaching it — re-check the second remoteWrite URL for typos.

5.3 — Confirm deduplication is active

Still in VMUI, run:

count by (replica) (up)

Two distinct replica values confirm both Prometheus instances are writing.

Then run:

count(up)

If deduplication is working, this returns roughly the number of scrape targets — not double it.

Confirm the flag actually reached the process:

kubectl -n monitoring exec victoria-metrics-victoria-metrics-single-server-0 -- \
  cat /proc/1/cmdline | tr '\0' '\n' | grep dedup

Expected output:

-dedup.minScrapeInterval=30s

If this prints nothing, the extraArgs block did not apply and you will be storing duplicate series.

Step 6 — Configure Grafana

Retrieve the admin password:

kubectl -n monitoring get secret kube-prometheus-stack-grafana \
  -o jsonpath="{.data.admin-password}" | base64 -d; echo

Port-forward:

kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80

Open http://localhost:3000.

Data sources

If you used the additionalDataSources block from Step 4, VictoriaMetrics is already configured and set as default. Verify under Connections → Data sources.

Note that VictoriaMetrics is registered with type: prometheus. This is correct — VictoriaMetrics implements the Prometheus query API, so Grafana treats it identically and all PromQL works unchanged.

Keep the Prometheus data source

Do not delete the default Prometheus data source. It remains useful for:

  • Comparing against VictoriaMetrics when debugging

  • Querying the last few minutes with minimal latency

  • Bundled dashboards that reference it explicitly

Just make sure VictoriaMetrics is the default, so new dashboards use long-term storage.

Defining data sources as code

Configuring data sources through the UI means losing them on reinstall. Defining them in prom-values.yaml — as above — keeps them reproducible and version-controlled.

Step 7 — Failure Testing

A high-availability setup you have not tested is a high-availability setup you do not have.

Test 1 — Kill one Prometheus replica

kubectl -n monitoring delete pod prometheus-kube-prometheus-stack-prometheus-0

While it restarts, query in Grafana:

up

Data should continue flowing, with no visible gap. Replica 1 covered the window.

Test 2 — Kill one VictoriaMetrics instance

kubectl -n monitoring delete pod victoria-metrics-victoria-metrics-single-server-0

Grafana should continue serving queries through vm-read, which routes to instance 1.

Meanwhile, Prometheus buffers writes destined for instance 0. Watch the buffer:

prometheus_remote_storage_pending_samples

When the pod returns, the buffer drains and instance 0 catches up automatically.

Test 3 — Verify alert deduplication

Trigger a test alert and confirm you receive one notification, not two.

Check that Alertmanager formed a cluster:

kubectl -n monitoring port-forward svc/kube-prometheus-stack-alertmanager 9093:9093

Open http://localhost:9093/#/status. Under Cluster Status, you should see two peers with status ready.

If it shows only one peer, the cluster did not form and you will receive duplicate alerts.

Troubleshooting

Grafana shows no data

Work backwards through the chain:

# 1. Is the read Service backed by pods?
kubectl -n monitoring get endpoints vm-read

# 2. Does VictoriaMetrics have data? (use VMUI)
kubectl -n monitoring port-forward victoria-metrics-victoria-metrics-single-server-0 8428:8428

# 3. Is Prometheus actually sending?
kubectl -n monitoring logs prometheus-kube-prometheus-stack-prometheus-0 -c prometheus | grep -i remote

Series counts are doubled

Deduplication is not working. Check three things:

  1. dedup.minScrapeInterval matches Prometheus scrapeInterval exactly

  2. The flag reached the process (/proc/1/cmdline check above)

  3. replicaExternalLabelName is set in the Prometheus spec

One VictoriaMetrics instance lags behind

Normal briefly after a restart while the buffer drains. If it persists:

kubectl -n monitoring logs prometheus-kube-prometheus-stack-prometheus-0 -c prometheus \
  | grep -i "remote_write"

Look for connection refused or timeout errors against that specific URL.

Both VictoriaMetrics pods on the same node

Anti-affinity did not apply. Check the actual pod labels and correct the matchLabels selector:

kubectl -n monitoring get pods -l app=server -o wide --show-labels

Duplicate alert notifications

Alertmanager did not form a cluster. Check the status page (Test 3) and confirm replicas: 2 is set under alertmanagerSpec, not at the chart's top level.

Design Decisions

Worth recording in your own architecture documentation:

Why Prometheus for scraping instead of vmagent? Prometheus is the industry standard. Its ecosystem, documentation and operational knowledge are far more widespread. vmagent would consume fewer resources, but adopting it trades familiarity for efficiency.

Why VictoriaMetrics instead of Thanos? Both solve long-term retention and HA query deduplication. Thanos requires four components (sidecar, query, store gateway, compactor) and mandatory object storage. VictoriaMetrics achieves comparable results with a single binary and optional object storage, at substantially lower memory cost. On a small cluster, this difference is decisive.

Why two single-node instances instead of VictoriaMetrics cluster mode? Cluster mode (vminsert / vmstorage / vmselect) provides true storage-level replication and horizontal scalability, at the cost of three components to operate instead of one. For moderate metric volumes, two independent single-node instances fed identical data provide equivalent redundancy with far less operational complexity. Migrate to cluster mode when a single node can no longer hold the dataset.

Why 6-hour Prometheus retention? Prometheus is a collector, not the system of record. A short window keeps memory and disk footprint small on both replicas. All durable history lives in VictoriaMetrics.

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

Amir Kolahi

5 posts