HashiCorp Vault — Learning Guide

A hands-on walkthrough of HashiCorp Vault covering secrets management, dynamic database credentials, and Kubernetes authentication.

What is Vault?
Phase 1 — Running Vault with Docker
Phase 2 — KV Secrets Engine
Phase 3 — Dynamic Database Secrets
Phase 4 — Kubernetes Authentication

1. What is Vault?

HashiCorp Vault is a secrets management tool that provides a centralized, secure place to store and access sensitive information such as:

Database passwords
API keys
TLS certificates
SSH private keys
Tokens

The Problem Vault Solves

Without Vault, teams typically hardcode secrets in .env files or source code:

# ❌ The wrong way — secret is exposed in code/git
DB_PASSWORD="super_secret_123"

This leads to:

Secrets leaking into Git history
No audit trail of who accessed what
Rotating secrets requires updating every service manually
If one secret leaks, there is no way to know who used it or from where

How Vault Fixes This

Vault acts as a central secure vault where:

All secrets are encrypted at rest
Every access is audited and logged
Secrets can expire automatically (TTL)
Applications get dynamic, short-lived credentials instead of static passwords
Access is controlled via policies

2. Phase 1 — Running Vault with Docker

Why Docker?

We use Docker to run Vault locally for learning purposes. Vault runs in dev mode, which means:

Everything is stored in memory (no persistence)
Auto-unsealed on startup
NOT suitable for production

Create a Docker Network

Before starting any containers, we need a shared network so Vault and the database can communicate with each other by name instead of IP address.

docker network create vault-net

Think of this like connecting two computers to the same switch — once on the same network, containers can reach each other by their container name (e.g., vault-dev, postgres-dev).

Start the Vault Container

docker run --rm -d \
  --name vault-dev \
  --network vault-net \
  -p 8200:8200 \
  -e VAULT_DEV_ROOT_TOKEN_ID=myroot \
  hashicorp/vault:latest \
  vault server -dev -dev-listen-address="0.0.0.0:8200"

What each flag does:

Flag	Purpose
`--rm`	Remove container when stopped
`-d`	Run in background (detached)
`--name vault-dev`	Give the container a name
`--network vault-net`	Attach to our shared network
`-p 8200:8200`	Expose port 8200 to the host
`VAULT_DEV_ROOT_TOKEN_ID=myroot`	Set the root token to `myroot`
`-dev-listen-address="0.0.0.0:8200"`	Listen on all interfaces (required inside Docker)

Connect to Vault

Enter the container and set environment variables:

docker exec -it vault-dev sh

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='myroot'

Verify Vault is running:

vault status

Expected output:

Seal Type       shamir
Initialized     true
Sealed          false      ← This means Vault is open and ready
Storage Type    inmem
Version         2.0.2

Understanding Seal / Unseal

When Vault starts, it is Sealed by default — no one can access secrets until it is unlocked.

Vault uses Shamir's Secret Sharing to split the master key into multiple pieces:

Example: split into 5 pieces, need any 3 to unseal
Each key holder has one piece — no single person can unlock Vault alone

In dev mode, Vault auto-unseals itself. In production, this is done manually or via auto-unseal (AWS KMS, etc.).

3. Phase 2 — KV Secrets Engine

What is KV?

KV (Key-Value) is the simplest secrets engine in Vault. It works like a secure dictionary:

Key   → Value
path  → secret data

Vault uses KV Version 2 by default, which adds versioning — every update to a secret creates a new version, and old versions are preserved.

Write a Secret

vault kv put secret/myapp db_password="super_secret_123" api_key="abc-xyz-789"

Output:

== Secret Path ==
secret/data/myapp     ← Note the /data/ prefix — this is KV v2

======= Metadata =======
version    1           ← First version

Read a Secret

vault kv get secret/myapp

Understanding the `/data/` Path

In KV v2, Vault automatically adds a /data/ layer to every path:

What you write:   secret/myapp
Where it lives:   secret/data/myapp       ← actual data
                  secret/metadata/myapp   ← version history

This is important when using the API directly:

# API path always includes /data/
curl \
  --header "X-Vault-Token: myroot" \
  http://127.0.0.1:8200/v1/secret/data/myapp

Versioning

Every time you update a secret, a new version is created:

# Update the secret — creates version 2
vault kv put secret/myapp db_password="new_password_456" api_key="abc-xyz-789"

# Read current version (v2)
vault kv get secret/myapp

# Read a specific old version
vault kv get -version=1 secret/myapp

Delete vs Destroy vs Rollback

Command	What it does
`vault kv delete -versions=2 secret/myapp`	Soft delete — marks version as deleted, data still exists
`vault kv destroy -versions=2 secret/myapp`	Hard delete — permanently removes the data
`vault kv rollback -version=1 secret/myapp`	Creates a new version with old data (e.g., version 3 = copy of version 1)

⚠️ Vault never modifies history. Rollback always creates a new version.

View All Version Metadata

vault kv metadata get secret/myapp

4. Phase 3 — Dynamic Database Secrets

The Problem with Static Credentials

Traditional approach:

DB_PASSWORD=super_secret_123  ← same password, forever, for everyone

Problems:

If it leaks, it works forever
Can't tell which service used the password
Rotating means updating every service

How Dynamic Secrets Work

Vault generates a unique, temporary credential every time an application asks for one:

App A asks Vault → Vault creates user v-token-abc / pass xyz123 → expires in 1h
App B asks Vault → Vault creates user v-token-def / pass abc456 → expires in 1h

If App B is compromised, you revoke only its credential. App A is unaffected.

Start PostgreSQL

docker run -d \
  --name postgres-dev \
  --network vault-net \
  -e POSTGRES_USER=root \
  -e POSTGRES_PASSWORD=rootpassword \
  -e POSTGRES_DB=mydb \
  postgres:15

Both vault-dev and postgres-dev are on vault-net, so Vault can reach Postgres at postgres-dev:5432.

Enable the Database Secrets Engine

vault secrets enable database

This activates a Vault plugin that knows how to talk to databases and manage credentials.

Configure the Database Connection

vault write database/config/mypostgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="my-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres-dev:5432/mydb?sslmode=disable" \
  username="root" \
  password="rootpassword"

What each field means:

Field	Purpose
`plugin_name`	Which database plugin to use
`allowed_roles`	Which roles are permitted to use this connection
`connection_url`	How to connect — `{{username}}` and `{{password}}` are filled in by Vault
`username` / `password`	The admin credentials Vault uses to create/revoke users

Create a Role

A role defines what kind of user Vault should create when asked for credentials:

vault write database/roles/my-role \
  db_name=mypostgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

What each field means:

Field	Purpose
`db_name`	Which database connection to use
`creation_statements`	The SQL Vault runs to create the user — `{{name}}`, `{{password}}`, `{{expiration}}` are filled in automatically
`default_ttl`	Credential expires after 1 hour
`max_ttl`	Even with renewal, credential expires after 24 hours

Generate a Dynamic Credential

vault read database/creds/my-role

Output:

lease_id           database/creds/my-role/spfqnE8B...
lease_duration     1h
password           uSf6vcWg4zJyD-JRMpKp
username           v-token-my-role-dgrQiyYxSs4iJNNeaZB4

Every time you run this command, Vault:

Connects to PostgreSQL as the admin user
Runs the creation_statements SQL with a random name and password
Returns the credentials to the caller
Sets a timer — after 1 hour, Vault deletes the user from PostgreSQL automatically

Verify in PostgreSQL

docker exec -it postgres-dev psql -U root -d mydb -c "\du"

You will see the dynamically created user in the list.

Revoke a Credential Immediately

vault lease revoke database/creds/my-role/<lease_id>

The user is deleted from PostgreSQL instantly.

Multiple Roles for Multiple Teams

# Backend team — read only
vault write database/roles/backend-role \
  db_name=mypostgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Data team — read and write
vault write database/roles/data-role \
  db_name=mypostgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="4h" \
  max_ttl="48h"

5. Phase 4 — Kubernetes Authentication

The Secret Zero Problem

To get secrets from Vault, an application needs a token. But where does that token come from? If it lives in a .env file or a ConfigMap, it is itself a secret — and we are back to square one.

Kubernetes Auth Method solves this. Instead of a pre-shared token, the application proves its identity using its Kubernetes Service Account Token — a credential that Kubernetes itself manages and rotates automatically.

Concepts

Service Account

Every Pod in Kubernetes has an identity called a Service Account. When a Pod starts, Kubernetes automatically mounts a JWT token inside it at:

/var/run/secrets/kubernetes.io/serviceaccount/token

This token contains information about who the Pod is:

{
  "namespace": "default",
  "pod": "myapp-test",
  "serviceaccount": "myapp-sa"
}

Policy

A Policy in Vault defines what paths a token is allowed to access and what operations are permitted:

path "secret/data/myapp" {
  capabilities = ["read"]   # can only read, not write or delete
}

Available capabilities: create, read, update, delete, list, deny

Kubernetes Role (in Vault)

A Vault Kubernetes Role links a Kubernetes Service Account to a Vault Policy:

myapp-sa (in namespace default) → myapp-policy → can read secret/data/myapp

ClusterRoleBinding

When a Pod presents its Service Account token to Vault, Vault needs to verify it with the Kubernetes API Server. By default, Kubernetes does not allow just anyone to query its API. We use a ClusterRoleBinding to grant Vault's Service Account the system:auth-delegator role — which allows it to call the TokenReview API to verify tokens.

Full Setup

Step 1 — Deploy Vault in Kubernetes

kubectl create namespace vault

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:latest
        args: ["server", "-dev"]
        env:
        - name: VAULT_DEV_ROOT_TOKEN_ID
          value: "myroot"
        - name: VAULT_DEV_LISTEN_ADDRESS
          value: "0.0.0.0:8200"
        ports:
        - containerPort: 8200
---
apiVersion: v1
kind: Service
metadata:
  name: vault
  namespace: vault
spec:
  selector:
    app: vault
  ports:
  - port: 8200
    targetPort: 8200
EOF

Vault is now reachable from any Pod inside the cluster at vault.vault.svc:8200.

Step 2 — Create a Service Account for the App

kubectl create serviceaccount myapp-sa -n default

This Service Account will be the identity of our application Pod. It must exist before the Pod is created.

Step 3 — Configure Vault

Enter the Vault Pod:

kubectl exec -it -n vault \
  $(kubectl get pod -n vault -l app=vault -o jsonpath='{.items[0].metadata.name}') -- sh

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='myroot'

Put a secret:

vault kv put secret/myapp \
  db_password="super_secret_123" \
  api_key="abc-xyz-789"

Enable Kubernetes auth:

vault auth enable kubernetes

Tell Vault where the Kubernetes API Server is:

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443"

kubernetes.default.svc is the internal DNS name for the Kubernetes API Server — it is the same in every cluster.

Create a Policy:

vault policy write myapp-policy - <<EOF
path "secret/data/myapp" {
  capabilities = ["read"]
}
EOF

Create a Kubernetes Role in Vault:

vault write auth/kubernetes/role/myapp-role \
  bound_service_account_names=myapp-sa \
  bound_service_account_namespaces=default \
  policies=myapp-policy \
  ttl=1h

This says: "Any Pod running as myapp-sa in the default namespace gets a Vault token with myapp-policy for 1 hour."

Step 4 — Grant Vault Permission to Verify Tokens

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-tokenreview
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: default
  namespace: vault
EOF

This grants the Vault Pod's Service Account the ability to call the Kubernetes TokenReview API — so Vault can verify whether the token a Pod presents is legitimate.

Step 5 — Deploy the Application Pod

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: myapp-test
  namespace: default
spec:
  serviceAccountName: myapp-sa
  containers:
  - name: myapp
    image: curlimages/curl:latest
    command: ["sleep", "3600"]
EOF

Step 6 — Test the Full Flow

Enter the Pod:

kubectl exec -it myapp-test -- sh

Read the Service Account token:

JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

Authenticate with Vault using the Service Account token:

curl -s \
  --request POST \
  --data "{\"jwt\": \"$JWT\", \"role\": \"myapp-role\"}" \
  http://vault.vault.svc:8200/v1/auth/kubernetes/login

Vault returns a token:

{
  "auth": {
    "client_token": "hvs.CAESI...",
    "policies": ["default", "myapp-policy"],
    "lease_duration": 3600
  }
}

Use the Vault token to read the secret:

VAULT_TOKEN="hvs.CAESI..."

curl -s \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  http://vault.vault.svc:8200/v1/secret/data/myapp

Output:

{
  "data": {
    "data": {
      "api_key": "abc-xyz-789",
      "db_password": "super_secret_123"
    }
  }
}

The Complete Flow

Pod starts
  ↓
Kubernetes gives Pod a Service Account token (JWT)
  ↓
Pod sends JWT to Vault: "this is who I am"
  ↓
Vault calls Kubernetes TokenReview API: "is this token valid?"
  ↓  (ClusterRoleBinding makes this possible)
Kubernetes confirms: "yes, this is myapp-sa in namespace default"
  ↓
Vault checks: "is myapp-sa in myapp-role?" → yes
  ↓
Vault issues a token with myapp-policy (read access to secret/myapp)
  ↓
Pod uses Vault token to read secret
  ↓
Pod gets db_password and api_key ✅

The key insight: The Pod never had a pre-shared password or token. It authenticated using its Kubernetes identity alone — solving the Secret Zero Problem.

Summary

Phase	What We Learned
Phase 1	Running Vault in Docker, Seal/Unseal concept
Phase 2	KV secrets engine, versioning, soft/hard delete
Phase 3	Dynamic database credentials with TTL and automatic cleanup
Phase 4	Kubernetes authentication, policies, and the Secret Zero solution

Command Palette

Table of Contents

1. What is Vault?

The Problem Vault Solves

How Vault Fixes This

2. Phase 1 — Running Vault with Docker

Why Docker?

Create a Docker Network

Start the Vault Container

Connect to Vault

Understanding Seal / Unseal

3. Phase 2 — KV Secrets Engine

What is KV?

Write a Secret

Read a Secret

Understanding the /data/ Path

Versioning

Delete vs Destroy vs Rollback

View All Version Metadata

4. Phase 3 — Dynamic Database Secrets

The Problem with Static Credentials

How Dynamic Secrets Work

Start PostgreSQL

Enable the Database Secrets Engine

Configure the Database Connection

Create a Role

Generate a Dynamic Credential

Verify in PostgreSQL

Revoke a Credential Immediately

Multiple Roles for Multiple Teams

5. Phase 4 — Kubernetes Authentication

The Secret Zero Problem

Concepts

Service Account

Policy

Kubernetes Role (in Vault)

ClusterRoleBinding

Full Setup

Step 1 — Deploy Vault in Kubernetes

Step 2 — Create a Service Account for the App

Step 3 — Configure Vault

Step 4 — Grant Vault Permission to Verify Tokens

Step 5 — Deploy the Application Pod

Step 6 — Test the Full Flow

The Complete Flow

Summary

Comments

More from this blog

Understanding the `/data/` Path