# HashiCorp Vault — Learning Guide

A hands-on walkthrough of HashiCorp Vault covering secrets management, dynamic database credentials, and Kubernetes authentication.

* * *

## Table of Contents

1.  [What is Vault?](#1-what-is-vault)
    
2.  [Phase 1 — Running Vault with Docker](#2-phase-1--running-vault-with-docker)
    
3.  [Phase 2 — KV Secrets Engine](#3-phase-2--kv-secrets-engine)
    
4.  [Phase 3 — Dynamic Database Secrets](#4-phase-3--dynamic-database-secrets)
    
5.  [Phase 4 — Kubernetes Authentication](#5-phase-4--kubernetes-authentication)
    

* * *

## 1\. What is Vault?

HashiCorp Vault is a secrets management tool that provides a centralized, secure place to store and access sensitive information such as:

*   Database passwords
    
*   API keys
    
*   TLS certificates
    
*   SSH private keys
    
*   Tokens
    

### The Problem Vault Solves

Without Vault, teams typically hardcode secrets in `.env` files or source code:

```bash
# ❌ The wrong way — secret is exposed in code/git
DB_PASSWORD="super_secret_123"
```

This leads to:

*   Secrets leaking into Git history
    
*   No audit trail of who accessed what
    
*   Rotating secrets requires updating every service manually
    
*   If one secret leaks, there is no way to know who used it or from where
    

### How Vault Fixes This

Vault acts as a central secure vault where:

*   All secrets are **encrypted at rest**
    
*   Every access is **audited and logged**
    
*   Secrets can **expire automatically** (TTL)
    
*   Applications get **dynamic, short-lived credentials** instead of static passwords
    
*   Access is controlled via **policies**
    

* * *

## 2\. Phase 1 — Running Vault with Docker

### Why Docker?

We use Docker to run Vault locally for learning purposes. Vault runs in **dev mode**, which means:

*   Everything is stored in memory (no persistence)
    
*   Auto-unsealed on startup
    
*   NOT suitable for production
    

### Create a Docker Network

Before starting any containers, we need a shared network so Vault and the database can communicate with each other by name instead of IP address.

```bash
docker network create vault-net
```

Think of this like connecting two computers to the same switch — once on the same network, containers can reach each other by their container name (e.g., `vault-dev`, `postgres-dev`).

### Start the Vault Container

```bash
docker run --rm -d \
  --name vault-dev \
  --network vault-net \
  -p 8200:8200 \
  -e VAULT_DEV_ROOT_TOKEN_ID=myroot \
  hashicorp/vault:latest \
  vault server -dev -dev-listen-address="0.0.0.0:8200"
```

**What each flag does:**

| Flag | Purpose |
| --- | --- |
| `--rm` | Remove container when stopped |
| `-d` | Run in background (detached) |
| `--name vault-dev` | Give the container a name |
| `--network vault-net` | Attach to our shared network |
| `-p 8200:8200` | Expose port 8200 to the host |
| `VAULT_DEV_ROOT_TOKEN_ID=myroot` | Set the root token to `myroot` |
| `-dev-listen-address="0.0.0.0:8200"` | Listen on all interfaces (required inside Docker) |

### Connect to Vault

Enter the container and set environment variables:

```bash
docker exec -it vault-dev sh

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='myroot'
```

Verify Vault is running:

```bash
vault status
```

Expected output:

```plaintext
Seal Type       shamir
Initialized     true
Sealed          false      ← This means Vault is open and ready
Storage Type    inmem
Version         2.0.2
```

### Understanding Seal / Unseal

When Vault starts, it is **Sealed** by default — no one can access secrets until it is unlocked.

Vault uses **Shamir's Secret Sharing** to split the master key into multiple pieces:

*   Example: split into 5 pieces, need any 3 to unseal
    
*   Each key holder has one piece — no single person can unlock Vault alone
    

In dev mode, Vault **auto-unseals** itself. In production, this is done manually or via auto-unseal (AWS KMS, etc.).

* * *

## 3\. Phase 2 — KV Secrets Engine

### What is KV?

KV (Key-Value) is the simplest secrets engine in Vault. It works like a secure dictionary:

```plaintext
Key   → Value
path  → secret data
```

Vault uses **KV Version 2** by default, which adds versioning — every update to a secret creates a new version, and old versions are preserved.

### Write a Secret

```bash
vault kv put secret/myapp db_password="super_secret_123" api_key="abc-xyz-789"
```

Output:

```plaintext
== Secret Path ==
secret/data/myapp     ← Note the /data/ prefix — this is KV v2

======= Metadata =======
version    1           ← First version
```

### Read a Secret

```bash
vault kv get secret/myapp
```

### Understanding the `/data/` Path

In KV v2, Vault automatically adds a `/data/` layer to every path:

```plaintext
What you write:   secret/myapp
Where it lives:   secret/data/myapp       ← actual data
                  secret/metadata/myapp   ← version history
```

This is important when using the API directly:

```bash
# API path always includes /data/
curl \
  --header "X-Vault-Token: myroot" \
  http://127.0.0.1:8200/v1/secret/data/myapp
```

### Versioning

Every time you update a secret, a new version is created:

```bash
# Update the secret — creates version 2
vault kv put secret/myapp db_password="new_password_456" api_key="abc-xyz-789"

# Read current version (v2)
vault kv get secret/myapp

# Read a specific old version
vault kv get -version=1 secret/myapp
```

### Delete vs Destroy vs Rollback

| Command | What it does |
| --- | --- |
| `vault kv delete -versions=2 secret/myapp` | **Soft delete** — marks version as deleted, data still exists |
| `vault kv destroy -versions=2 secret/myapp` | **Hard delete** — permanently removes the data |
| `vault kv rollback -version=1 secret/myapp` | Creates a new version with old data (e.g., version 3 = copy of version 1) |

> ⚠️ Vault never modifies history. Rollback always creates a **new** version.

### View All Version Metadata

```bash
vault kv metadata get secret/myapp
```

* * *

## 4\. Phase 3 — Dynamic Database Secrets

### The Problem with Static Credentials

Traditional approach:

```plaintext
DB_PASSWORD=super_secret_123  ← same password, forever, for everyone
```

Problems:

*   If it leaks, it works forever
    
*   Can't tell which service used the password
    
*   Rotating means updating every service
    

### How Dynamic Secrets Work

Vault generates a **unique, temporary credential** every time an application asks for one:

```plaintext
App A asks Vault → Vault creates user v-token-abc / pass xyz123 → expires in 1h
App B asks Vault → Vault creates user v-token-def / pass abc456 → expires in 1h
```

If App B is compromised, you revoke only its credential. App A is unaffected.

### Start PostgreSQL

```bash
docker run -d \
  --name postgres-dev \
  --network vault-net \
  -e POSTGRES_USER=root \
  -e POSTGRES_PASSWORD=rootpassword \
  -e POSTGRES_DB=mydb \
  postgres:15
```

Both `vault-dev` and `postgres-dev` are on `vault-net`, so Vault can reach Postgres at `postgres-dev:5432`.

### Enable the Database Secrets Engine

```bash
vault secrets enable database
```

This activates a Vault plugin that knows how to talk to databases and manage credentials.

### Configure the Database Connection

```bash
vault write database/config/mypostgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="my-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres-dev:5432/mydb?sslmode=disable" \
  username="root" \
  password="rootpassword"
```

**What each field means:**

| Field | Purpose |
| --- | --- |
| `plugin_name` | Which database plugin to use |
| `allowed_roles` | Which roles are permitted to use this connection |
| `connection_url` | How to connect — `{{username}}` and `{{password}}` are filled in by Vault |
| `username` / `password` | The admin credentials Vault uses to create/revoke users |

### Create a Role

A role defines **what kind of user** Vault should create when asked for credentials:

```bash
vault write database/roles/my-role \
  db_name=mypostgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"
```

**What each field means:**

| Field | Purpose |
| --- | --- |
| `db_name` | Which database connection to use |
| `creation_statements` | The SQL Vault runs to create the user — `{{name}}`, `{{password}}`, `{{expiration}}` are filled in automatically |
| `default_ttl` | Credential expires after 1 hour |
| `max_ttl` | Even with renewal, credential expires after 24 hours |

### Generate a Dynamic Credential

```bash
vault read database/creds/my-role
```

Output:

```plaintext
lease_id           database/creds/my-role/spfqnE8B...
lease_duration     1h
password           uSf6vcWg4zJyD-JRMpKp
username           v-token-my-role-dgrQiyYxSs4iJNNeaZB4
```

Every time you run this command, Vault:

1.  Connects to PostgreSQL as the admin user
    
2.  Runs the `creation_statements` SQL with a random name and password
    
3.  Returns the credentials to the caller
    
4.  Sets a timer — after 1 hour, Vault deletes the user from PostgreSQL automatically
    

### Verify in PostgreSQL

```bash
docker exec -it postgres-dev psql -U root -d mydb -c "\du"
```

You will see the dynamically created user in the list.

### Revoke a Credential Immediately

```bash
vault lease revoke database/creds/my-role/<lease_id>
```

The user is deleted from PostgreSQL instantly.

### Multiple Roles for Multiple Teams

```bash
# Backend team — read only
vault write database/roles/backend-role \
  db_name=mypostgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Data team — read and write
vault write database/roles/data-role \
  db_name=mypostgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="4h" \
  max_ttl="48h"
```

* * *

## 5\. Phase 4 — Kubernetes Authentication

### The Secret Zero Problem

To get secrets from Vault, an application needs a token. But where does that token come from? If it lives in a `.env` file or a ConfigMap, it is itself a secret — and we are back to square one.

**Kubernetes Auth Method solves this.** Instead of a pre-shared token, the application proves its identity using its Kubernetes Service Account Token — a credential that Kubernetes itself manages and rotates automatically.

### Concepts

#### Service Account

Every Pod in Kubernetes has an identity called a **Service Account**. When a Pod starts, Kubernetes automatically mounts a JWT token inside it at:

```plaintext
/var/run/secrets/kubernetes.io/serviceaccount/token
```

This token contains information about who the Pod is:

```json
{
  "namespace": "default",
  "pod": "myapp-test",
  "serviceaccount": "myapp-sa"
}
```

#### Policy

A Policy in Vault defines what paths a token is allowed to access and what operations are permitted:

```hcl
path "secret/data/myapp" {
  capabilities = ["read"]   # can only read, not write or delete
}
```

Available capabilities: `create`, `read`, `update`, `delete`, `list`, `deny`

#### Kubernetes Role (in Vault)

A Vault Kubernetes Role links a Kubernetes Service Account to a Vault Policy:

```plaintext
myapp-sa (in namespace default) → myapp-policy → can read secret/data/myapp
```

#### ClusterRoleBinding

When a Pod presents its Service Account token to Vault, Vault needs to verify it with the Kubernetes API Server. By default, Kubernetes does not allow just anyone to query its API. We use a `ClusterRoleBinding` to grant Vault's Service Account the `system:auth-delegator` role — which allows it to call the **TokenReview API** to verify tokens.

### Full Setup

#### Step 1 — Deploy Vault in Kubernetes

```bash
kubectl create namespace vault

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:latest
        args: ["server", "-dev"]
        env:
        - name: VAULT_DEV_ROOT_TOKEN_ID
          value: "myroot"
        - name: VAULT_DEV_LISTEN_ADDRESS
          value: "0.0.0.0:8200"
        ports:
        - containerPort: 8200
---
apiVersion: v1
kind: Service
metadata:
  name: vault
  namespace: vault
spec:
  selector:
    app: vault
  ports:
  - port: 8200
    targetPort: 8200
EOF
```

Vault is now reachable from any Pod inside the cluster at `vault.vault.svc:8200`.

#### Step 2 — Create a Service Account for the App

```bash
kubectl create serviceaccount myapp-sa -n default
```

This Service Account will be the identity of our application Pod. It must exist before the Pod is created.

#### Step 3 — Configure Vault

Enter the Vault Pod:

```bash
kubectl exec -it -n vault \
  $(kubectl get pod -n vault -l app=vault -o jsonpath='{.items[0].metadata.name}') -- sh

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='myroot'
```

Put a secret:

```bash
vault kv put secret/myapp \
  db_password="super_secret_123" \
  api_key="abc-xyz-789"
```

Enable Kubernetes auth:

```bash
vault auth enable kubernetes
```

Tell Vault where the Kubernetes API Server is:

```bash
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443"
```

`kubernetes.default.svc` is the internal DNS name for the Kubernetes API Server — it is the same in every cluster.

Create a Policy:

```bash
vault policy write myapp-policy - <<EOF
path "secret/data/myapp" {
  capabilities = ["read"]
}
EOF
```

Create a Kubernetes Role in Vault:

```bash
vault write auth/kubernetes/role/myapp-role \
  bound_service_account_names=myapp-sa \
  bound_service_account_namespaces=default \
  policies=myapp-policy \
  ttl=1h
```

This says: **"Any Pod running as** `myapp-sa` **in the** `default` **namespace gets a Vault token with** `myapp-policy` **for 1 hour."**

#### Step 4 — Grant Vault Permission to Verify Tokens

```bash
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-tokenreview
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: default
  namespace: vault
EOF
```

This grants the Vault Pod's Service Account the ability to call the Kubernetes TokenReview API — so Vault can verify whether the token a Pod presents is legitimate.

#### Step 5 — Deploy the Application Pod

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: myapp-test
  namespace: default
spec:
  serviceAccountName: myapp-sa
  containers:
  - name: myapp
    image: curlimages/curl:latest
    command: ["sleep", "3600"]
EOF
```

#### Step 6 — Test the Full Flow

Enter the Pod:

```bash
kubectl exec -it myapp-test -- sh
```

Read the Service Account token:

```bash
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
```

Authenticate with Vault using the Service Account token:

```bash
curl -s \
  --request POST \
  --data "{\"jwt\": \"$JWT\", \"role\": \"myapp-role\"}" \
  http://vault.vault.svc:8200/v1/auth/kubernetes/login
```

Vault returns a token:

```json
{
  "auth": {
    "client_token": "hvs.CAESI...",
    "policies": ["default", "myapp-policy"],
    "lease_duration": 3600
  }
}
```

Use the Vault token to read the secret:

```bash
VAULT_TOKEN="hvs.CAESI..."

curl -s \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  http://vault.vault.svc:8200/v1/secret/data/myapp
```

Output:

```json
{
  "data": {
    "data": {
      "api_key": "abc-xyz-789",
      "db_password": "super_secret_123"
    }
  }
}
```

### The Complete Flow

```plaintext
Pod starts
  ↓
Kubernetes gives Pod a Service Account token (JWT)
  ↓
Pod sends JWT to Vault: "this is who I am"
  ↓
Vault calls Kubernetes TokenReview API: "is this token valid?"
  ↓  (ClusterRoleBinding makes this possible)
Kubernetes confirms: "yes, this is myapp-sa in namespace default"
  ↓
Vault checks: "is myapp-sa in myapp-role?" → yes
  ↓
Vault issues a token with myapp-policy (read access to secret/myapp)
  ↓
Pod uses Vault token to read secret
  ↓
Pod gets db_password and api_key ✅
```

**The key insight:** The Pod never had a pre-shared password or token. It authenticated using its Kubernetes identity alone — solving the Secret Zero Problem.

* * *

## Summary

| Phase | What We Learned |
| --- | --- |
| Phase 1 | Running Vault in Docker, Seal/Unseal concept |
| Phase 2 | KV secrets engine, versioning, soft/hard delete |
| Phase 3 | Dynamic database credentials with TTL and automatic cleanup |
| Phase 4 | Kubernetes authentication, policies, and the Secret Zero solution |