Chapter 9.1: Secrets Management (Vault)
Part 1: The "Why" - What is a Secret?
In DevOps, a **"secret"** is any piece of data that grants access or privilege. It's not just passwords. It includes:
- Database usernames and passwords (e.g.,
DB_USER,DB_PASS) - API keys for third-party services (e.g.,
STRIPE_API_KEY,AWS_SECRET_ACCESS_KEY) - TLS/SSL certificates and private keys
- SSH keys for accessing servers
- JWT signing keys
The Problem: Secret Sprawl
Where do most developers store these secrets?
1. Hard-coded in their code (val apiKey = "sk_live_...") - **TERRIBLE!**
2. In .env files or .properties files. - **BAD!**
3. As "Secrets" in their CI/CD tool (like GitHub Actions Secrets). - **BETTER, but still not perfect.**
This is called **Secret Sprawl**. Your secrets are "sprawled" (phaile hue) across dozens of files, repositories, and servers. This creates a massive security hole.
Why is .env or GitHub Secrets Bad?
- Static:** The
DB_PASSWORDis the same for all 500 of your containers. If *one* developer or *one* container is compromised, the hacker gets the password and has access to your entire database. - No Audit Trail:** Who accessed the
AWS_SECRET_KEY? When? You have no idea. - No Revocation:** If a key leaks, you must manually go to AWS, generate a new key, and then *manually* update it in all 50 places you used it. This is slow and error-prone.
- No "Time-to-Live" (TTL):** The secret lives forever.
The Solution: Centralized Secrets Management
You need a "single source of truth" for all secrets. A central, secure, encrypted, and auditable "tijori" (vault). This is **HashiCorp Vault**.
Vault** is an open-source tool that provides a **secure API for secrets**. Instead of your app reading from an .env file, it *asks* Vault for the database password. Vault checks who the app is (authentication), checks if it has permission (authorization), and *only then* gives it the secret.
Even better, Vault can **dynamically generate** secrets. It can create a *brand new, temporary* database password for your app that *automatically expires* in 5 minutes. Now, even if the secret leaks, it's useless 5 minutes later.
Part 2: Vault Core Architecture
To understand Vault, you must understand its core components:
- Vault Server:** The main
vaultprogram that runs as a server. It listens on a port (e.g., 8200) and provides an HTTP API. - Storage Backend:** Vault *does not* store data itself. It *encrypts* data and gives it to a storage backend. This could be a
filesystem, AWS S3, or (most commonly) **Consul** (another HashiCorp tool). - Auth Methods:** How you *prove* you are who you say you are. This is the "login" system. Examples:
token,userpass(username/password),github,aws,kubernetes. - Secrets Engines:** The "plugins" that store or generate secrets.
kv(Key-Value): A simple, static key-value store (like a secure.envfile).database: Connects to a database (like PostgreSQL) and generates *dynamic* credentials.aws: Generates *dynamic*, temporary AWS IAM credentials.
- Policies:** The "IAM" of Vault. The rulebook that defines *who* (which token) can access *what* (which secret path).
- Sealing / Unsealing:** Vault's ultimate security feature. When a Vault server starts, it's **Sealed**. It knows *where* its data is (the storage backend), but it does **not** know *how* to decrypt it (the "encryption key"). The encryption key is split into 5 "Unseal Keys." To "unseal" Vault, 3 of the 5 key holders must enter their key. This prevents any single person (even the root admin) from accessing all secrets.
Part 3: Installation & Running in Dev Mode
The best way to learn Vault is to run it in **"-dev" mode**. This is a special, in-memory, auto-unsealed mode for development. **DO NOT** use this in production.
Step 1: Install Vault
Go to the official Vault website and download the binary for your OS. It's a single file.
# On macOS (using Homebrew)
$ brew tap hashicorp/tap
$ brew install vault
# On Linux (Ubuntu/Debian)
$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
$ sudo apt update && sudo apt install vault
Step 2: Run in Dev Mode
Open your terminal and run this one command:
$ vault server -dev
==> Vault server configuration:
Api Address: http://127.0.0.1:8200
Cgo: disabled
Cluster Address: https://127.0.0.1:8201
Listener 1: tcp (addr: "127.0.0.1:8200", ...)
Log Level: info
...
==> Vault server started!
...
Unseal Key: some-long-random-key
Root Token: hvs.SOME_VERY_SECRET_ROOT_TOKEN
...
This is critical. Vault is now running, and it has printed your two most important pieces of information:
- Unseal Key:** In production, this would be 5 keys. In dev mode, it's just one.
- Root Token:** This is the "root user" password. It has god-mode access to everything. **Copy this token!**
Step 3: Configure Your Shell
Open a **new, second terminal** (leave the server running). You need to tell your vault CLI (client) how to find the server and what your token is.
# Set the server address
$ export VAULT_ADDR="http://127.0.0.1:8200"
# Set your token (paste the Root Token from the server output)
$ export VAULT_TOKEN="hvs.SOME_VERY_SECRET_ROOT_TOKEN"
# 4. Check your status
$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false <-- (false means it's UNSEALED and ready)
Total Shares 1
Threshold 1
Version 1.15.0
...
You are now logged in as root and ready to issue commands.
Part 4: Storing Static Secrets (KV Engine)
The kv (Key-Value) engine is the simplest secrets engine. It's a secure "hash map" or "dictionary." You use it for data that doesn't change often, like an API key for a third-party service.
Dev mode enables `kv` (version 2) at the path secret/ by default.
Writing a Secret (`kv put`)
The command is vault kv put [path] [key=value]. The path secret/ is reserved for the `kv` engine.
# Store the AWS API keys for "my-app"
$ vault kv put secret/my-app/aws \
aws_access_key_id="AKIA123456789" \
aws_secret_access_key="MySuperSecretKey"
==== Data ====
Key Value
--- -----
created_time 2025-11-10T14:30:00Z
deletion_time n/a
destroyed false
version 1
Reading a Secret (`kv get`)
The command is vault kv get [path].
$ vault kv get secret/my-app/aws
==== Metadata ====
Key Value
--- -----
created_time 2025-11-10T14:30:00Z
...
version 1
==== Data ====
Key Value
--- -----
aws_access_key_id AKIA123456789
aws_secret_access_key MySuperSecretKey
Versioning (KV v2)
By default, the kv engine is **versioned**. If you write a new secret to the same path, it creates a new version (v2) and "hides" the old one (v1). This is a lifesaver if you accidentally save a wrong password.
# Overwrite the secret
$ vault kv put secret/my-app/aws aws_secret_access_key="ANOTHER_NEW_KEY"
==== Data ====
Key Value
--- -----
created_time 2025-11-10T14:35:00Z
...
version 2
# Get the *latest* version (v2)
$ vault kv get secret/my-app/aws
...
version 2
...
aws_secret_access_key ANOTHER_NEW_KEY
# Get the *previous* version (v1)
$ vault kv get -version=1 secret/my-app/aws
...
version 1
...
aws_access_key_id AKIA123456789
aws_secret_access_key MySuperSecretKey
Deleting Secrets
# 'delete' just creates a new "deleted" version (soft delete)
$ vault kv delete secret/my-app/aws
# 'destroy' permanently destroys a specific version's data
$ vault kv destroy -versions=1 secret/my-app/aws
Part 5: Auth Methods & Policies (The "Real" Way)
You should **NEVER** give the **Root Token** to your applications or other users. The Root Token is for initial setup *only*.
The proper workflow is:
1. You (the admin) create a **Policy** (a set of rules).
2. You enable an **Auth Method** (like userpass).
3. You create a **User** (e.g., dev-user) and *attach* the Policy to them.
4. The `dev-user` then logs in and gets a *new, less-privileged token* that can only do what the Policy allows.
Step 1: Write a Policy (HCL)
Policies are written in **HCL (HashiCorp Configuration Language)**. They are "deny by default." You only add capabilities that you want to *allow*.
Let's create a "read-only" policy for our app.
my-app-policy.hcl# This policy grants READ-ONLY access to the secrets
# at the path "secret/data/my-app/*"
path "secret/data/my-app/*" {
capabilities = [ "read", "list" ]
}
# Note: For KV v2, you MUST add 'data/' after the engine path ('secret/')
path: The path of the secret you are controlling. The*is a wildcard.capabilities: The list of allowed actions. The most common arecreate,read,update,delete,list, andsudo.
Step 2: Upload the Policy
# 'vault policy write'
$ vault policy write my-app-readonly my-app-policy.hcl
Success! Uploaded policy: my-app-readonly
Step 3: Enable Auth Method & Create User
Let's create a simple username/password login for our app.
# 1. Enable the 'userpass' auth method
$ vault auth enable userpass
Success! Enabled userpass auth method at: userpass/
# 2. Create a new user named 'my-app-user' and attach our policy
$ vault write auth/userpass/users/my-app-user \
password="my-strong-password" \
policies="default,my-app-readonly"
Success! Data written to: auth/userpass/users/my-app-user
Step 4: Test it!
First, let's "un-set" our root token so we are logged out.
$ unset VAULT_TOKEN
# Now, try to read the secret. It will fail.
$ vault kv get secret/my-app/aws
Error...: missing client token: 403
# Now, log in as our new, less-privileged user
$ vault login -method=userpass username="my-app-user"
Password (will be hidden): *****************
Success! You are now authenticated. The token metadata is:
Key Value
--- -----
token hvs.A_BRAND_NEW_LIMITED_TOKEN
token_policies [default, my-app-readonly]
token_duration 768h
...
A *new* token (hvs.A_BRAND_NEW_LIMITED_TOKEN) is automatically put in the VAULT_TOKEN environment variable. This new token *only* has the "my-app-readonly" policy.
# Let's try to read the secret we HAVE permission for:
$ vault kv get secret/my-app/aws
==== Data ====
Key Value
--- -----
aws_access_key_id AKIA123456789
...
# Now, let's try to write a secret (we DON'T have permission):
$ vault kv put secret/my-app/new-secret value="test"
Error writing data to secret/data/my-app/new-secret: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/secret/data/my-app/new-secret
Code: 403. Errors:
* 1 error occurred:
* permission denied
It worked! Our policy successfully blocked the write action. This is the core loop of DevSecOps.
Part 6: Dynamic Secrets (Vault's Killer Feature)
Static secrets (like our AWS key) are good, but they still live forever. The *best* security is a secret that doesn't exist until you need it, and disappears after you use it. This is **Dynamic Secrets**.
How it Works (Database Example)
We will configure Vault to be a "manager" for our PostgreSQL database. Our app will *never* have the admin password.
Instead, our app will say: "Hey Vault, I need to talk to the database."
Vault will *dynamically create* a *new* PostgreSQL user (e.g., v-app-abc123) with a *new* password, give it `READ ONLY` permissions, and set its expiration to **5 minutes**.
Your app uses this temporary credential. 5 minutes later, Vault automatically logs into PostgreSQL and *deletes* that user.
Now, even if your app's credential leaks, it's useless 5 minutes later.
Step 1: Enable the Database Secrets Engine
$ vault secrets enable database
Success! Enabled the database secrets engine at: database/
Step 2: Configure Vault's access to PostgreSQL
(This assumes you have a running PostgreSQL server and a "root" user named vaultadmin that Vault can use to create *other* users.)
$ vault write database/config/my-postgres-db \
plugin_name="postgresql-database-plugin" \
allowed_roles="my-app-role" \
connection_url="postgresql://vaultadmin:pass@postgres-server:5432/mydb?sslmode=disable"
Step 3: Create a "Role" (The "Recipe" for new users)
This is the most important step. You write the SQL that Vault will use to create new users.
# Define a role named 'my-app-role'
$ vault write database/roles/my-app-role \
db_name="my-postgres-db" \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl="1m" \
max_ttl="5m"
creation_statements: The SQL command. Vault will replace{{name}},{{password}}, and{{expiration}}with new, random values.default_ttl="1m": The secret expires in 1 minute (but the app can renew it).max_ttl="5m": The secret can *never* live longer than 5 minutes.
Step 4: The App Workflow (Requesting a Secret)
Now, your app (which has a token with a policy allowing it to access database/creds/my-app-role) makes *this* call:
$ vault read database/creds/my-app-role
Key Value
--- -----
lease_id database/creds/my-app-role/abc-123
lease_duration 1m
lease_renewable true
password "A1b-super-random-password"
username "v-app-my-app-role-123xyz"
Vault has just created a *brand new* user in PostgreSQL (v-app-my-app-role-123xyz) and given the credentials to your app. Your app now connects to the database *using this temporary user*.
When the 1-minute TTL is up, Vault automatically revokes the lease and runs the `revocation_statements` (which is DROP ROLE "{{name}}") in your database. The user is gone.
**This is the future of secrets management. Your app *never* touches a static, long-lived password.**
Part 7: Vault in Kubernetes (The `Vault Agent`)
How does your *Pod* in Kubernetes log in to Vault to get secrets?
You use the **Vault Agent Injector**. This is a service you install in your K8s cluster. It watches for new Pods being created. When it sees a Pod with special "annotations" (metadata), it *automatically* injects two containers into that Pod:
vault-agent:** An "init container" that runs *first*. It talks to Vault, authenticates (using the Pod's Kubernetes Service Account), gets the secrets, and writes them to a shared *in-memory volume* (/vault/secrets).vault-agent-sidecar:** A "sidecar container" that runs *alongside* your app. Its job is to keep renewing the secrets (like a dynamic password) before they expire.
Now, your main application container just reads its password from a file: /vault/secrets/db-password.txt. It has *no idea* Vault even exists. This is perfect separation of concerns.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
...
template:
metadata:
# === These annotations trigger the Vault Agent ===
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/role: 'my-app-role' # The Vault role
vault.hashicorp.com/agent-inject-secret-db-pass: 'database/creds/my-app-role'
vault.hashicorp.com/agent-inject-template-db-pass: |
{{- with secret "database/creds/my-app-role" -}}
export DB_PASSWORD={{ .Data.password }}
{{- end -}}
spec:
containers:
- name: my-app-container
image: my-app:v1
# This script is run by the agent to load the secrets
command: ["/bin/sh", "-c", "source /vault/secrets/db-pass && /app/start"]
Read More about Vault + Kubernetes Integration →