Chapter 9.1: Secrets Management (Vault)

Part 1: The "Why" - What is a Secret?

In DevOps, a **"secret"** is any piece of data that grants access or privilege. It's not just passwords. It includes:

  • Database usernames and passwords (e.g., DB_USER, DB_PASS)
  • API keys for third-party services (e.g., STRIPE_API_KEY, AWS_SECRET_ACCESS_KEY)
  • TLS/SSL certificates and private keys
  • SSH keys for accessing servers
  • JWT signing keys

The Problem: Secret Sprawl

Where do most developers store these secrets?
1. Hard-coded in their code (val apiKey = "sk_live_...") - **TERRIBLE!**
2. In .env files or .properties files. - **BAD!**
3. As "Secrets" in their CI/CD tool (like GitHub Actions Secrets). - **BETTER, but still not perfect.**

This is called **Secret Sprawl**. Your secrets are "sprawled" (phaile hue) across dozens of files, repositories, and servers. This creates a massive security hole.

Why is .env or GitHub Secrets Bad?

  • Static:** The DB_PASSWORD is the same for all 500 of your containers. If *one* developer or *one* container is compromised, the hacker gets the password and has access to your entire database.
  • No Audit Trail:** Who accessed the AWS_SECRET_KEY? When? You have no idea.
  • No Revocation:** If a key leaks, you must manually go to AWS, generate a new key, and then *manually* update it in all 50 places you used it. This is slow and error-prone.
  • No "Time-to-Live" (TTL):** The secret lives forever.

The Solution: Centralized Secrets Management

You need a "single source of truth" for all secrets. A central, secure, encrypted, and auditable "tijori" (vault). This is **HashiCorp Vault**.

Vault** is an open-source tool that provides a **secure API for secrets**. Instead of your app reading from an .env file, it *asks* Vault for the database password. Vault checks who the app is (authentication), checks if it has permission (authorization), and *only then* gives it the secret.
Even better, Vault can **dynamically generate** secrets. It can create a *brand new, temporary* database password for your app that *automatically expires* in 5 minutes. Now, even if the secret leaks, it's useless 5 minutes later.

Part 2: Vault Core Architecture

To understand Vault, you must understand its core components:

  • Vault Server:** The main vault program that runs as a server. It listens on a port (e.g., 8200) and provides an HTTP API.
  • Storage Backend:** Vault *does not* store data itself. It *encrypts* data and gives it to a storage backend. This could be a filesystem, AWS S3, or (most commonly) **Consul** (another HashiCorp tool).
  • Auth Methods:** How you *prove* you are who you say you are. This is the "login" system. Examples: token, userpass (username/password), github, aws, kubernetes.
  • Secrets Engines:** The "plugins" that store or generate secrets.
    • kv (Key-Value): A simple, static key-value store (like a secure .env file).
    • database: Connects to a database (like PostgreSQL) and generates *dynamic* credentials.
    • aws: Generates *dynamic*, temporary AWS IAM credentials.
  • Policies:** The "IAM" of Vault. The rulebook that defines *who* (which token) can access *what* (which secret path).
  • Sealing / Unsealing:** Vault's ultimate security feature. When a Vault server starts, it's **Sealed**. It knows *where* its data is (the storage backend), but it does **not** know *how* to decrypt it (the "encryption key"). The encryption key is split into 5 "Unseal Keys." To "unseal" Vault, 3 of the 5 key holders must enter their key. This prevents any single person (even the root admin) from accessing all secrets.

Part 3: Installation & Running in Dev Mode

The best way to learn Vault is to run it in **"-dev" mode**. This is a special, in-memory, auto-unsealed mode for development. **DO NOT** use this in production.

Step 1: Install Vault

Go to the official Vault website and download the binary for your OS. It's a single file.

# On macOS (using Homebrew)
$ brew tap hashicorp/tap
$ brew install vault

# On Linux (Ubuntu/Debian)
$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
$ sudo apt update && sudo apt install vault

Step 2: Run in Dev Mode

Open your terminal and run this one command:

$ vault server -dev

==> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Listener 1: tcp (addr: "127.0.0.1:8200", ...)
               Log Level: info
...
==> Vault server started!
...
Unseal Key: some-long-random-key
Root Token: hvs.SOME_VERY_SECRET_ROOT_TOKEN

...

This is critical. Vault is now running, and it has printed your two most important pieces of information:

  • Unseal Key:** In production, this would be 5 keys. In dev mode, it's just one.
  • Root Token:** This is the "root user" password. It has god-mode access to everything. **Copy this token!**

Step 3: Configure Your Shell

Open a **new, second terminal** (leave the server running). You need to tell your vault CLI (client) how to find the server and what your token is.

# Set the server address
$ export VAULT_ADDR="http://127.0.0.1:8200"

# Set your token (paste the Root Token from the server output)
$ export VAULT_TOKEN="hvs.SOME_VERY_SECRET_ROOT_TOKEN"

# 4. Check your status
$ vault status

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false  <-- (false means it's UNSEALED and ready)
Total Shares    1
Threshold       1
Version         1.15.0
...

You are now logged in as root and ready to issue commands.

Part 4: Storing Static Secrets (KV Engine)

The kv (Key-Value) engine is the simplest secrets engine. It's a secure "hash map" or "dictionary." You use it for data that doesn't change often, like an API key for a third-party service.

Dev mode enables `kv` (version 2) at the path secret/ by default.

Writing a Secret (`kv put`)

The command is vault kv put [path] [key=value]. The path secret/ is reserved for the `kv` engine.

# Store the AWS API keys for "my-app"
$ vault kv put secret/my-app/aws \
    aws_access_key_id="AKIA123456789" \
    aws_secret_access_key="MySuperSecretKey"

==== Data ====
Key              Value
---              -----
created_time     2025-11-10T14:30:00Z
deletion_time    n/a
destroyed        false
version          1

Reading a Secret (`kv get`)

The command is vault kv get [path].

$ vault kv get secret/my-app/aws

==== Metadata ====
Key              Value
---              -----
created_time     2025-11-10T14:30:00Z
...
version          1

==== Data ====
Key                      Value
---                      -----
aws_access_key_id        AKIA123456789
aws_secret_access_key    MySuperSecretKey

Versioning (KV v2)

By default, the kv engine is **versioned**. If you write a new secret to the same path, it creates a new version (v2) and "hides" the old one (v1). This is a lifesaver if you accidentally save a wrong password.

# Overwrite the secret
$ vault kv put secret/my-app/aws aws_secret_access_key="ANOTHER_NEW_KEY"

==== Data ====
Key              Value
---              -----
created_time     2025-11-10T14:35:00Z
...
version          2


# Get the *latest* version (v2)
$ vault kv get secret/my-app/aws

...
version          2
...
aws_secret_access_key    ANOTHER_NEW_KEY


# Get the *previous* version (v1)
$ vault kv get -version=1 secret/my-app/aws

...
version          1
...
aws_access_key_id        AKIA123456789
aws_secret_access_key    MySuperSecretKey

Deleting Secrets

# 'delete' just creates a new "deleted" version (soft delete)
$ vault kv delete secret/my-app/aws

# 'destroy' permanently destroys a specific version's data
$ vault kv destroy -versions=1 secret/my-app/aws

Part 5: Auth Methods & Policies (The "Real" Way)

You should **NEVER** give the **Root Token** to your applications or other users. The Root Token is for initial setup *only*.
The proper workflow is: 1. You (the admin) create a **Policy** (a set of rules). 2. You enable an **Auth Method** (like userpass). 3. You create a **User** (e.g., dev-user) and *attach* the Policy to them. 4. The `dev-user` then logs in and gets a *new, less-privileged token* that can only do what the Policy allows.

Step 1: Write a Policy (HCL)

Policies are written in **HCL (HashiCorp Configuration Language)**. They are "deny by default." You only add capabilities that you want to *allow*.

Let's create a "read-only" policy for our app.

my-app-policy.hcl
# This policy grants READ-ONLY access to the secrets
# at the path "secret/data/my-app/*"

path "secret/data/my-app/*" {
  capabilities = [ "read", "list" ]
}

# Note: For KV v2, you MUST add 'data/' after the engine path ('secret/')
  • path: The path of the secret you are controlling. The * is a wildcard.
  • capabilities: The list of allowed actions. The most common are create, read, update, delete, list, and sudo.

Step 2: Upload the Policy

# 'vault policy write'  
$ vault policy write my-app-readonly my-app-policy.hcl
Success! Uploaded policy: my-app-readonly

Step 3: Enable Auth Method & Create User

Let's create a simple username/password login for our app.

# 1. Enable the 'userpass' auth method
$ vault auth enable userpass
Success! Enabled userpass auth method at: userpass/

# 2. Create a new user named 'my-app-user' and attach our policy
$ vault write auth/userpass/users/my-app-user \
    password="my-strong-password" \
    policies="default,my-app-readonly"
Success! Data written to: auth/userpass/users/my-app-user

Step 4: Test it!

First, let's "un-set" our root token so we are logged out.

$ unset VAULT_TOKEN

# Now, try to read the secret. It will fail.
$ vault kv get secret/my-app/aws

Error...: missing client token: 403


# Now, log in as our new, less-privileged user
$ vault login -method=userpass username="my-app-user"

Password (will be hidden): *****************
Success! You are now authenticated. The token metadata is:
Key                  Value
---                  -----
token                hvs.A_BRAND_NEW_LIMITED_TOKEN
token_policies       [default, my-app-readonly]
token_duration       768h
...

A *new* token (hvs.A_BRAND_NEW_LIMITED_TOKEN) is automatically put in the VAULT_TOKEN environment variable. This new token *only* has the "my-app-readonly" policy.

# Let's try to read the secret we HAVE permission for:
$ vault kv get secret/my-app/aws

==== Data ====
Key                      Value
---                      -----
aws_access_key_id        AKIA123456789
...


# Now, let's try to write a secret (we DON'T have permission):
$ vault kv put secret/my-app/new-secret value="test"

Error writing data to secret/data/my-app/new-secret: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/secret/data/my-app/new-secret
Code: 403. Errors:
* 1 error occurred:
* permission denied

It worked! Our policy successfully blocked the write action. This is the core loop of DevSecOps.

Part 6: Dynamic Secrets (Vault's Killer Feature)

Static secrets (like our AWS key) are good, but they still live forever. The *best* security is a secret that doesn't exist until you need it, and disappears after you use it. This is **Dynamic Secrets**.

How it Works (Database Example)

We will configure Vault to be a "manager" for our PostgreSQL database. Our app will *never* have the admin password.
Instead, our app will say: "Hey Vault, I need to talk to the database."
Vault will *dynamically create* a *new* PostgreSQL user (e.g., v-app-abc123) with a *new* password, give it `READ ONLY` permissions, and set its expiration to **5 minutes**.
Your app uses this temporary credential. 5 minutes later, Vault automatically logs into PostgreSQL and *deletes* that user.
Now, even if your app's credential leaks, it's useless 5 minutes later.

Step 1: Enable the Database Secrets Engine

$ vault secrets enable database
Success! Enabled the database secrets engine at: database/

Step 2: Configure Vault's access to PostgreSQL

(This assumes you have a running PostgreSQL server and a "root" user named vaultadmin that Vault can use to create *other* users.)

$ vault write database/config/my-postgres-db \
    plugin_name="postgresql-database-plugin" \
    allowed_roles="my-app-role" \
    connection_url="postgresql://vaultadmin:pass@postgres-server:5432/mydb?sslmode=disable"

Step 3: Create a "Role" (The "Recipe" for new users)

This is the most important step. You write the SQL that Vault will use to create new users.

# Define a role named 'my-app-role'
$ vault write database/roles/my-app-role \
    db_name="my-postgres-db" \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
                         GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1m" \
    max_ttl="5m"
  • creation_statements: The SQL command. Vault will replace {{name}}, {{password}}, and {{expiration}} with new, random values.
  • default_ttl="1m": The secret expires in 1 minute (but the app can renew it).
  • max_ttl="5m": The secret can *never* live longer than 5 minutes.

Step 4: The App Workflow (Requesting a Secret)

Now, your app (which has a token with a policy allowing it to access database/creds/my-app-role) makes *this* call:

$ vault read database/creds/my-app-role

Key                Value
---                -----
lease_id           database/creds/my-app-role/abc-123
lease_duration     1m
lease_renewable    true
password           "A1b-super-random-password"
username           "v-app-my-app-role-123xyz"

Vault has just created a *brand new* user in PostgreSQL (v-app-my-app-role-123xyz) and given the credentials to your app. Your app now connects to the database *using this temporary user*.
When the 1-minute TTL is up, Vault automatically revokes the lease and runs the `revocation_statements` (which is DROP ROLE "{{name}}") in your database. The user is gone.
**This is the future of secrets management. Your app *never* touches a static, long-lived password.**

Read More about Dynamic Secrets →

Part 7: Vault in Kubernetes (The `Vault Agent`)

How does your *Pod* in Kubernetes log in to Vault to get secrets?
You use the **Vault Agent Injector**. This is a service you install in your K8s cluster. It watches for new Pods being created. When it sees a Pod with special "annotations" (metadata), it *automatically* injects two containers into that Pod:

  1. vault-agent:** An "init container" that runs *first*. It talks to Vault, authenticates (using the Pod's Kubernetes Service Account), gets the secrets, and writes them to a shared *in-memory volume* (/vault/secrets).
  2. vault-agent-sidecar:** A "sidecar container" that runs *alongside* your app. Its job is to keep renewing the secrets (like a dynamic password) before they expire.

Now, your main application container just reads its password from a file: /vault/secrets/db-password.txt. It has *no idea* Vault even exists. This is perfect separation of concerns.

my-app-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  ...
  template:
    metadata:
      # === These annotations trigger the Vault Agent ===
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'my-app-role' # The Vault role
        vault.hashicorp.com/agent-inject-secret-db-pass: 'database/creds/my-app-role'
        vault.hashicorp.com/agent-inject-template-db-pass: |
          {{- with secret "database/creds/my-app-role" -}}
          export DB_PASSWORD={{ .Data.password }}
          {{- end -}}
    spec:
      containers:
      - name: my-app-container
        image: my-app:v1
        # This script is run by the agent to load the secrets
        command: ["/bin/sh", "-c", "source /vault/secrets/db-pass && /app/start"]
Read More about Vault + Kubernetes Integration →

© 2025 CodeWithMSMAXPRO. All rights reserved.