HashiCorp Vault: Setup, Configuration, and Best Practices

Kubernetes Authentication

For services running in Kubernetes, Vault can authenticate pods using their service account tokens:

# Enable Kubernetes auth
vault auth enable kubernetes

# Configure the Kubernetes auth backend
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc" \
  kubernetes_ca_cert="@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

# Create a role that binds a Kubernetes service account to a Vault policy
vault write auth/kubernetes/role/my-app \
  bound_service_account_names="my-app-sa" \
  bound_service_account_namespaces="production" \
  policies="app-policy" \
  ttl="1h"

# Pod authenticates using its JWT token
# POST /v1/auth/kubernetes/login
curl --request POST \
  --data '{"jwt":"eyJhbGciOi...","role":"my-app"}' \
  https://vault.example.com/v1/auth/kubernetes/login

Policies and Access Control

Policies define what an authenticated entity can do within Vault. They use path-based matching and are written in HCL or JSON.

Writing Effective Policies

# Example: A policy for a web application that needs database credentials
# and access to specific secrets

# Allow listing and reading secrets under app/web
path "secret/data/app/web/*" {
  capabilities = ["read", "list"]
}

# Allow reading dynamic database credentials
path "database/creds/web-role" {
  capabilities = ["read"]
}

# Deny access to admin paths
path "secret/data/app/admin/*" {
  capabilities = ["deny"]
}

# Allow renewal of own tokens
path "auth/token/renew-self" {
  capabilities = ["update"]
}

# Allow looking up own token properties
path "auth/token/lookup-self" {
  capabilities = ["read"]
}

# Grant access to Transit encryption for app data
path "transit/encrypt/app-key" {
  capabilities = ["update"]
}
path "transit/decrypt/app-key" {
  capabilities = ["update"]
}

Policy Best Practices

• Least Privilege: Grant only the minimum permissions required. Start with nothing and add capabilities incrementally.
• Use Parameter Constraints: In Vault Enterprise, you can use Sentinel policies or parameter constraints to enforce fine-grained conditions (e.g., time-of-day access, source IP ranges).
• Separate Policies by Function: Instead of one monolithic policy, create multiple small policies and attach them together. This makes management and auditing easier.
• Regular Policy Audits: Review policies quarterly to ensure they still reflect actual access needs.

# Create a policy from a file
vault policy write web-app web-app-policy.hcl

# List all policies
vault policy list

# View a policy's rules
vault policy read web-app

# Attach policy to a token, AppRole, or other auth method
vault token create -policy="web-app"

# Test capabilities (useful for debugging)
vault token capabilities hvs.token_here secret/data/app/web/config
# Output: read, list

Auditing and Monitoring

Audit devices log every request and response to Vault, providing an immutable record of all access. This is critical for compliance and security investigations.

# Enable file audit logging
vault audit enable file file_path=/var/log/vault/audit.log

# Enable syslog audit
vault audit enable syslog tag="vault-audit" facility="AUTH"

# Check enabled audit devices
vault audit list

# Audit logs contain detailed entries:
# {
#   "type": "response",
#   "auth": {
#     "client_token": "hvs.xxx",
#     "policies": ["default", "web-app"],
#     "metadata": {"user": "jane-doe"}
#   },
#   "request": {
#     "path": "secret/data/app/web/config",
#     "operation": "read"
#   },
#   "response": {
#     "data": {"data": {"password": "***REDACTED***"}}
#   }
# }

Vault automatically redacts sensitive data in audit logs. For enhanced monitoring, integrate Vault with telemetry systems:

# Add telemetry stanza to configuration
telemetry {
  statsite_address = "statsite.example.com:8125"
  disable_hostname = false
  enable_hostname_label = true
  prometheus_retention_time = "60s"
}

# Key metrics to monitor:
# - vault.core.unsealed (gauge)
# - vault.core.handle_request (counter)
# - vault.core.handle_request..
# - vault.token.create.
# - vault.expire.num_leases

Vault Agent and Sidecar Pattern

For applications that cannot natively integrate with Vault's API, the Vault Agent provides automatic authentication and secret retrieval, writing secrets to a template file or environment variable source.

# vault-agent.hcl
pid_file = "/var/run/vault-agent.pid"

auto_auth {
  method {
    type = "approle"
    config = {
      role_id_file_path   = "/etc/vault/role-id"
      secret_id_file_path = "/etc/vault/secret-id"
      remove_secret_id_file_after_reading = true
    }
  }
}

cache {
  use_auto_auth_token = true
}

listener "tcp" {
  address = "127.0.0.1:8100"
  tls_disable = true
}

template {
  source      = "/etc/vault/templates/app-config.ctmpl"
  destination = "/etc/myapp/config.yaml"
  command     = "systemctl reload myapp"
}

# template/app-config.ctmpl
# {{- with secret "secret/data/app/database" }}
# database:
#   username: {{ .Data.data.username }}
#   password: {{ .Data.data.password }}
# {{- end }}
# {{- with secret "database/creds/readonly" }}
# db_creds:
#   username: {{ .Data.username }}
#   password: {{ .Data.password }}
# {{- end }}

Run the agent alongside your application (sidecar pattern in Kubernetes):

vault agent -config=/etc/vault/vault-agent.hcl

The agent automatically authenticates, keeps the token renewed, fetches secrets, renders templates, and optionally runs commands on changes. This is the recommended pattern for legacy applications that expect secrets in files or environment variables.

High Availability and Disaster Recovery

HA Clustering with Integrated Storage (Raft)

Vault's integrated Raft storage provides both data persistence and HA clustering without external dependencies:

# Node 1 configuration
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "node1"

  retry_join {
    leader_api_addr = "https://node2.example.com:8200"
    leader_ca_cert_file = "/etc/vault/tls/ca.crt"
  }
  retry_join {
    leader_api_addr = "https://node3.example.com:8200"
    leader_ca_cert_file = "/etc/vault/tls/ca.crt"
  }
}

# After starting node1, join additional nodes
# On node2:
vault operator raft join https://node1.example.com:8200

# Check cluster status
vault operator raft list-peers

# Node       Address                    State     Age
# node1      node1.example.com:8200     leader    5d
# node2      node2.example.com:8200     follower  2d
# node3      node3.example.com:8200     follower  1d

Disaster Recovery (Vault Enterprise)

For multi-datacenter deployments, Vault Enterprise provides Disaster Recovery (DR) replication that asynchronously replicates secrets and policies to secondary clusters. In the event of a primary cluster failure, you can promote a DR secondary to primary.

Backup and Restore

Always maintain regular backups of Vault's underlying storage or use the Raft snapshot functionality:

# Create a Raft snapshot
vault operator raft snapshot save vault-backup-2024-01-01.snap

# Restore from a snapshot (on a fresh cluster)
vault operator raft snapshot restore vault-backup-2024-01-01.snap

# Verify snapshot integrity
vault operator raft snapshot verify vault-backup-2024-01-01.snap

Store snapshots in encrypted, access-controlled storage with strict retention policies. Test your restore procedure regularly — an untested backup is not a backup.

Best Practices Summary

Architecture and Deployment

• Use Auto-Unseal: Leverage cloud KMS or HSM for automatic unsealing. Manual unseal is error-prone and delays recovery.
• Run in HA Mode: Deploy at least three nodes in a cluster behind a load balancer. This ensures availability during node failures or maintenance.
• Enable TLS Everywhere: Never expose Vault over plain HTTP. Use strong TLS certificates and enforce mTLS between cluster nodes.
• Isolate the Vault Network: Place Vault in a restricted network segment with strict firewall rules. Only allow access from authorized services and administrators.

Secret Management

• Prefer Dynamic Secrets: Whenever possible, use dynamic secrets (database credentials, cloud provider tokens) over static secrets. This ensures automatic rotation and revocation.
• Use Short TTLs: Set short default and maximum TTLs for credentials. If a service can function with 5-minute credentials, don't give it 24-hour credentials.
• Wrap Sensitive Values: Use response wrapping to securely distribute initial credentials, tokens, or encryption keys without exposing them in transit.
• Version Secrets: Use KV v2 for storing configuration secrets. Versioning allows rollback and audit of changes over time.

Access Control

• Apply Least Privilege Rigorously: Start with empty policies and add capabilities one at a time. Never use the root token for day-to-day operations.
• Use AppRole or Kubernetes Auth for Machines: Avoid distributing long-lived tokens to services. AppRole with short-lived Secret IDs or Kubernetes auth with bound service accounts are far more secure.
• Rotate Root Credentials: Regenerate root tokens and recovery keys periodically. Implement a formal key ceremony process.
• Separate Human and Machine Access: Use OIDC/LDAP auth for human operators with short TTLs, and AppRole/Kubernetes auth for services with appropriate constraints.

Operational Excellence

• Enable Audit Logging: Always enable at least one audit device. Store audit logs in a tamper-proof, append-only storage system. Monitor them for anomalies.
• Monitor Key Metrics: Track seal status, request rates, error rates, lease counts, and token creation events. Alert on any unexpected changes.
• Test Recovery Procedures: Regularly test unsealing, snapshot restore, and DR promotion procedures. Document them and train your team.
• Upgrade Carefully: Follow HashiCorp's upgrade guides. Always back up before upgrading. Test upgrades in a staging environment first.
• Implement Namespaces (Enterprise): Use namespaces to isolate teams, environments, or tenants within a single Vault cluster. This provides strong multi-tenancy without managing separate clusters.

Developer Workflow

• Never Hardcode Secrets: All secrets should come from Vault at runtime. Use Vault Agent or native SDKs to fetch secrets dynamically.
• Use Environment Variables or Files: For applications that cannot call Vault's API, use Vault Agent templates to write secrets to a well-known file or environment source.
• Handle Lease Expiry: Applications must handle credential expiration gracefully. Implement retry logic and credential refresh in your Vault client code.
• Integrate with CI/CD: Use Vault's API or CLI in CI/CD pipelines to retrieve build secrets, signing keys, or deployment credentials. Never store these in pipeline variables.

Example: Complete Application Integration Workflow

Here's a complete end-to-end example of integrating a Python application with Vault using the hvac library:

# Install hvac
pip install hvac

# app.py
import hvac
import psycopg2
import time
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class VaultClient:
    def __init__(self, vault_addr, role_id, secret_id):
        self.client = hvac.Client(url=vault_addr)
        self.role_id = role_id
        self.secret_id = secret_id
        self.token = None
        self.token_expiry = 0

    def authenticate(self):
        """Authenticate using AppRole and obtain a token."""
        response = self.client.auth.approle.login(
            role_id=self.role_id,
            secret_id=self.secret_id,
            use_token=True
        )
        self.token = response['auth']['client_token']
        self.client.token = self.token
        self.token_expiry = time.time() + response['auth']['lease_duration'] - 60
        logger.info("Successfully authenticated with Vault")
        return self.token

    def ensure_token_valid(self):
        """Check if token needs refresh."""
        if time.time() > self.token_expiry:
            logger.info("Token expired or expiring soon, re-authenticating...")
            self.authenticate()

    def get_database_credentials(self):
        """Retrieve dynamic database credentials."""
        self.ensure_token_valid()
        response = self.client.secrets.database.generate_credentials(
            name='readonly',
            mount_point='database'
        )
        return response['data']['username'], response['data']['password']

    def get_app_secret(self, path):
        """Retrieve a secret from KV v2 store."""
        self.ensure_token_valid()
        response = self.client.secrets.kv.v2.read_secret_version(
            path=path,
            mount_point='secret'
        )
        return response['data']['data']

def connect_to_database(vault):
    """Establish database connection using dynamic credentials."""
    username, password = vault.get_database_credentials()
    conn = psycopg2.connect(
        host='postgres.example.com',
        database='appdb',
        user=username,
        password=password,
        connect_timeout=10
    )
    logger.info(f"Connected to database with user: {username}")
    return conn

def main():
    vault = VaultClient(
        vault_addr='https://vault.example.com',
        role_id='ROLE_ID_FROM_ENV',
        secret_id='SECRET_ID_FROM_ENV'
    )
    
    # Initial authentication
    vault.authenticate()
    
    # Get application configuration
    config = vault.get_app_secret('app/config')
    logger.info(f"Loaded config: {list(config.keys())}")
    
    # Get database credentials and connect
    conn = connect_to_database(vault)
    
    # Perform database operations...
    cursor = conn.cursor()
    cursor.execute("SELECT count(*) FROM users")
    count = cursor.fetchone()[0]
    logger.info(f"User count: {count}")
    
    cursor.close()
    conn.close()

if __name__ == "__main__":
    main()

This example demonstrates proper credential lifecycle management: authenticating with AppRole, handling token renewal transparently, fetching dynamic database credentials that auto-expire, and retrieving static configuration secrets — all without a single hardcoded secret.

Conclusion

HashiCorp Vault is not merely a secret store — it is a comprehensive secrets lifecycle management platform that transforms how organizations handle sensitive data. By centralizing secrets, enforcing identity-based access control, automating credential rotation, and providing complete auditability, Vault eliminates secret sprawl and dramatically reduces the risk of credential compromise.

The journey from initial setup to full production deployment requires careful planning: choose the right storage backend, implement auto-unseal for operational simplicity, design granular policies following least privilege, prefer dynamic secrets over static ones, and integrate applications through Vault Agent or native SDKs with proper lease renewal handling. The practices outlined in this tutorial — from HA clustering to regular backup testing, from response wrapping to audit log monitoring — form the foundation of a robust, secure, and maintainable Vault deployment.

As your infrastructure grows, Vault scales with you. Its plugin ecosystem, extensible auth methods, and enterprise features like DR replication and namespaces ensure that your secrets management remains centralized, auditable, and secure — whether you're running a single application or orchestrating thousands of services across multiple clouds. Start with development mode to explore, move to a properly initialized production cluster with TLS and HA, adopt the best practices incrementally, and build a culture where secrets are never hardcoded, always auditable, and automatically rotated. Your security posture will be stronger for it.