Create a secure vault-backed secret fetch with rotation and break-glass.
- User
Create a secure vault-backed secret fetch with rotation and break-glass.
- Assistant
We are going to design a system for fetching secrets from a secure vault (like HashiCorp Vault) with rotation and break-glass capabilities. The system will have:
- A secret fetcher that retrieves the current secret from the vault and caches it for a short period to avoid overloading the vault.
- Automatic rotation: the secret will be rotated periodically (by the vault or by this service if we are managing rotation).
- Break-glass: in case of emergency, an authorized user can force a rotation and immediately fetch the new secret, bypassing the cache and potentially the rotation schedule.
However, note that in a typical setup, rotation is often managed by the vault itself (via lease renewal and revocation) or by the application that uses the secret (by requesting a new one when the old one expires).
But let's assume we are building a middleware that sits between the application and the vault, and we want to add:
- Caching to reduce vault calls.
- Break-glass: an endpoint to force a rotation and clear the cache, so the next fetch gets a new secret.
Steps for the secret fetcher:
- We'll have a configuration for the vault address, token, path to the secret, and cache duration.
- We'll maintain an in-memory cache (or distributed cache if multiple instances) of the secret and its expiration time.
- When fetching:
- If cache is valid (not expired), return the cached secret.
- Else, fetch a new secret from the vault, update the cache, and return the secret.
- We'll also have an endpoint (break-glass) that:
- Invalidates the cache (so the next fetch will get a new secret).
- Optionally, we can also trigger a rotation in the vault (if the vault supports it) or just let the next fetch get a new secret because the old one might be revoked.
However, note: if the vault is configured to rotate secrets automatically (e.g., via dynamic secrets or lease renewal), then we don't need to manage rotation in our service. But if we are managing static secrets and need to rotate them on a schedule, we might have a separate process that writes a new secret to the vault and then we want to fetch the new one.
Alternatively, we can design two modes: Mode 1: The vault manages rotation (dynamic secrets or lease-based). We just fetch and cache until the lease expires. Mode 2: We manage rotation by having a scheduled job that writes a new secret to the vault and then break-glass or cache invalidation to pick it up.
But for simplicity, let's assume the vault does not automatically rotate and we are managing a static secret that we want to rotate periodically (e.g., every 30 days).
How about:
- We have a secret stored in the vault at a known path (e.g.,
secret/data/myapp). - We have a rotation schedule (e.g., every 30 days) that writes a new secret to that path (via a separate process, maybe a CI/CD job or a cron job).
- Our service will cache the secret for, say, 1 hour (or less) to avoid hitting the vault on every request.
But then, when the rotation happens, we want the service to pick up the new secret as soon as possible? Or we can wait until the cache expires?
We can have:
- The rotation process (external) writes the new secret to the vault and then calls our break-glass endpoint to invalidate the cache immediately.So:
Steps for normal operation: 1. Check cache -> if valid, return. 2. Else, fetch from vault, update cache, return. Break-glass: 1. Invalidate the cache (so next fetch will get from vault). 2. Optionally, we can also log the event and require authentication/authorization.But note: if we are caching for 1 hour and the secret is rotated, then for up to 1 hour the old secret might still be used until cache expires and we fetch the new one.
So we might want to set the cache duration to be less than the rotation interval, but then we hit the vault more often.
Alternatively, we can have the rotation process also notify our service (via break-glass) so that we invalidate the cache and fetch the new one immediately.
So the break-glass endpoint is used by the rotation process to force a cache invalidation.
Also, we might want to have a way for an operator to manually force a rotation (break-glass) in case of a suspected compromise.
In that case, the operator would:
1. Call the break-glass endpoint (which invalidates the cache). 2. Then, the next fetch will get the current secret from the vault. But if we haven't rotated the secret in the vault, we get the same one.So we need two things for break-glass in case of compromise:
a) Invalidate the cache (so we stop using the old secret in memory, but if the vault still has the old secret, we get the same). b) Actually rotate the secret in the vault (write a new one). This could be done by the same operator or an automated process.Therefore, the break-glass endpoint might be split into:
- `POST /break-glass/cache` -> invalidates cache. - `POST /break-glass/rotate` -> invalidates cache AND triggers a rotation (if we have a way to generate a new secret and write to the vault).But generating a new secret might be application-specific. So perhaps:
We have: 1. `POST /break-glass/invalidate` -> just invalidate the cache. 2. `POST /break-glass/rotate` -> which does: a. Generate a new secret (using a random generator, or by calling some secret generation service). b. Write the new secret to the vault. c. Invalidate the cache.However, note that writing to the vault requires appropriate permissions and the new secret must be in the format the vault expects.
Alternatively, we might not want our service to write to the vault (separation of duties). Then:
- The break-glass/invalidate just invalidates the cache, and then an operator must manually write a new secret to the vault (or an automated rotation process does it).But then the break-glass/rotate would be:
- Invalidate cache. - Trigger an external process (via a message, or by calling an API) that writes a new secret to the vault.Given the complexity and security concerns, it might be better to separate:
- Our service only reads from the vault and caches. - Rotation is done by a separate, more privileged process that writes to the vault. - Break-glass in our service only invalidates the cache, and then the rotation process (or operator) writes a new secret.But then, if we want a single endpoint for break-glass that does both, we need to have write access to the vault in our service, which might be a security risk.
Let's design:
We have two roles: 1. Secret Fetcher (our service) with read-only access to the vault. 2. Secret Rotator (a separate service or process) with write access to the vault. Then: - Our service provides: GET /secret -> returns the current secret (cached). POST /break-glass -> invalidates the cache (so next GET gets from vault) and optionally triggers the rotator (via an internal event or by calling the rotator's API). - The rotator service: - Can be triggered by our break-glass endpoint (if we have an internal event bus or direct API call) or by a schedule. - Generates a new secret and writes to the vault. But then our break-glass endpoint in the fetcher service would: Step 1: Invalidate the cache. Step 2: If configured to do so, trigger the rotator (by publishing an event or calling the rotator's endpoint). However, if the rotator fails, then the next fetch will get the old secret (because the vault wasn't updated). So we might want to wait until the rotator succeeds? But that would make the break-glass endpoint synchronous and potentially slow.Alternatively, we can have:
Break-glass does: 1. Invalidate cache. 2. Return immediately, and then the rotator (which might be triggered asynchronously) will write a new secret. Then the next fetch might get the old secret until the rotator finishes and writes the new one? But then the cache is invalidated, so we will fetch from the vault. If the rotator hasn't written yet, we get the old one and cache it again.So we need to coordinate:
We want break-glass to: - Immediately stop using the cached secret (invalidate cache). - Then, we want to ensure that the vault is updated with a new secret as soon as possible. But if the rotator is asynchronous, then there is a window where we fetch the old secret from the vault (because the rotation hasn't happened yet) and cache it again. So maybe: Break-glass should: 1. Invalidate cache. 2. Trigger the rotator and wait for it to finish (synchronously) so that when we invalidate the cache and then the next fetch happens, the vault already has the new secret. But then the break-glass endpoint becomes slow (if rotator takes time) and might time out.Given the complexity, maybe we split:
Endpoint 1: Invalidate cache (fast, returns immediately). Endpoint 2: Rotate (calls the rotator and waits, then invalidates cache?). Then the operator would: Call /break-glass/rotate -> which does: a. Call rotator to generate and write new secret to vault. b. Invalidate cache (so next fetch gets the new one). But note: the rotator might write the new secret and then we invalidate the cache. Then the next fetch (which might be concurrent) will get the new one. However, what if the rotator fails? Then we don't invalidate the cache? Or we invalidate anyway? If we invalidate and the rotator fails, then the next fetch will get the old secret (because the vault wasn't updated) and cache it again. So: We want to invalidate the cache only if the rotation succeeded. So: POST /break-glass/rotate: try: call rotator -> if success, invalidate cache and return success. else, return error, do not invalidate cache. Then the operator can retry. But then, if the rotator succeeds, we invalidate the cache and the next fetch gets the new secret. However, note that the rotator might write the new secret but we don't know when the vault update is visible? We assume it's immediate.Given the above, let's design:
We have: - A cache (with a TTL, but we also manually invalidate). - A vault client (read-only for the fetcher). - A rotator client (if we are going to call the rotator from the break-glass/rotate endpoint). But note: the fetcher service should not have write access to the vault? Then we cannot have the rotator in the same service if we want to separate duties. So: Option A: Fetcher and rotator are separate services. Fetcher: - GET /secret -> returns secret (cached or fresh from vault). - POST /break-glass/invalidate -> invalidates cache. - POST /break-glass/rotate -> calls rotator service and on success invalidates cache. Rotator: - POST /rotate -> generates new secret and writes to vault. Option B: Fetcher and rotator are the same service, but then we have to give it write access to the vault. For security, Option A is better.Let's go with Option A.
Implementation:
Fetcher Service: Cache: secret: string expires_at: timestamp Configuration: vault_address, vault_token, secret_path, cache_ttl (e.g., 1 hour) On startup: load secret from vault and set cache. GET /secret: if current time < expires_at: return secret from cache. else: secret = read_from_vault() update cache (secret and new expires_at = now + cache_ttl) return secret. POST /break-glass/invalidate: invalidate cache (set expires_at to 0 or a past time). POST /break-glass/rotate: call rotator service's /rotate endpoint (with appropriate authentication). if rotator returns success: invalidate cache (so next GET will fetch the new secret from vault). return success. else: return error. Rotator Service: Configuration: vault_address, vault_token, secret_path POST /rotate: generate_new_secret() (or get from a secure generator, or from a KMS, etc.) write_to_vault(new_secret) return success.But note: the rotator must have write access to the vault at secret_path.
Also, we must consider:
- The fetcher service must have read access to the vault. - The rotator service must have write access.Security:
- Both services must run with minimal privileges. - The break-glass endpoints must be protected (authentication and authorization). Only authorized operators can call them.However, note that the break-glass/rotate endpoint in the fetcher service will call the rotator service, so we must secure the communication between them (mutual TLS or network isolation).
Alternatively, we can have:
The fetcher service does not call the rotator service. Instead, the operator calls the rotator service first, then calls the fetcher's break-glass/invalidate. Then: Operator does: 1. Call rotator service /rotate -> writes new secret to vault. 2. Call fetcher service /break-glass/invalidate -> invalidates cache. This way, the fetcher service doesn't need to know about the rotator service. But then, if the rotator fails, the operator must handle it and not call invalidate. So we can have: Fetcher service only has: GET /secret POST /break-glass/invalidate And the rotator service is separate and called by the operator or by a scheduled job. Then break-glass for rotation is a two-step process: Step 1: Rotate (by calling rotator service). Step 2: Invalidate cache (by calling fetcher service). But if we want a single endpoint for break-glass that does both, we need the fetcher to call the rotator. Given the principle of separation of duties, it's better to keep them separate. So we design: - Fetcher: read-only vault access, cache, and invalidate. - Rotator: write-only vault access, and rotation logic. Operators (or an automated system) must call both in sequence. But then, if an operator forgets to invalidate the cache, the fetcher will continue to use the old secret until cache expires. So we can have the rotator also send a message (e.g., to a message bus) that the secret was rotated, and the fetcher can listen and invalidate the cache. Then: Rotator: - Writes new secret to vault. - Publishes an event: "secret_rotated" with the path. Fetcher: - Subscribes to "secret_rotated" events for its secret_path and invalidates the cache when received. This way, the operator only needs to call the rotator, and the fetcher will automatically invalidate the cache. But then we have to have a message bus and the fetcher must be always listening. Alternatively, the rotator can call the fetcher's invalidate endpoint (if we allow the rotator to call it, then we must secure that endpoint to only accept calls from the rotator). Let's do: Rotator: - Writes new secret to vault. - Then, calls fetcher's invalidate endpoint (with a shared secret or mutual TLS) to invalidate the cache. This way, the operator only calls the rotator, and the rotator takes care of invalidating the cache. But then the rotator must know the fetcher's endpoint. We can have: The rotator reads the fetcher's endpoint from configuration. So: Operator -> Rotator (writes new secret and then invalidates fetcher's cache) -> Fetcher (cache invalidated, next fetch gets new secret). And for break-glass in case of compromise: Operator calls the rotator's /rotate endpoint (which does the above) and that's it. But what if we want to break-glass without rotating? (i.e., just invalidate the cache because we suspect the cache is compromised? But the cache is in memory, so if the server is compromised, we have bigger problems.) So maybe: We have two break-glass scenarios: 1. We suspect the secret in the vault is compromised -> we rotate (call rotator) and then invalidate cache. 2. We suspect the cache is stale (but the vault secret is still good) -> we just invalidate the cache. But if the cache is stale, we can just wait for it to expire. So scenario 2 is not common. So we can have: - A single break-glass endpoint that does rotation and cache invalidation. And if we only want to invalidate the cache (without rotating), we can have a separate endpoint, but that might be less necessary. Given the above, let's design: Fetcher Service: GET /secret POST /break-glass/invalidate (protected, called by rotator or operator) Rotator Service: POST /rotate 1. Generate new secret. 2. Write to vault. 3. Call fetcher's /break-glass/invalidate (if configured to do so, and if the write succeeded). 4. Return success or failure. Then, the operator only needs to call the rotator's /rotate for a break-glass rotation. And if the operator only wants to invalidate the cache (without rotating), they can call the fetcher's invalidate directly. But note: the rotator must have network access to the fetcher's invalidate endpoint. Alternatively, we can have the fetcher listen to a message from the rotator (as above) to avoid the rotator having to know the fetcher's endpoint. But then we need a message bus. Given the complexity of setting up a message bus, let's go with the rotator calling the fetcher's invalidate endpoint. We'll secure the fetcher's invalidate endpoint to only accept calls from the rotator (by IP, or by a shared token, or mutual TLS).Summary:
Fetcher: Cache: secret: string expires_at: time.Time Config: vault_addr, vault_token, secret_path, cache_ttl, invalidate_endpoint_secret (or TLS client cert) GET /secret: if time.Now().Before(expires_at) { return secret } secret = vault.Read(secret_path) expires_at = time.Now().Add(cache_ttl) return secret POST /break-glass/invalidate: // Check authentication (e.g., validate a token or client cert that only rotator has) expires_at = time.Time{} // zero time, so cache is invalid. Rotator: Config: vault_addr, vault_token, secret_path, fetcher_invalidate_url, fetcher_invalidate_secret (or client cert for mutual TLS) POST /rotate: new_secret = generate_secret() vault.Write(secret_path, new_secret) // Now call fetcher invalidate call_fetcher_invalidate() return OK call_fetcher_invalidate(): http.Post(fetcher_invalidate_url, with authentication)This design:
- Allows the fetcher to cache and reduce vault reads. - Allows the rotator to rotate and then invalidate the cache so the next fetch gets the new secret. - The break-glass is a call to the rotator's /rotate, which does both.However, what if the call to invalidate fails? Then the cache might still have the old secret until it expires.
We can have the rotator retry the invalidate call a few times, and if it still fails, log an error and maybe alert.
Also, note that the fetcher's invalidate endpoint must be highly available and secure.
Another consideration: the fetcher's cache TTL. We set it to, say, 1 hour. Then if the rotator fails to invalidate, we have at most 1 hour of old secret.
But if the rotation is critical (break-glass), we want to make sure the cache is invalidated. So we might want the rotator to retry and if it still fails, maybe try again later? Or we can have the rotator write a flag in the vault that the secret was rotated, and the fetcher can check that flag on every fetch (but that would be an extra read).
Alternatively, we can have the fetcher check the vault for a "rotation version" or timestamp and compare with the cache. But that would be an extra read on every fetch, which we are trying to avoid.
Given the trade-offs, we'll accept that if the invalidate call fails, we might have a stale cache for up to the TTL.
We can also have the fetcher, when it reads from the vault, check if the secret has changed (by comparing with the cached one) and if changed, update the cache even if the TTL hasn't expired? That would be an extra comparison but not an extra vault read.
But then we are reading from the vault on every fetch when the cache is valid? No, we only read when the cache is expired.
So we are back to: if we want to pick up a new secret immediately without waiting for cache expiration, we need to invalidate the cache.
So the invalidate call is critical.
We can also have the rotator write the new secret and then set a short TTL on the fetcher's cache? But the fetcher's TTL is fixed.
Given the above, we'll stick with the rotator calling the invalidate endpoint.
Let's code the two services:
Fetcher Service:
We'll use Go for example, but the design is language-agnostic.
Steps for fetcher:
- Initialize vault client. - Load initial secret and set cache. - Start HTTP server with two endpoints.But note: the fetcher must be able to handle multiple instances? If we have multiple instances of the fetcher, each has its own cache. Then the rotator must call the invalidate endpoint on every instance? Or we use a distributed cache (like Redis) so that all instances share the same cache.
Given the complexity, let's assume a single instance for now. For multiple instances, we would need a distributed cache and the invalidate would clear the distributed cache.
We'll design for a single instance, but note that for production we might need to scale.
Alternatively, we can have the fetcher service be a library and the application uses it, then we don't have a network endpoint for the fetcher? But then how does the rotator invalidate?
We are designing a service, so we assume the fetcher is a standalone service that provides an API.
But if we have multiple instances, we need to invalidate all of them.
So we have two options:
1. Use a distributed cache (like Redis) and store the secret and expiration there. Then the invalidate endpoint clears the Redis key. All instances read from the same Redis key. 2. Have the rotator call the invalidate endpoint on every fetcher instance (if we have a fixed set of instances, or via a load balancer that broadcasts?).Option 1 is better for dynamic scaling.
Let's change the fetcher to use Redis for cache:
Cache: key: "secret:myapp" value: the secret string expiration: set with EXPIRE (so Redis automatically removes it after TTL) Then: GET /secret: secret, err := redis.Get(key) if err == nil and secret != "" { return secret } else { // cache miss or expired secret = vault.Read(...) redis.Set(key, secret, cache_ttl) return secret } POST /break-glass/invalidate: redis.Del(key) Then the rotator only needs to call the invalidate endpoint of one fetcher instance (or the load balancer) and it will clear the Redis cache, so all instances will fetch fresh on next request. But note: if we have multiple fetcher instances, they all use the same Redis, so invalidating one key in Redis invalidates for all. Then we don't need the rotator to call every instance.So we change:
Fetcher uses Redis for cache. Rotator calls the fetcher's invalidate endpoint (which deletes the Redis key) after a successful rotation.Now, the fetcher service can be scaled horizontally.
Let's code accordingly.
However, note: the fetcher service must have access to Redis and Vault.
Steps:
Fetcher: type Fetcher struct { vaultClient *vault.Client redisClient *redis.Client secretPath string cacheTTL time.Duration } func (f *Fetcher) GetSecret() (string, error) { // Try Redis secret, err := f.redisClient.Get("secret").Result() if err == nil && secret != "" { return secret, nil } // Redis miss, fetch from vault secret, err = f.vaultClient.Read(f.secretPath) if err != nil { return "", err } // Set in Redis with TTL f.redisClient.Set("secret", secret, f.cacheTTL) return secret, nil } func (f *Fetcher) InvalidateCache() { f.redisClient.Del("secret") } HTTP handlers: GET /secret -> calls GetSecret and returns. POST /break-glass/invalidate -> calls InvalidateCache. Rotator: type Rotator struct { vaultClient *vault.Client fetcherURL string fetcherAuthToken string // or use mutual TLS } func (r *Rotator) Rotate() error { newSecret := generateSecret() err := r.vaultClient.Write(r.secretPath, newSecret) if err != nil { return err } // Invalidate fetcher cache req, _ := http.NewRequest("POST", r.fetcherURL+"/break-glass/invalidate", nil) req.Header.Add("Authorization", "Bearer "+r.fetcherAuthToken) client := &http.Client{} resp, err := client.Do(req) if err != nil { return err } if resp.StatusCode != 200 { return fmt.Errorf("invalidate failed: %s", resp.Status) } return nil } HTTP handler: POST /rotate -> calls Rotate() and returns.Security:
- The fetcher's invalidate endpoint must be protected. We use a shared token (Bearer) or mutual TLS. - The rotator must have the token or client cert. Also, the fetcher's GET /secret should be protected so that only authorized applications can fetch the secret? Or we assume the fetcher is internal and the network is trusted? We should also protect the GET endpoint. We can use the same authentication for both endpoints, but the rotator is the only one that should call invalidate. So: Fetcher: GET /secret: require authentication (e.g., API key) for the applications that need the secret. POST /break-glass/invalidate: require a different authentication (e.g., a break-glass token) that only the rotator has. We can have: For GET /secret: check for a header "X-API-Key" (or use mTLS with client certs for the applications). For POST /break-glass/invalidate: check for a header "X-Break-Glass-Token". Then: The applications that need the secret have the API key. The rotator has the break-glass token. And the rotator does not need the API key to call invalidate.But note: the rotator must have network access to the fetcher's invalidate endpoint.
We assume they are in the same network.
This design:
- Uses Redis for distributed caching among multiple fetcher instances. - Uses Vault as the secret store. - Has a rotator service that rotates the secret and then invalidates the cache.Break-glass:
- The operator calls the rotator's /rotate endpoint (with appropriate authentication) to rotate the secret and invalidate the cache.If the operator only wants to invalidate the cache (without rotating), they can call the fetcher's /break-glass/invalidate directly (with the break-glass token).
But note: if the operator invalidates the cache without rotating, then the next fetch will get the current secret from the vault (which might be old) and cache it again. So that doesn't help if the secret is compromised in the vault. So the operator should rotate and then invalidate.
Therefore, the primary break-glass is the rotator's /rotate.
We can also have the rotator's /rotate endpoint be idempotent? If called multiple times, it rotates again.
But note: the rotator must generate a new secret every time.
Now, what about the initial deployment?
- We must have an initial secret in the vault. - The fetcher will read it and cache it. - The rotator will write a new one and invalidate the cache.Also, we must secure the vault tokens:
- The fetcher's vault token must have read access to the secret path. - The rotator's vault token must have write access to the secret path.And we must rotate the vault tokens themselves periodically (but that's a separate concern).
This is a high-level design. We must also consider:
- Error handling and logging. - Monitoring (cache hits/misses, rotation events, etc.) - Alerting if the rotator fails or the invalidate call fails.Let's write some pseudo-code for the two services.
We'll assume:
- Vault client is the official HashiCorp Vault client. - Redis client is go-redis.Fetcher Service:
Note: We are not handling the Vault token renewal (if using a token that expires). We assume the token is static or we have a way to renew it.
Similarly for the rotator.
We might want to use a Vault agent or AppRole for better security.
But for simplicity, we use tokens.
Code structure:
main.go for fetcher: func main() { // Load config (from env or file) vaultToken := os.Getenv("VAULT_TOKEN") vaultAddr := os.Getenv("VAULT_ADDR") secretPath := os.Getenv("SECRET_PATH") redisAddr := os.Getenv("REDIS_ADDR") cacheTTL, _ := time.ParseDuration(os.Getenv("CACHE_TTL")) // Init vault client vaultConfig := api.DefaultConfig() vaultConfig.Address = vaultAddr vaultClient, err := api.NewClient(vaultConfig) if err != nil { log.Fatal(err) } vaultClient.SetToken(vaultToken) // Init redis redisClient := redis.NewClient(&redis.Options{ Addr: redisAddr, Password: "", // no password set DB: 0, // use default DB }) fetcher := &Fetcher{ vaultClient: vaultClient, redisClient: redisClient, secretPath: secretPath, cacheTTL: cacheTTL, } // HTTP handlers http.HandleFunc("/secret", fetcher.handleGetSecret) http.HandleFunc("/break-glass/invalidate", fetcher.handleInvalidate) log.Fatal(http.ListenAndServe(":8080", nil)) } type Fetcher struct { vaultClient *api.Client redisClient *redis.Client secretPath string cacheTTL time.Duration } func (f *Fetcher) handleGetSecret(w http.ResponseWriter, r *http.Request) { // Check API key apiKey := r.Header.Get("X-API-Key") if apiKey != expectedAPIKey { http.Error(w, "Unauthorized", http.StatusUnauthorized) return } secret, err := f.GetSecret() if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } fmt.Fprint(w, secret) } func (f *Fetcher) handleInvalidate(w http.ResponseWriter, r *http.Request) { // Check break-glass token token := r.Header.Get("X-Break-Glass-Token") if token != breakGlassToken { http.Error(w, "Unauthorized", http.StatusUnauthorized) return } f.InvalidateCache() w.WriteHeader(http.StatusOK) } func (f *Fetcher) GetSecret() (string, error) { // Try redis secret, err := f.redisClient.Get("secret").Result() if err == nil && secret != "" { return secret, nil } // Redis miss or error, fetch from vault secret, err = f.readFromVault() if err != nil { return "", err } // Set in redis f.redisClient.Set("secret", secret, f.cacheTTL) return secret, nil } func (f *Fetcher) readFromVault() (string, error) { secret, err := f.vaultClient.Logical().Read(f.secretPath) if err != nil { return "", err } data, ok := secret.Data["data"].(map[string]interface{}) if !ok { return "", fmt.Errorf("invalid secret data") } // Assuming the secret is stored as a string under key "value"? // Adjust according to your vault secret structure. value, ok := data["value"].(string) if !ok { return "", fmt.Errorf("secret not found") } return value, nil } func (f *Fetcher) InvalidateCache() { f.redisClient.Del("secret") }Rotator Service:
main.go for rotator: func main() { vaultToken := os.Getenv("VAULT_TOKEN") vaultAddr := os.Getenv("VAULT_ADDR") secretPath := os.Getenv("SECRET_PATH") fetcherURL := os.Getenv("FETCHER_URL") fetcherBreakGlassToken := os.Getenv("FETCHER_BREAK_GLASS_TOKEN") vaultConfig := api.DefaultConfig() vaultConfig.Address = vaultAddr vaultClient, err := api.NewClient(vaultConfig) if err != nil { log.Fatal(err) } vaultClient.SetToken(vaultToken) rotator := &Rotator{ vaultClient: vaultClient, fetcherURL: fetcherURL, fetcherAuth: fetcherBreakGlassToken, } http.HandleFunc("/rotate", rotator.handleRotate) log.Fatal(http.ListenAndServe(":8081", nil)) } type Rotator struct { vaultClient *api.Client fetcherURL string fetcherAuth string } func (r *Rotator) handleRotate(w http.ResponseWriter, r *http.Request) { // Check authentication for the rotator endpoint? Only authorized operators should call this. // Let's assume we have a separate API key for the rotator endpoint. apiKey := r.Header.Get("X-API-Key") if apiKey != operatorAPIKey { http.Error(w, "Unauthorized", http.StatusUnauthorized) return } err := r.Rotate() if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } w.WriteHeader(http.StatusOK) } func (r *Rotator) Rotate() error { newSecret := generateSecret() err := r.writeToVault(newSecret) if err != nil { return err } // Invalidate fetcher cache return r.invalidateFetcherCache() } func (r *Rotator) writeToVault(secret string) error { // Write to vault at secretPath // The structure depends on how you store the secret. // We assume a simple key "value" under "data". data := map[string]interface{}{ "value": secret, } _, err := r.vaultClient.Logical().Write(r.secretPath, data) return err } func (r *Rotator) invalidateFetcherCache() error { url := r.fetcherURL + "/break-glass/invalidate" req, _ := http.NewRequest("POST", url, nil) req.Header.Add("X-Break-Glass-Token", r.fetcherAuth) client := &http.Client{Timeout: 10 * time.Second} resp, err := client.Do(req) if err != nil { return err } defer resp.Body.Close() if resp.StatusCode != http.StatusOK { return fmt.Errorf("invalidate returned %s", resp.Status) } return nil } func generateSecret() string { // Generate a random string of desired length. // For example, 32 bytes base64. b := make([]byte, 32) rand.Read(b) return base64.StdEncoding.EncodeToString(b) }This is a basic implementation.
We must also consider:
- The fetcher's Redis might be down, then it will always fetch from vault (and not cache). That's acceptable but might overload vault. - The rotator's call to invalidate might fail if fetcher is down, then we have a stale cache until TTL. We can have the rotator retry a few times and then alert.Also, we must secure the Redis (use password, TLS, etc.) and Vault (use TLS, proper token management).
This design is a starting point.
Let me know if you have any questions or need adjustments. I'll design a comprehensive secure vault-backed secret management system with rotation and break-glass capabilities. This solution uses HashiCorp Vault as the secrets backend, but the pattern applies to other secret managers.
Architecture Overview
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Application │────│ Secret Client │────│ Vault CLI │ │ (Consumer) │ │ (SDK/Library) │ │ (Break-Glass) │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ ▼ ┌─────────────────┐ │ Cache Layer │ │ (Redis/In-mem) │ └─────────────────┘ │ ▼ ┌─────────────────┐ │ Vault │ │ (Secrets DB) │ └─────────────────┘1. Core Components
A. Secret Client Library (Application Integration)
# secret_client.py import os import json import requests import redis from datetime import datetime, timedelta from typing import Optional, Dict, Any from functools import lru_cache import logging import hashlib import hmac logger = logging.getLogger(__name__) class SecretClient: def __init__( self, vault_addr: str, vault_token: str, secret_path: str, cache_ttl: int = 300, # 5 minutes redis_url: Optional[str] = None, break_glass_role: str = "emergency-access" ): """ Initialize secret client with Vault integration """ self.vault_addr = vault_addr.rstrip('/') self.vault_token = vault_token self.secret_path = secret_path self.cache_ttl = cache_ttl self.break_glass_role = break_glass_role # Initialize cache if redis_url: self.cache = redis.from_url(redis_url) self.cache_enabled = True else: self.cache = {} self.cache_enabled = False # Session for Vault API calls self.session = requests.Session() self.session.headers.update({ 'X-Vault-Token': self.vault_token, 'Content-Type': 'application/json' }) # Metrics self.cache_hits = 0 self.cache_misses = 0 def _get_cache_key(self, version: Optional[int] = None) -> str: """Generate cache key with optional version""" base_key = f"secret:{self.secret_path}" if version is not None: return f"{base_key}:v{version}" return base_key def _fetch_from_vault(self, version: Optional[int] = None) -> Dict[str, Any]: """Fetch secret directly from Vault with optional version""" url = f"{self.vault_addr}/v1/{self.secret_path}" params = {} if version is not None: params['version'] = version try: response = self.session.get(url, params=params) response.raise_for_status() data = response.json() # Extract secret data (adjust based on your Vault secret structure) if 'data' in data: return data['data'] return data except requests.RequestException as e: logger.error(f"Failed to fetch secret from Vault: {e}") raise @lru_cache(maxsize=1) def _get_current_version(self) -> int: """Get current secret version from Vault metadata""" url = f"{self.vault_addr}/v1/{self.secret_path}/metadata" try: response = self.session.get(url) response.raise_for_status() return response.json().get('current_version', 1) except: return 1 def get_secret( self, key: Optional[str] = None, force_refresh: bool = False, break_glass: bool = False ) -> Any: """ Get secret with caching and break-glass support Args: key: Specific key within secret (for structured secrets) force_refresh: Bypass cache and fetch fresh from Vault break_glass: Use emergency access (bypass normal auth checks) Returns: Secret value or structured secret dict """ cache_key = self._get_cache_key() # Check cache first (unless force_refresh or break_glass) if not force_refresh and not break_glass and self._is_cache_valid(cache_key): self.cache_hits += 1 cached = self._get_from_cache(cache_key) if key: return cached.get(key) if isinstance(cached, dict) else cached return cached self.cache_misses += 1 # Fetch from Vault if break_glass: # Use emergency access role secret = self._break_glass_fetch() else: secret = self._fetch_from_vault() # Update cache self._set_cache(cache_key, secret) self._get_current_version.cache_clear() # Clear version cache if key: return secret.get(key) if isinstance(secret, dict) else secret return secret def _is_cache_valid(self, cache_key: str) -> bool: """Check if cache entry is still valid""" if not self.cache_enabled: return False ttl = self.cache.ttl(cache_key) if hasattr(self.cache, 'ttl') else None if ttl is not None and ttl > 0: return True return False def _get_from_cache(self, cache_key: str) -> Any: """Retrieve from cache (Redis or memory)""" if self.cache_enabled: cached = self.cache.get(cache_key) return json.loads(cached) if cached else None return self.cache.get(cache_key) def _set_cache(self, cache_key: str, value: Any): """Store in cache""" serialized = json.dumps(value) if isinstance(value, (dict, list)) else str(value) if self.cache_enabled: self.cache.setex(cache_key, self.cache_ttl, serialized) else: self.cache[cache_key] = value def _break_glass_fetch(self) -> Dict[str, Any]: """ Emergency access: fetch using break-glass workflow - Requires special Vault policy/role - Logs all access - May require multi-person approval (workflow) """ logger.warning("BREAK-GLASS ACCESS INITIATED - Emergency secret access") # In production, this would: # 1. Use a different Vault token with break-glass policy # 2. Trigger audit logging # 3. Possibly require MFA re-authentication # 4. Send alerts to security team # Example: Use emergency token stored in secure location emergency_token = os.getenv('VAULT_EMERGENCY_TOKEN') if emergency_token: original_token = self.vault_token self.vault_token = emergency_token self.session.headers.update({ 'X-Vault-Token': emergency_token }) try: secret = self._fetch_from_vault() # Audit log self._audit_log("break_glass_access", { "secret_path": self.secret_path, "timestamp": datetime.utcnow().isoformat(), "requester": self._get_caller_identity() }) return secret finally: # Restore original token self.vault_token = original_token self.session.headers.update({ 'X-Vault-Token': original_token }) raise PermissionError("Break-glass access not configured") def _audit_log(self, action: str, metadata: Dict[str, Any]): """Send audit log to security system""" audit_msg = { "action": action, "secret_path": self.secret_path, "timestamp": datetime.utcnow().isoformat(), **metadata } logger.info(f"AUDIT: {json.dumps(audit_msg)}") # Send to SIEM/audit system here def _get_caller_identity(self) -> str: """Get identity of caller (from service account, JWT, etc.)""" # Implementation depends on your auth system return os.getenv('SERVICE_ACCOUNT', 'unknown') def rotate_secret( self, new_secret: Optional[Any] = None, notify_callback: Optional[callable] = None ) -> Dict[str, Any]: """ Rotate the secret in Vault Args: new_secret: Optional new secret value (if None, generate) notify_callback: Callback to notify applications of rotation Returns: Rotation metadata """ # Generate new secret if not provided if new_secret is None: new_secret = self._generate_secret() # Write to Vault (this creates new version) url = f"{self.vault_addr}/v1/{self.secret_path}" payload = {'data': new_secret} if isinstance(new_secret, dict) else {'data': {'value': new_secret}} response = self.session.post(url, json=payload) response.raise_for_status() # Get new version metadata = response.json() new_version = metadata.get('data', {}).get('version', self._get_current_version() + 1) # Clear cache so next fetch gets new version self._clear_cache() # Notify if callback provided if notify_callback: try: notify_callback({ 'secret_path': self.secret_path, 'new_version': new_version, 'rotated_at': datetime.utcnow().isoformat() }) except Exception as e: logger.error(f"Rotation notification failed: {e}") # Audit log self._audit_log("secret_rotation", { "new_version": new_version, "rotated_by": self._get_caller_identity() }) return { 'status': 'rotated', 'new_version': new_version, 'secret_path': self.secret_path } def _clear_cache(self): """Clear all cached versions of this secret""" if self.cache_enabled: pattern = f"secret:{self.secret_path}*" keys = self.cache.keys(pattern) if keys: self.cache.delete(*keys) else: # Clear memory cache keys_to_remove = [k for k in self.cache.keys() if k.startswith(f"secret:{self.secret_path}")] for k in keys_to_remove: self.cache.pop(k, None) # Clear version cache self._get_current_version.cache_clear() def _generate_secret(self, length: int = 32) -> str: """Generate a cryptographically secure random secret""" import secrets import string alphabet = string.ascii_letters + string.digits return ''.join(secrets.choice(alphabet) for _ in range(length)) def health_check(self) -> Dict[str, Any]: """Check health of secret system""" status = { 'vault_connected': False, 'cache_connected': False, 'secret_accessible': False, 'current_version': None } # Check Vault try: self.session.get(f"{self.vault_addr}/v1/sys/health", timeout=2) status['vault_connected'] = True except: pass # Check cache if self.cache_enabled: try: self.cache.ping() status['cache_connected'] = True except: pass # Check secret access try: secret = self.get_secret(force_refresh=True) status['secret_accessible'] = secret is not None status['current_version'] = self._get_current_version() except: pass return statusB. Rotation Service (Scheduled Rotation)
# rotation_service.py import schedule import time from datetime import datetime from typing import List import threading class RotationScheduler: def __init__( self, secret_client: SecretClient, rotation_schedule: str = "daily at 02:00", pre_rotation_warning_hours: int = 24, post_rotation_validation: bool = True ): """ scheduled secret rotation manager Args: rotation_schedule: Cron-style schedule (e.g., "daily at 02:00", "weekly on monday") pre_rotation_warning_hours: Hours before rotation to send warnings post_rotation_validation: Validate secret after rotation """ self.secret_client = secret_client self.rotation_schedule = rotation_schedule self.pre_rotation_warning_hours = pre_rotation_warning_hours self.post_rotation_validation = post_rotation_validation self._running = False self._thread = None # Track rotation history self.rotation_history = [] def start(self): """Start the rotation scheduler""" self._running = True self._thread = threading.Thread(target=self._run_scheduler, daemon=True) self._thread.start() logger.info("Rotation scheduler started") def stop(self): """Stop the rotation scheduler""" self._running = False if self._thread: self._thread.join() logger.info("Rotation scheduler stopped") def _run_scheduler(self): """Main scheduler loop""" while self._running: schedule.run_pending() time.sleep(60) # Check every minute def _schedule_rotation(self): """Setup rotation schedule""" # Clear existing jobs schedule.clear('rotation') # Parse schedule (simplified - use croniter for production) if "daily" in self.rotation_schedule: schedule.every().day.at("02:00").do(self._rotate_all_secrets).tag('rotation') elif "weekly" in self.rotation_schedule: schedule.every().monday.at("02:00").do(self._rotate_all_secrets).tag('rotation') # Add more schedule parsers as needed else: # Default: rotate every 30 days schedule.every(30).days.do(self._rotate_all_secrets).tag('rotation') def _rotate_all_secrets(self): """Rotate all managed secrets""" logger.info("Starting scheduled rotation") # Get list of secrets to rotate (from config/DB) secrets_to_rotate = self._get_secrets_to_rotate() for secret_config in secrets_to_rotate: try: self._rotate_secret(secret_config) except Exception as e: logger.error(f"Failed to rotate {secret_config['path']}: {e}") # Send alert self._send_alert(f"Rotation failed: {secret_config['path']}", str(e)) def _rotate_secret(self, secret_config: Dict[str, Any]): """Rotate a single secret""" path = secret_config['path'] client = secret_config['client'] # Pre-rotation warning self._send_warning(path, hours_before=self.pre_rotation_warning_hours) # Perform rotation result = client.rotate_secret() # Post-rotation validation if self.post_rotation_validation: self._validate_rotation(path, result['new_version']) # Record history self.rotation_history.append({ 'path': path, 'version': result['new_version'], 'timestamp': datetime.utcnow().isoformat(), 'scheduled': True }) # Post-rotation notification self._send_notification(path, result['new_version']) def _validate_rotation(self, path: str, version: int): """Validate that rotated secret is accessible""" client = self._get_client_for_path(path) try: secret = client.get_secret(force_refresh=True) if secret: logger.info(f"Validation successful for {path} v{version}") else: raise ValueError("Secret is empty after rotation") except Exception as e: logger.error(f"Validation failed for {path}: {e}") # Consider rolling back? raise def _send_warning(self, path: str, hours_before: int): """Send pre-rotation warning""" # Send to monitoring/notification system logger.info(f"Rotation warning: {path} will rotate in {hours_before} hours") def _send_notification(self, path: str, version: int): """Send post-rotation notification to consumers""" # Notify applications via message queue, webhook, etc. logger.info(f"Secret rotated: {path} now at version {version}") def _send_alert(self, title: str, message: str): """Send alert to security team""" # Integrate with PagerDuty, Slack, email, etc. logger.error(f"ALERT: {title} - {message}") def _get_secrets_to_rotate(self) -> List[Dict[str, Any]]: """Get list of secrets configured for rotation""" # This would come from configuration database or config file return [ { 'path': 'secret/data/application/db_password', 'client': SecretClient(...), # Initialize with proper config 'rotation_frequency': '30d' }, # Add more secrets ] def force_rotate(self, path: str) -> Dict[str, Any]: """Manually force rotation of a specific secret""" secret_config = next( (s for s in self._get_secrets_to_rotate() if s['path'] == path), None ) if not secret_config: raise ValueError(f"Secret path {path} not found in rotation schedule") return self._rotate_secret(secret_config)C. Break-Glass Emergency Access
#!/bin/bash # break-glass-emergency.sh # Emergency break-glass script for secret access set -e VAULT_ADDR="${VAULT_ADDR:-https://vault.example.com}" EMERGENCY_TOKEN="${VAULT_EMERGENCY_TOKEN?Need emergency token}" SECRET_PATH="${1?Usage: $0 <secret-path>}" AUDIT_LOG="/var/log/vault-break-glass.log" # Log the emergency access echo "$(date -Iseconds) - BREAK-GLASS ACCESS - Path: $SECRET_PATH - User: $USER - PID: $$" >> "$AUDIT_LOG" # Send alert to security team curl -X POST "https://alerts.example.com/v1/emergency" \ -H "Content-Type: application/json" \ -d "{\"service\":\"vault-emergency\",\"severity\":\"critical\",\"message\":\"Break-glass access for $SECRET_PATH by $USER\"}" # Fetch secret using emergency credentials response=$(curl -s \ --header "X-Vault-Token: $EMERGENCY_TOKEN" \ "$VAULT_ADDR/v1/$SECRET_PATH" \ | jq -r '.data.data.value // .data') # Log successful access echo "$(date -Iseconds) - ACCESS GRANTED - Value: ${response:0:20}..." >> "$AUDIT_LOG" # Output secret (with warning) echo "⚠️ EMERGENCY SECRET ACCESS - This action is logged and monitored" echo "Secret path: $SECRET_PATH" echo "Value: $response" echo "" echo "⚠️ REMEMBER: Change this secret immediately after use!" echo "Alert sent to security team. Audit log: $AUDIT_LOG"D. Vault Policies & Configuration
# vault-policies.hcl # Normal application policy (read-only, limited TTL) path "secret/data/application/*" { capabilities = ["read"] allowed_parameters = { "ttl" = ["1h"] } } # Emergency break-glass policy (requires MFA, logs everything) path "secret/data/application/*" { capabilities = ["read", "update", "create", "delete"] allowed_parameters = {} # Require MFA for break-glass control { mfa_methods = ["break-glass-mfa"] } } # Rotation service policy (update only, no delete) path "secret/data/application/*" { capabilities = ["update"] allowed_parameters = { "data" = ["*"] } }2. Deployment Configuration
# deployment.yaml (Kubernetes example) apiVersion: apps/v1 kind: Deployment metadata: name: secret-client spec: replicas: 3 template: spec: containers: - name: app image: myapp:latest env: - name: VAULT_ADDR valueFrom: secretKeyRef: name: vault-config key: address - name: VAULT_TOKEN valueFrom: secretKeyRef: name: vault-token key: token - name: SECRET_PATH value: "secret/data/application/db_password" - name: REDIS_URL value: "redis://redis-master:6379/0" # Security context securityContext: runAsNonRoot: true runAsUser: 1000 readOnlyRootFilesystem: true volumeMounts: - name: vault-break-glass-token mountPath: "/etc/vault/emergency" readOnly: true volumes: - name: vault-break-glass-token secret: secretName: vault-emergency-token3. Usage Examples
Regular Secret Fetching
# In your application client = SecretClient( vault_addr=os.getenv('VAULT_ADDR'), vault_token=os.getenv('VAULT_TOKEN'), secret_path='secret/data/application/db_password', redis_url=os.getenv('REDIS_URL') ) # Get secret (cached) db_password = client.get_secret(key='password') # Force fresh fetch db_password = client.get_secret(force_refresh=True)Scheduled Rotation
# Initialize rotation scheduler scheduler = RotationScheduler( secret_client=client, rotation_schedule="weekly on monday", pre_rotation_warning_hours=48 ) scheduler.start() # Or manually rotate rotation_result = client.rotate_secret( notify_callback=lambda meta: notify_services(meta) )Break-Glass Emergency
# Using emergency script ./break-glass-emergency.sh "secret/data/application/db_password"Rotate via CLI
# Emergency rotation client = SecretClient( vault_addr='https://vault.example.com', vault_token=os.getenv('VAULT_EMERGENCY_TOKEN'), secret_path='secret/data/application/db_password' ) # Force rotation with new secret result = client.rotate_secret(new_secret="new-emergency-password-123") print(f"Rotated to version {result['new_version']}")4. Security Considerations
A. Access Control
- Least Privilege: Separate policies for normal access vs break-glass
- MFA Requirement: Break-glass requires multi-factor authentication
- Time-limited Tokens: Use Vault's token TTL and renewable tokens
- IP Restrictions: Limit Vault access to known service IPs
B. Monitoring & Alerting
# metrics.py import prometheus_client CACHE_HITS = prometheus_client.Counter('secret_cache_hits', 'Cache hits') CACHE_MISSES = prometheus_client.Counter('secret_cache_misses', 'Cache misses') BREAK_GLASS_ACCESS = prometheus_client.Counter('break_glass_access', 'Break-glass accesses', ['secret_path']) ROTATION_COUNT = prometheus_client.Counter('secret_rotation_total', 'Total rotations', ['secret_path'])C. Audit Trail
Every access and rotation should be logged with:
- Who/when/what
- Source IP/user agent
- Secret path accessed
- Whether it was break-glass
- Cache hit/miss status
- Rotation details (old/new version hashes)
D. Disaster Recovery
- Offline Backup: Store encrypted backups of critical secrets
- Split Knowledge: Require multiple people to reconstruct master keys
- Geographic Distribution: Vault cluster across multiple regions
- Recovery Procedures: Documented break-glass recovery process
5. Best Practices Implementation
# secure_config.py from cryptography.fernet import Fernet import base64 import os class SecureConfig: """Additional layer for extremely sensitive configs""" def __init__(self, master_key_env: str = "MASTER_KEY"): self.master_key = os.getenv(master_key_env) if not self.master_key: raise ValueError("Master key not set") self.cipher = Fernet(self.master_key) def encrypt_config(self, config: Dict) -> bytes: """Encrypt config before storing in Vault""" json_str = json.dumps(config, sort_keys=True) return self.cipher.encrypt(json_str.encode()) def decrypt_config(self, encrypted_data: bytes) -> Dict: """Decrypt config after fetching from Vault""" decrypted = self.cipher.decrypt(encrypted_data) return json.loads(decrypted.decode())6. Testing Strategy
# test_secret_client.py import pytest from unittest.mock import Mock, patch class TestSecretClient: def test_cache_hit(self): client = SecretClient(vault_addr='...', vault_token='...', secret_path='test') with patch.object(client, '_fetch_from_vault') as mock_fetch: mock_fetch.return_value = {'value': 'secret1'} # First call - cache miss client.get_secret() assert mock_fetch.call_count == 1 # Second call - cache hit client.get_secret() assert mock_fetch.call_count == 1 # No additional call def test_break_glass_audit(self): client = SecretClient(vault_addr='...', vault_token='...', secret_path='test', break_glass_role='emergency') with patch.object(client, '_break_glass_fetch') as mock_break: with patch.object(client, '_audit_log') as mock_audit: mock_break.return_value = {'value': 'emergency_secret'} client.get_secret(break_glass=True) mock_break.assert_called_once() mock_audit.assert_called_with('break_glass_access', pytest.approx ANY)Key Features Summary:
✅ Secure Caching: Redis-backed with TTL to reduce Vault load
✅ Automatic Rotation: Scheduled rotations with pre/post validation
✅ Break-Glass Access: Emergency access with full audit trail
✅ Version Management: Track and access secret versions
✅ Multi-layer Security: Vault policies + application-level controls
✅ Comprehensive Audit: Every access logged with context
✅ High Availability: Stateless client, distributed cache
✅ Validation: Post-rotation validation to ensure secrets work
✅ Alerting: Pre-rotation warnings, failure alerts
✅ Disaster Recovery: Emergency procedures documentedImplementation Notes:
- Never store master keys in code - Use environment variables or secret management for init tokens
- Rotate Vault root keys regularly - Use Vault's key rotation features
- Implement circuit breakers - Don't crash app if Vault is down (use last known good)
- Use dynamic secrets when possible - Databases, APIs with short TTL
- Regular security audits - Review break-glass usage, rotation logs
- Test disaster scenarios - Regular fire drills for break-glass procedures
- Encrypt at rest - Ensure Redis/Vault storage encryption enabled
- Network segmentation - Secret client should only talk to Vault/Redis on internal network
This provides enterprise-grade secret management with proper break-glass procedures that maintain security while allowing emergency access when absolutely necessary.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.