Secret Management #

In modern software architecture, our applications need various sensitive information to operate and interact with other services. This sensitive information includes database passwords, third-party API keys (like Stripe or SendGrid), TLS certificate private keys, symmetric encryption keys, and OAuth client secrets. This information is technically called Secrets. How we store, distribute, and rotate secrets is one of the most frequently ignored and incorrectly handled security aspects by developers. Secret leaks in public code repositories are the leading cause of thousands of large-scale confidential data leak incidents every year. This article dissects secret storage anti-patterns, how managed secrets services work, runtime environment integration, and zero-downtime rotation strategies.

Common Secret Storage Anti-Patterns #

Many beginner developers make fatal mistakes in handling secrets for practicality reasons during local debugging.

1. Hardcoding Credentials Directly in Source Code #

This is the most dangerous anti-pattern. Writing password strings directly in variable declarations inside our application code.

  • Danger: Anyone with read-access to the code repository can see the password. Worse, the secret string enters the Git commit history. This history is permanent; deleting the code line in a new commit doesn’t remove the secret from old commits. Hackers can easily browse commit history to find the secret again.

2. Committing Environment Configuration Files (.env) #

Storing secrets in a local .env file is good practice, but committing that .env file to the Git repository is a heavy anti-pattern.

  • Danger: Often the .env file is accidentally committed because developers forgot to add it to the ignored files list (.gitignore).

3. Embedding Secrets into Container Images at Build Time #

Putting API keys into ENV instructions in a Dockerfile so containers can access them at runtime.

  • Danger: Container image layers are public inside the internal registry. Anyone able to docker pull the image can inspect layers using utilities like docker history to read the secret values in plain text.
# ANTI-PATTERN: Putting plain-text passwords in a Kubernetes deployment manifest
apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-service
spec:
  template:
    spec:
      containers:
      - name: billing-app
        image: billing:v1.0.0
        env:
        - name: DB_PASSWORD
          value: "SuperSecretProdDBPassword123" # DON'T: Plain text & exposed in the Git repo!

# --- Separator ---

# CORRECT: Using a secure reference to an external Secrets driver (CSI Driver)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-service
spec:
  template:
    spec:
      containers:
      - name: billing-app
        image: billing:v1.0.0
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aws-secrets-provider" # ✓ CORRECT: Dynamic integration

The Impact of Credential Leaks in Git Repositories #

If we make the mistake of committing cloud API credentials (e.g., AWS Access Keys) to a public GitHub repository, financial and security consequences happen within minutes.

Anatomy of a Git-Scanner Bot Attack: #

  1. Constant Scanning: Hackers run thousands of automated bots monitoring GitHub’s public event stream in real-time.
  2. Instant Detection: The moment we perform a git push containing an access key, scanner bots detect the key’s distinctive text pattern (like AKIA... on AWS) in under 10 seconds.
  3. Mass Exploitation: Bots automatically try using that access key to trigger large-scale GPU VM creation APIs in remote cloud regions for cryptocurrency mining.
  4. Financial Disaster: Within just 1 hour, our cloud instance rental bill can balloon to tens of thousands of dollars before we realize the mistake.

How to Clean Up Already-Leaked Git History #

If a leak occurs, simply deleting the file and making a new commit won’t help. We must clean the entire Git tree history using special tools:

# CORRECT: Using git-filter-repo to remove .env files from all Git commit history
# (Install via Homebrew: brew install git-filter-repo)
$ git filter-repo --path .env --invert-paths

# Critical Warnings: 
# 1. Deleting Git history damages commit history (force push required).
# 2. Consider any exposed secrets as compromised. 
#    Rotate/change the passwords at the target database system immediately!

How Managed Secrets Services Work #

To solve all the anti-patterns above, cloud providers offer dedicated managed services for securely storing secrets, like AWS Secrets Manager, Google Cloud Secret Manager, or HashiCorp Vault.

flowchart TD
    subgraph AppServer ["Application Server (Runtime)"]
        App["Web Application"]
    end
    
    subgraph SecretManager ["Managed Secrets Service"]
        KMS["Key Management Service (KMS)<br>(AES-256 Encryption)"]
        Storage["Encrypted Secrets DB"]
        AccessEngine["IAM Authorization Engine"]
    end
    
    App -->|"1. API Request with IAM Role token<br>GetSecretValue('prod/db/password')"| AccessEngine
    AccessEngine -->|"2. Validate Role Permissions"| Storage
    Storage -->|"3. Decrypt data using KMS key"| KMS
    KMS -->|"4. Return plain-text secret string"| App

Main Features of Secrets Management Services: #

  1. Encryption-at-Rest: Secrets aren’t stored in plain text on the provider’s storage servers. Secret data is physically encrypted using the AES-256 algorithm with encryption keys managed by the external Key Management Service (KMS).
  2. Centralized Access Control (IAM Integration): Access to read secrets is tightly locked using IAM policies. We can define very specific rules, e.g.: “Only VMs with the ‘AppServer’ IAM Role may read the ‘prod/db/password’ secret”.
  3. Audit Trail: Every time an entity tries to read or modify a secret value, the activity is permanently recorded in audit logs. This simplifies security compliance auditing.
  4. Dynamic Secrets: The most advanced feature where the secrets management system can dynamically create new database accounts that only stay active for a few hours to serve one application process, then automatically delete that database account after the application finishes working.

Runtime Implementation Strategies: Pull vs Inject #

There are two main patterns commonly used to distribute secret data from the Secrets Manager to our applications at runtime:

1. Pull Method (SDK Runtime Query) #

Our application actively calls the Secrets Manager API using the official SDK library at application startup to fetch secret values directly into RAM memory.

  • Advantages: Secrets are never physically written to the VM’s local hard drive and never stored in static config files, minimizing theft risk.
# Example secure secret retrieval implementation using the AWS Python SDK
import boto3
import json
import logging
from botocore.exceptions import ClientError

def get_database_credentials():
    secret_name = "prod/app/database"
    region_name = "ap-southeast-3" # Jakarta Region

    # Create a Secrets Manager client using the VM instance's IAM Role authentication
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )

    try:
        get_secret_value_response = client.get_secret_value(
            SecretId=secret_name
        )
    except ClientError as e:
        logging.error(f"Failed to fetch secret: {e}")
        raise e

    # Decrypt the JSON secret string
    secret = get_secret_value_response['SecretString']
    return json.loads(secret)

# Call during application database connection initialization
db_creds = get_database_credentials()
# db_creds['password'] is ready for use in the database connection pool

2. Inject Method (Sidecar / CSI Driver) #

This method is commonly used in Kubernetes. An external operator agent monitors the cloud Secrets Manager, downloads the values, and injects them into an in-memory temporary volume (in-memory tmpfs volume) inside our application Pod container as read-only files.

  • Advantages: Our application code doesn’t need to import cloud provider SDKs, keeping the application code clean and agnostic (portable) to any cloud vendor.

Zero-Downtime Secret Rotation Design #

One bad habit of technology organizations is never changing their database passwords for years because they fear the replacement process will trigger application downtime.

Yet, letting secrets stay active forever dramatically increases security risk. The best solution is implementing Zero-Downtime Secret Rotation using the multi-credential method:

sequenceDiagram
    autonumber
    participant App as Application Client
    participant SM as Secrets Manager
    participant DB as Database Server
    
    Note over SM: Step 1: Scheduled Rotation Trigger
    SM->>DB: Create new credential user (User_B) with same permissions
    SM->>SM: Store User_B as "PENDING" version
    
    Note over SM: Step 2: Connection Testing
    SM->>DB: Test connection using User_B credentials
    DB-->>SM: Success
    
    Note over SM: Step 3: Credential Promotion
    SM->>SM: Change User_B status to "CURRENT" (Active)<br>Change User_A to "PREVIOUS" (Backup)
    
    Note over App: Step 4: App Session Refresh
    App->>SM: Fetch CURRENT credentials (Gets User_B)
    App->>DB: Connect new database connection using User_B
    DB-->>App: Success
    
    Note over SM: Step 5: Cleanup
    SM->>DB: Delete old credentials (User_A) from the database

Rotation Flow Explanation: #

  1. Step 1 (Create Secret): Secrets Manager triggers the automatic rotation function. The system contacts the database and creates a new backup database user set (e.g., user_version_2) with a new password without deleting the currently active user (user_version_1).
  2. Step 2 (Test Secret): The system tests the new database connection to ensure there are no typos or firewall configuration errors.
  3. Step 3 (Promote Secret): Secrets Manager marks the new credentials as the CURRENT (active) version. The old credentials shift status to PREVIOUS (backup).
  4. Step 4 (App Refresh): The application periodically (e.g., every 1 hour) contacts Secrets Manager to check for version updates. The application detects the new version, gradually closes old database connections, and opens new database connections using user_version_2.
  5. Step 5 (Revoke Secret): After the application is confirmed to have fully switched to the new credentials, Secrets Manager contacts the database again to permanently delete the old credentials (user_version_1).

Leak Prevention: Pre-commit Hooks and CI/CD Scanners #

The best defense against secret leaks is preventing secrets from being committed to repositories from the start, on the developer’s local computer.

Using Git Pre-commit Hooks #

We can use open-source utilities like Gitleaks or TruffleHog to automatically scan every code line modification before the Git commit process is allowed to run.

# Example gitleaks configuration to block commits when secrets are detected
# Step 1: Install gitleaks locally
$ brew install gitleaks

# Step 2: Run manual detection on the local directory
$ gitleaks detect --source=. --verbose

# Step 3: Integrate gitleaks into the git pre-commit hook
# (Gitleaks automatically analyzes every new code line when we type 'git commit')
$ gitleaks protect --verbose

If the Gitleaks bot detects strings resembling access key patterns, passwords, or SSL certificates, the git commit process is immediately aborted with an error message, forcing developers to move those strings out of the source code before trying to commit again.

Summary #

  • Credentials committed to Git must be considered leaked — We must rotate those credentials immediately and clean the Git history using git-filter-repo.
  • Use Managed Secrets Services to store all secret data with AES-256 at-rest encryption guarantees integrated with IAM authorization.
  • Apply the runtime pull pattern (Pull Method) — Applications pull secrets directly into RAM memory at startup, not reading them from static config files.
  • Implement zero-downtime rotation using alternating multi-user strategies to avoid operational disruptions.
  • Install Git Pre-commit Hooks (Gitleaks) on developer local computers to instantly block any accidental secret commit attempts.

← Previous: Authentication vs Authorization   Next: Zero Trust →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact