Secrets Management

From the CTO — Required reading for all product and engineering teams. A hardcoded credential is a vulnerability waiting to be discovered. No secret belongs in source code. Ever.

---

Table of Contents

1. Why This Matters
2. The Standard: No Secrets in Code
3. Approved Vault Managers
4. Accessing Secrets at Runtime
5. Environment Variables: Rules and Limits
6. Rotation and Expiry
7. Auditing Your Codebase
8. Quick Reference

---

Why This Matters

Secrets — API keys, database passwords, tokens, certificates — control access to our systems and our users' data. When a secret leaks, an attacker can use it immediately, and we may not know for months.

Hardcoded credentials get exposed in git

Every commit is permanent. Even if you delete the file later, git history keeps the secret forever. Public repositories are scanned by bots within seconds of a push. Private repositories are not safe either: a stolen laptop, a misconfigured permission, or a disgruntled former employee can expose the full history.

Forks and clones carry the history. If a secret was ever committed, it exists in every fork, every clone, and every CI artifact that checked out that commit. Rotation is the only fix — and rotation is expensive.

Secrets in config files get checked in by accident

.env files are committed more often than people think. A developer adds a .env file locally, forgets to add it to .gitignore, and the next git add . sends the secret to the remote. This is one of the most common causes of credential leaks in startups.

Config files are copied. A developer copies a working config to a new environment, a new service, or a Slack message to get help from a teammate. Each copy is a new exposure point.

Log lines capture what your code prints

Secrets appear in logs when you debug. A developer adds a console.log(config) to debug a startup issue. The config object includes the database password. The log goes to Datadog or CloudWatch, is searchable by anyone on the team, and is retained for 90 days. The secret is now in a third-party system with a different access control model.

Architectural warning: There is no safe place to hardcode a secret. Not in a comment, not in a test, not in a seed script, not in a Docker build argument that gets cached in an image layer. The only safe place for a secret is a vault manager with access controls and an audit log.

---

The Standard: No Secrets in Code

A secret is any value that grants access to a system or protects sensitive data. If someone else having this value would be a problem, it is a secret.

Category	Examples
Database credentials	DATABASE_URL, usernames, passwords
API keys	Stripe, Twilio, SendGrid, OpenAI, AWS keys
Auth tokens	JWT signing secrets, OAuth client secrets
Certificates and private keys	TLS private keys, SSH keys
Encryption keys	AES keys, HMAC secrets
Internal service credentials	Service account tokens, webhook signing secrets

Rules

Rule	Detail
No secrets in source code	Not in .ts, .py, .go, .yaml, .json, .tf, or any other file
No secrets in git history	If it was ever committed, rotate it immediately
No secrets in Docker images	Not in ENV, ARG, COPY, or RUN instructions
No secrets in CI logs	Mask all secrets in CI; never echo them
No secrets in error messages	Never include credential values in exceptions or API responses
No secrets in client-side code	Never send a backend secret to the browser

Before vs. after

Pattern	❌ Wrong	✅ Correct
Database URL	postgres://admin:hunter2@prod-db:5432/app hardcoded in db.ts	Fetched from AWS Secrets Manager at startup
API key	const STRIPE_KEY = "sk_live_abc123..." in payments.ts	process.env.STRIPE_SECRET_KEY set by the platform at deploy time
JWT secret	secret: "my-super-secret" in auth.config.ts	Retrieved from Vault at boot; injected as an environment variable
SSH private key	Committed to deploy/keys/id_rsa	Stored in Secrets Manager; retrieved by CI pipeline at runtime

---

Approved Vault Managers

All secrets must live in one of these systems. No other storage is approved for production secrets.

AWS Secrets Manager

Use this when your infrastructure runs on AWS.

AWS Secrets Manager stores secrets as JSON blobs, supports automatic rotation for RDS databases, and integrates natively with IAM for access control. Every access is logged in CloudTrail.

# Store a secret
aws secretsmanager create-secret \
  --name "prod/payments/stripe-key" \
  --secret-string '{"secret_key":"sk_live_abc123..."}'

# Retrieve a secret
aws secretsmanager get-secret-value \
  --secret-id "prod/payments/stripe-key" \
  --query SecretString \
  --output text

Naming convention: {env}/{service}/{secret-name} Examples: prod/auth/jwt-secret, staging/db/postgres-url, prod/integrations/sendgrid-api-key

HashiCorp Vault

Use this when your infrastructure is multi-cloud or on-premise.

HashiCorp Vault is a purpose-built secrets manager with fine-grained access policies, dynamic secrets (credentials that Vault generates on demand and expires automatically), and detailed audit logs. It supports multiple auth methods: AppRole, Kubernetes, AWS IAM, and others.

# Write a secret (Vault CLI)
vault kv put secret/prod/payments stripe_key="sk_live_abc123..."

# Read a secret
vault kv get -field=stripe_key secret/prod/payments

What NOT to use for secrets:

Tool	Why it is not approved for secrets
Git repository	Permanent history; no access controls on individual values
.env files committed to git	Too easy to commit by accident
AWS Parameter Store (Standard)	Acceptable for non-sensitive config only; use Secrets Manager for secrets
Kubernetes ConfigMap	Not encrypted at rest by default; not for secrets
Hardcoded values in Helm values files	Committed to git
Slack / Notion / Confluence	No audit log; no rotation; not a secret store

Kubernetes Secrets (kind: Secret) are acceptable only if your cluster has envelope encryption enabled and the secrets are populated from AWS Secrets Manager or Vault via an operator (e.g., External Secrets Operator). Never write secret values directly into Kubernetes manifest files committed to git.

---

Accessing Secrets at Runtime

Option 1: Inject as environment variables at deploy time (preferred for most services)

The platform (ECS, Lambda, Kubernetes) retrieves the secret from the vault and injects it as an environment variable before the process starts. Application code reads process.env.SECRET_NAME. The secret value never touches your codebase or your CI logs.

AWS ECS task definition example:

{
  "secrets": [
    {
      "name": "DATABASE_URL",
      "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:prod/db/postgres-url"
    },
    {
      "name": "STRIPE_SECRET_KEY",
      "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:prod/payments/stripe-key:secret_key::"
    }
  ]
}

Application code:

// ✅ Correct — value comes from the platform, not the codebase
const db = new Client({ connectionString: process.env.DATABASE_URL });
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!);

Option 2: Fetch from the SDK at startup (for secrets that rotate often)

Some secrets — particularly short-lived tokens or database credentials with automatic rotation — should be fetched directly from the SDK at application startup (or on each request if the TTL is very short).

import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";

const client = new SecretsManagerClient({ region: "us-east-1" });

async function getSecret(secretId: string): Promise<string> {
  const response = await client.send(
    new GetSecretValueCommand({ SecretId: secretId })
  );
  return response.SecretString!;
}

// At startup
const jwtSecret = await getSecret("prod/auth/jwt-secret");

Cache the result in memory for the duration of the process. Do not call Secrets Manager on every HTTP request — it adds latency and costs money.

Option 3: External Secrets Operator for Kubernetes

If you are running on Kubernetes, use the External Secrets Operator to sync secrets from AWS Secrets Manager or Vault into Kubernetes Secrets automatically. The operator handles rotation.

# ExternalSecret manifest — references the vault, not the value
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: stripe-key
spec:
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: stripe-key-k8s-secret
  data:
    - secretKey: STRIPE_SECRET_KEY
      remoteRef:
        key: prod/payments/stripe-key
        property: secret_key

The manifest references the secret path in the vault, not the value. It is safe to commit.

---

Environment Variables: Rules and Limits

Environment variables are the correct way to pass secrets to application code at runtime. They are not a storage mechanism.

Rule	Reason
Never commit .env files that contain real values	They end up in git
Always add .env, .env.local, .env.production to .gitignore	Prevents accidental commits
Use .env.example with placeholder values	Documents what variables are needed without real values
Never log process.env or the full config object	Secrets appear in log output
Never pass secrets as CLI arguments	They appear in process lists visible to other users on the system

.env.example (committed to git):

# Database
DATABASE_URL=postgres://user:password@host:5432/dbname

# Stripe
STRIPE_SECRET_KEY=sk_live_...

# Auth
JWT_SECRET=your-jwt-signing-secret

.env (never committed — local dev only with non-production values):

DATABASE_URL=postgres://dev:devpassword@localhost:5432/app_dev
STRIPE_SECRET_KEY=sk_test_...
JWT_SECRET=local-dev-secret-not-for-production

---

Rotation and Expiry

Every secret must have a rotation plan. Static, never-expiring credentials are a risk — if they leak, they are valid forever.

Secret type	Maximum lifetime	Rotation method
Database passwords	90 days	AWS Secrets Manager automatic rotation
API keys (third-party)	Per provider policy	Manual; use provider's key management UI
JWT signing secrets	1 year	Manual rotation with a dual-key overlap period
Service account tokens	90 days	Automated via Vault TTL or IAM policy
TLS certificates	Per CA (usually 90–398 days)	Automated via ACM or cert-manager

What to do when a secret is compromised

1. Rotate immediately. Do not investigate first — stop the bleeding. Generate a new secret, update it in the vault, and redeploy.
2. Revoke the old secret. Do not just replace it — explicitly invalidate the leaked value with the issuer (AWS, Stripe, your database, etc.).
3. Check the audit log. CloudTrail (for AWS) or Vault's audit log shows every time and every system that accessed the secret. Identify the blast radius.
4. Check for unauthorized use. Look for unexpected API calls, database connections, or transactions in the window between when the secret was exposed and when it was rotated.
5. File an incident report. Every credential leak is a security incident. Document what happened, when, and what was done.

---

Auditing Your Codebase

Run these commands to find secrets that should not be in your repository.

# Find common hardcoded credential patterns
grep -rn \
  -e "password\s*=\s*['\"][^'\"]\{6,\}" \
  -e "secret\s*=\s*['\"][^'\"]\{6,\}" \
  -e "api_key\s*=\s*['\"][^'\"]\{6,\}" \
  -e "sk_live_" \
  -e "AKIA[0-9A-Z]\{16\}" \
  --include="*.ts" --include="*.js" --include="*.py" --include="*.go" \
  src/

# Find .env files that may have been committed
git log --all --full-history -- "**/.env" "**/.env.production" "**/.env.local"

# Scan entire git history for secrets (use truffleHog or gitleaks)
# Install: brew install gitleaks
gitleaks detect --source . --verbose

# Find private keys in the repository
grep -rn "BEGIN.*PRIVATE KEY\|BEGIN RSA PRIVATE KEY\|BEGIN EC PRIVATE KEY" .

Add to CI: Run gitleaks detect or a similar tool (truffleHog, detect-secrets) on every pull request. A finding should block the merge.

If you find a secret in git history: Rotating the secret is mandatory, even if the repository is private. Removing it from history with git filter-repo reduces exposure but does not eliminate it — any clone taken before the rewrite still has the secret.

---

Quick Reference

✅  Store secrets in:   AWS Secrets Manager or HashiCorp Vault
✅  Access secrets via: environment variables injected at deploy time, or SDK at startup
✅  Name secrets as:    {env}/{service}/{secret-name}
✅  Rotate secrets:     every 90 days for credentials; immediately if compromised
✅  Commit to git:      .env.example with placeholder values only

❌  Never hardcode a secret in source code
❌  Never commit .env files with real values
❌  Never log process.env or full config objects
❌  Never pass secrets as CLI arguments
❌  Never store secrets in Kubernetes ConfigMaps or unencrypted Kubernetes Secrets
❌  Never share secrets over Slack, email, or any messaging tool

Task	How to do it
Store a new production secret	aws secretsmanager create-secret or vault kv put
Access a secret in application code	Read from process.env.* injected by the platform
Access a secret in a Kubernetes pod	Use External Secrets Operator to sync from the vault
Rotate a compromised secret	Rotate in the vault, redeploy, revoke the old value, file an incident
Document required env vars	Add to .env.example with placeholder values — commit this file
Scan for hardcoded secrets	Run gitleaks detect locally and in CI
Give a service access to a secret	Use IAM roles (AWS) or Vault policies — not shared credentials

---

Questions or edge cases not covered here? Open a discussion in the #engineering-standards channel.