🚀 K8s YSeries #2-Why File mounting beats environment variables

rjshk013 June 21, 2025

🔐 Kubernetes Secrets: Why File Mounting Beats Environment Variables

🚨 The Problem: Environment Variable Security Risks

Real-World Security Incident

# Common but DANGEROUS pattern
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  template:
    spec:
      containers:
      - name: payment-service
        image: payment-api:v1.2.0
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: payment-db-secret
              key: password
        - name: STRIPE_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: payment-secrets
              key: stripe-key
        - name: JWT_SECRET
          valueFrom:
            secretKeyRef:
              name: auth-secrets
              key: jwt-secret

What Actually Happens in Production:

1. Process Listing Exposure

# Any user with kubectl access can see this:
kubectl exec payment-api-7d4f8b9c-xyz -- ps aux

# Output reveals ALL secrets:
USER  PID COMMAND
root  1   payment-api --db-pass=super_secret_password_123 --stripe-key=sk_live_abc123... --jwt=eyJhbGc...

2. Container Logs Leak Secrets

# Application startup logs
kubectl logs payment-api-7d4f8b9c-xyz

# Typical output:
2024-01-15 10:30:21 INFO  Starting payment service...
2024-01-15 10:30:22 DEBUG Config loaded: DB_PASSWORD=super_secret_password_123
2024-01-15 10:30:22 DEBUG Stripe key configured: STRIPE_SECRET_KEY=sk_live_abc123def456...
2024-01-15 10:30:22 ERROR Failed to connect: connection string contains password

3. Crash Dump Exposure

# When application crashes, environment variables are included in core dumps
kubectl exec payment-api-7d4f8b9c-xyz -- cat /proc/1/environ
# Output: DB_PASSWORD=super_secret_password_123STRIPE_SECRET_KEY=sk_live_abc123...

4. Monitoring System Exposure

# Prometheus metrics accidentally expose secrets
# payment_service_config{db_password="super_secret_password_123"} 1
# stripe_api_calls_total{api_key="sk_live_abc123"} 42

✅ The Solution: File-Based Secret Mounting

Secure Implementation Pattern

apiVersion: v1
kind: Secret
metadata:
  name: payment-db-credentials
  namespace: payments
type: Opaque
data:
  username: cGF5bWVudHNfZGJfdXNlcg==  # payments_db_user (base64)
  password: c3VwZXJfc2VjcmV0X3Bhc3N3b3JkXzEyMw==  # super_secret_password_123 (base64)
  host: cGF5bWVudHMtZGIucmRzLmFtYXpvbmF3cy5jb20=  # payments-db.rds.amazonaws.com (base64)
—
apiVersion: v1
kind: Secret
metadata:
  name: payment-api-keys
  namespace: payments
type: Opaque
data:
  stripe-secret-key: c2tfbGl2ZV9hYmMxMjNkZWY0NTY=  # sk_live_abc123def456 (base64)
  jwt-secret: ZXlKaGJHY2lPaUpJVXpJMU5pSjk=  # eyJhbGciOiJIUzI1NiJ9 (base64)
—
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: payments
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payment-api
  template:
    metadata:
      labels:
        app: payment-api
    spec:
      serviceAccountName: payment-api-sa
      containers:
      – name: payment-service
        image: payment-api:v1.2.0
        ports:
        – containerPort: 8080
        # NO environment variables for secrets!
        env:
        – name: SERVER_PORT
          value: “8080”
        – name: LOG_LEVEL
          value: “INFO”
        
        # Mount secrets as files instead
        volumeMounts:
        – name: db-credentials
          mountPath: /etc/secrets/database
          readOnly: true
        – name: api-keys
          mountPath: /etc/secrets/api
          readOnly: true
          
        # Health checks
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          
        # Resource limits
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi
            
      volumes:
      – name: db-credentials
        secret:
          secretName: payment-db-credentials
          defaultMode: 0400  # Read-only for owner only
      – name: api-keys
        secret:
          secretName: payment-api-keys
          defaultMode: 0400
          items:
          – key: stripe-secret-key
            path: stripe_key
          – key: jwt-secret
            path: jwt_secret

apiVersion: v1
kind: Secret
metadata:
  name: payment-db-credentials
  namespace: payments
type: Opaque
data:
  username: cGF5bWVudHNfZGJfdXNlcg==  # payments_db_user (base64)
  password: c3VwZXJfc2VjcmV0X3Bhc3N3b3JkXzEyMw==  # super_secret_password_123 (base64)
  host: cGF5bWVudHMtZGIucmRzLmFtYXpvbmF3cy5jb20=  # payments-db.rds.amazonaws.com (base64)
---
apiVersion: v1
kind: Secret
metadata:
  name: payment-api-keys
  namespace: payments
type: Opaque
data:
  stripe-secret-key: c2tfbGl2ZV9hYmMxMjNkZWY0NTY=  # sk_live_abc123def456 (base64)
  jwt-secret: ZXlKaGJHY2lPaUpJVXpJMU5pSjk=  # eyJhbGciOiJIUzI1NiJ9 (base64)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: payments
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payment-api
  template:
    metadata:
      labels:
        app: payment-api
    spec:
      serviceAccountName: payment-api-sa
      containers:
      - name: payment-service
        image: payment-api:v1.2.0
        ports:
        - containerPort: 8080
        # NO environment variables for secrets!
        env:
        - name: SERVER_PORT
          value: "8080"
        - name: LOG_LEVEL
          value: "INFO"
        
        # Mount secrets as files instead
        volumeMounts:
        - name: db-credentials
          mountPath: /etc/secrets/database
          readOnly: true
        - name: api-keys
          mountPath: /etc/secrets/api
          readOnly: true
          
        # Health checks
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          
        # Resource limits
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi
            
      volumes:
      - name: db-credentials
        secret:
          secretName: payment-db-credentials
          defaultMode: 0400  # Read-only for owner only
      - name: api-keys
        secret:
          secretName: payment-api-keys
          defaultMode: 0400
          items:
          - key: stripe-secret-key
            path: stripe_key
          - key: jwt-secret
            path: jwt_secret

Application Code Changes

Before (Environment Variables):

// main.go - INSECURE
package main

import (
    "os"
    "fmt"
    "log"
)

func main() {
    // These appear in process listing!
    dbPassword := os.Getenv("DB_PASSWORD")
    stripeKey := os.Getenv("STRIPE_SECRET_KEY")
    jwtSecret := os.Getenv("JWT_SECRET")
    
    // Accidentally logged!
    log.Printf("Starting with DB password: %s", dbPassword)
    
    // Initialize services...
}

After (File Reading):

// main.go - SECURE
package main

import (
    "io/ioutil"
    "log"
    "strings"
)

func readSecret(filepath string) (string, error) {
    content, err := ioutil.ReadFile(filepath)
    if err != nil {
        return "", err
    }
    return strings.TrimSpace(string(content)), nil
}

func main() {
    // Read secrets from mounted files
    dbPassword, err := readSecret("/etc/secrets/database/password")
    if err != nil {
        log.Fatal("Failed to read database password:", err)
    }
    
    stripeKey, err := readSecret("/etc/secrets/api/stripe_key")
    if err != nil {
        log.Fatal("Failed to read Stripe key:", err)
    }
    
    jwtSecret, err := readSecret("/etc/secrets/api/jwt_secret")
    if err != nil {
        log.Fatal("Failed to read JWT secret:", err)
    }
    
    // Secrets never appear in logs or process listing
    log.Println("Starting payment service with secrets loaded from files")
    
    // Initialize services with loaded secrets...
}

🎯 Real-World Use Case Scenarios

Scenario 1: E-commerce Platform

# E-commerce secrets management
apiVersion: v1
kind: Secret
metadata:
  name: ecommerce-secrets
  namespace: ecommerce
type: Opaque
data:
  # Database credentials
  db-username: ZWNvbW1lcmNlX3VzZXI=
  db-password: cGFzc3dvcmQxMjM0NTY=
  
  # Payment gateway keys
  paypal-client-id: QVc5dGVzdF9jbGllbnRfaWQ=
  paypal-secret: QVc5dGVzdF9zZWNyZXQ=
  stripe-publishable: cGtfdGVzdF9hYmMxMjM=
  stripe-secret: c2tfdGVzdF94eXo3ODk=
  
  # Email service
  sendgrid-api-key: U0cuYWJjZGVmZ2hpams=
  
  # Redis cache
  redis-password: cmVkaXNwYXNzd29yZA==
—
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ecommerce-api
  namespace: ecommerce
spec:
  replicas: 5
  selector:
    matchLabels:
      app: ecommerce-api
  template:
    metadata:
      labels:
        app: ecommerce-api
    spec:
      containers:
      – name: api
        image: ecommerce-api:v2.1.0
        ports:
        – containerPort: 8080
        
        # Environment variables for non-sensitive config only
        env:
        – name: SERVER_PORT
          value: “8080”
        – name: REDIS_HOST
          value: “redis-cluster.cache.svc.cluster.local”
        – name: DB_HOST
          value: “postgres.database.svc.cluster.local”
        
        # Mount all secrets as files
        volumeMounts:
        – name: database-creds
          mountPath: /etc/secrets/database
          readOnly: true
        – name: payment-keys
          mountPath: /etc/secrets/payments
          readOnly: true
        – name: service-keys
          mountPath: /etc/secrets/services
          readOnly: true
          
        # Security context
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          
      volumes:
      – name: database-creds
        secret:
          secretName: ecommerce-secrets
          defaultMode: 0400
          items:
          – key: db-username
            path: username
          – key: db-password
            path: password
      – name: payment-keys
        secret:
          secretName: ecommerce-secrets
          defaultMode: 0400
          items:
          – key: paypal-client-id
            path: paypal_client_id
          – key: paypal-secret
            path: paypal_secret
          – key: stripe-publishable
            path: stripe_publishable
          – key: stripe-secret
            path: stripe_secret
      – name: service-keys
        secret:
          secretName: ecommerce-secrets
          defaultMode: 0400
          items:
          – key: sendgrid-api-key
            path: sendgrid_key
          – key: redis-password
            path: redis_password

# E-commerce secrets management
apiVersion: v1
kind: Secret
metadata:
  name: ecommerce-secrets
  namespace: ecommerce
type: Opaque
data:
  # Database credentials
  db-username: ZWNvbW1lcmNlX3VzZXI=
  db-password: cGFzc3dvcmQxMjM0NTY=
  
  # Payment gateway keys
  paypal-client-id: QVc5dGVzdF9jbGllbnRfaWQ=
  paypal-secret: QVc5dGVzdF9zZWNyZXQ=
  stripe-publishable: cGtfdGVzdF9hYmMxMjM=
  stripe-secret: c2tfdGVzdF94eXo3ODk=
  
  # Email service
  sendgrid-api-key: U0cuYWJjZGVmZ2hpams=
  
  # Redis cache
  redis-password: cmVkaXNwYXNzd29yZA==
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ecommerce-api
  namespace: ecommerce
spec:
  replicas: 5
  selector:
    matchLabels:
      app: ecommerce-api
  template:
    metadata:
      labels:
        app: ecommerce-api
    spec:
      containers:
      - name: api
        image: ecommerce-api:v2.1.0
        ports:
        - containerPort: 8080
        
        # Environment variables for non-sensitive config only
        env:
        - name: SERVER_PORT
          value: "8080"
        - name: REDIS_HOST
          value: "redis-cluster.cache.svc.cluster.local"
        - name: DB_HOST
          value: "postgres.database.svc.cluster.local"
        
        # Mount all secrets as files
        volumeMounts:
        - name: database-creds
          mountPath: /etc/secrets/database
          readOnly: true
        - name: payment-keys
          mountPath: /etc/secrets/payments
          readOnly: true
        - name: service-keys
          mountPath: /etc/secrets/services
          readOnly: true
          
        # Security context
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          
      volumes:
      - name: database-creds
        secret:
          secretName: ecommerce-secrets
          defaultMode: 0400
          items:
          - key: db-username
            path: username
          - key: db-password
            path: password
      - name: payment-keys
        secret:
          secretName: ecommerce-secrets
          defaultMode: 0400
          items:
          - key: paypal-client-id
            path: paypal_client_id
          - key: paypal-secret
            path: paypal_secret
          - key: stripe-publishable
            path: stripe_publishable
          - key: stripe-secret
            path: stripe_secret
      - name: service-keys
        secret:
          secretName: ecommerce-secrets
          defaultMode: 0400
          items:
          - key: sendgrid-api-key
            path: sendgrid_key
          - key: redis-password
            path: redis_password

Scenario 2: Microservices with Service Mesh

# Auth service with mTLS certificates
apiVersion: v1
kind: Secret
metadata:
  name: auth-service-certs
  namespace: auth
type: kubernetes.io/tls
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t...
  tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0t...
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t...
---
apiVersion: v1
kind: Secret
metadata:
  name: auth-service-secrets
  namespace: auth
type: Opaque
data:
  jwt-private-key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQ==
  jwt-public-key: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0=
  oauth-client-secret: b2F1dGhfc2VjcmV0XzEyMzQ1Ng==
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth-service
  namespace: auth
spec:
  replicas: 3
  selector:
    matchLabels:
      app: auth-service
  template:
    metadata:
      labels:
        app: auth-service
      annotations:
        # Istio sidecar injection
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: auth-service
        image: auth-service:v1.5.0
        ports:
        - containerPort: 8080
        - containerPort: 8443  # HTTPS port
        
        # Mount TLS certificates
        volumeMounts:
        - name: tls-certs
          mountPath: /etc/ssl/certs
          readOnly: true
        - name: auth-secrets
          mountPath: /etc/secrets/auth
          readOnly: true
          
        # Liveness probe using HTTPS
        livenessProbe:
          httpGet:
            path: /health
            port: 8443
            scheme: HTTPS
          initialDelaySeconds: 30
          
      volumes:
      - name: tls-certs
        secret:
          secretName: auth-service-certs
          defaultMode: 0400
      - name: auth-secrets
        secret:
          secretName: auth-service-secrets
          defaultMode: 0400

Scenario 3: CI/CD Pipeline Secrets

# Jenkins pipeline secrets
apiVersion: v1
kind: Secret
metadata:
  name: ci-cd-secrets
  namespace: jenkins
type: Opaque
data:
  # Git credentials
  git-username: Z2l0aHViX3VzZXI=
  git-token: Z2hwX2FiY2RlZmdoaWprbG1ub3A=
  
  # Docker registry
  docker-username: ZG9ja2VyX3VzZXI=
  docker-password: ZG9ja2VyX3Bhc3N3b3Jk
  
  # Cloud credentials
  aws-access-key: QUtJQUlPU0ZPRE5ON0VYQU1QTEU=
  aws-secret-key: d0phbHJRVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQ==
  
  # Kubernetes config
  kubeconfig: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoK...
---
apiVersion: batch/v1
kind: Job
metadata:
  name: deploy-application
  namespace: jenkins
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: deploy
        image: kubectl:v1.25.0
        command: ["/bin/sh"]
        args:
        - -c
        - |
          # Read secrets from files
          export AWS_ACCESS_KEY_ID=$(cat /etc/secrets/aws/access-key)
          export AWS_SECRET_ACCESS_KEY=$(cat /etc/secrets/aws/secret-key)
          
          # Use kubeconfig from file
          export KUBECONFIG=/etc/secrets/k8s/config
          
          # Deploy application
          kubectl apply -f deployment.yaml
          
        volumeMounts:
        - name: aws-creds
          mountPath: /etc/secrets/aws
          readOnly: true
        - name: k8s-config
          mountPath: /etc/secrets/k8s
          readOnly: true
          
      volumes:
      - name: aws-creds
        secret:
          secretName: ci-cd-secrets
          defaultMode: 0400
          items:
          - key: aws-access-key
            path: access-key
          - key: aws-secret-key
            path: secret-key
      - name: k8s-config
        secret:
          secretName: ci-cd-secrets
          defaultMode: 0400
          items:
          - key: kubeconfig
            path: config

🔄 Advanced Patterns: Secret Rotation

1. External Secrets Operator

# Using External Secrets Operator with AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          accessKeyID:
            name: aws-creds
            key: access-key-id
          secretAccessKey:
            name: aws-creds
            key: secret-access-key
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-db-secret
  namespace: production
spec:
  refreshInterval: 300s  # Refresh every 5 minutes
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: payment-db-credentials
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: production/payment-db
      property: username
  - secretKey: password
    remoteRef:
      key: production/payment-db
      property: password

2. HashiCorp Vault Integration

# Vault Agent sidecar pattern
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
  namespace: production
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "payment-service"
        vault.hashicorp.com/agent-inject-secret-db-creds: "secret/data/db"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "secret/data/db" -}}
          username={{ .Data.data.username }}
          password={{ .Data.data.password }}
          {{- end -}}
    spec:
      containers:
      - name: app
        image: payment-service:v1.0.0
        volumeMounts:
        - name: vault-secrets
          mountPath: /vault/secrets
          readOnly: true
        # Application reads from /vault/secrets/db-creds
      volumes:
      - name: vault-secrets
        emptyDir:
          medium: Memory  # Store in memory only

🛡️ Security Best Practices

1. RBAC for Secrets

# Limit secret access with RBAC
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: payments
  name: payment-secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["payment-db-credentials", "payment-api-keys"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payment-service-binding
  namespace: payments
subjects:
- kind: ServiceAccount
  name: payment-api-sa
  namespace: payments
roleRef:
  kind: Role
  name: payment-secret-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-api-sa
  namespace: payments

2. Secret Encryption at Rest

# EncryptionConfiguration for etcd
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: c2VjcmV0IGlzIHNlY3VyZQ==
  - identity: {}

3. Monitoring Secret Access

# Falco rule for secret access monitoring
- rule: Secret Access
  desc: Detect access to Kubernetes secrets
  condition: >
    kevt and ka and
    ka.verb in (get, list) and
    ka.resource.resource=secrets and
    ka.response_code<400
  output: >
    Secret accessed (user=%ka.user.name verb=%ka.verb 
    resource=%ka.target.resource reason=%ka.response_reason)
  priority: INFO
  tags: [k8s_audit, secrets]

📊 Comparison Matrix

Aspect	Environment Variables	File Mounting
Process Visibility	❌ Visible in `ps aux`	✅ Hidden from process listing
Log Exposure	❌ Often logged	✅ Never in application logs
Container Sharing	❌ Exposed to all containers	✅ Per-container control
Runtime Updates	❌ Requires restart	✅ Automatic with `refreshInterval`
File Permissions	❌ Not applicable	✅ Fine-grained (0400, 0440)
Audit Trail	❌ Limited	✅ File access logs
Memory Dumps	❌ Included in dumps	✅ Excluded from dumps
Debugging Safety	❌ Easy to leak	✅ Safe to debug
Secret Rotation	❌ Manual restart needed	✅ Automatic with external tools
Multi-tenant Safety	❌ Namespace-wide exposure	✅ Pod-specific mounting

🎯 Migration Checklist

Phase 1: Assessment

[ ] Audit all deployments using env for secrets
[ ] Identify applications that need code changes
[ ] Plan downtime windows for critical services
[ ] Test file-reading code in development

Phase 2: Implementation

[ ] Update application code to read from files
[ ] Create new secret manifests with proper permissions
[ ] Update deployment manifests with volume mounts
[ ] Test secret rotation scenarios

Phase 3: Deployment

[ ] Deploy to staging environment first
[ ] Validate secret access and application functionality
[ ] Monitor for any secret-related errors
[ ] Roll out to production with gradual deployment

Phase 4: Cleanup

[ ] Remove environment variable references
[ ] Update documentation and runbooks
[ ] Set up monitoring for secret file access
[ ] Implement automated secret rotation

Bottom Line: File-based secret mounting is not just a best practice—it’s essential for production security. The small code changes required are vastly outweighed by the security benefits and operational improvements you gain.

Tags: Environment variables, K8s secrets, kubernetes, Microservice, Service mesh