Kubernetes Interview Series
rjshk013
November 8, 2025
Kubernetes Interview Series – Part 2
🎯 Kubernetes API Server – 15 Quiz Questions & Answers
Master the cluster’s front door – from basics to production scenarios!
📚 Fundamentals (Questions 1-10)
1️⃣ Where does the API Server fit into the Kubernetes architecture?
Answer: The API Server is a core control plane component that runs on control-plane (master) nodes and acts as the front door to the cluster.
Architectural Position:
┌─────────────────────────────────┐
│ Control Plane (Master) │
│ ┌───────────────────────────┐ │
│ │ 🚪 API Server │ │ ← You are here
│ └───────────────────────────┘ │
│ ┌─────────┐ ┌──────────────┐ │
│ │ etcd │ │ Controller │ │
│ │ │ │ Manager │ │
│ └─────────┘ └──────────────┘ │
│ ┌─────────────────────────┐ │
│ │ Scheduler │ │
│ └─────────────────────────┘ │
└─────────────────────────────────┘
↕️
┌─────────────────────────────────┐
│ Worker Nodes │
│ ┌──────────┐ ┌──────────┐ │
│ │ kubelet │ │ kubelet │ │
│ └──────────┘ └──────────┘ │
└─────────────────────────────────┘
Key Point: It’s the only component that users and other control plane components interact with directly.
2️⃣ What is the main function of the API Server?
Answer: The API Server exposes the Kubernetes API and acts as the single entry point for all cluster operations.
Primary Functions:
- RESTful API Gateway
- Exposes HTTP/HTTPS endpoints
- Handles CRUD operations (Create, Read, Update, Delete)
- Request Validator
- Validates all incoming requests
- Ensures data consistency
- State Manager
- Persists cluster state to etcd
- Serves as the source of truth
Example API Call:
# Create a Pod via API Server
kubectl create -f pod.yaml
# ↓ Translates to:
curl -X POST https://api-server:6443/api/v1/namespaces/default/pods \
-H "Authorization: Bearer $TOKEN" \
-d @pod.yaml
Interview Tip: All kubectl commands eventually become REST API calls to the API Server!
3️⃣ Who communicates with the API Server?
Answer: Everyone and everything in the cluster communicates with the API Server!
External Clients:
- 👤 Users via kubectl
- 🖥️ CI/CD tools (GitHub Actions, Jenkins)
- 🔧 Management tools (Helm, ArgoCD)
Internal Components:
- 🎯 Controllers (ReplicaSet, Deployment, StatefulSet)
- 📅 Scheduler (to bind Pods to nodes)
- 🤖 Kubelet (worker nodes reporting status)
- 💾 etcd (for state persistence)
Communication Flow:
kubectl → API Server → etcd
↓
Controller Manager
↓
Scheduler
↓
Kubelet
Real-World Example:
# When you run:
kubectl get pods
# Flow:
1. kubectl → API Server (with your credentials)
2. API Server → AuthN/AuthZ checks
3. API Server → etcd (query Pod data)
4. etcd → API Server (return data)
5. API Server → kubectl (formatted response)
4️⃣ Where does the API Server store the cluster state?
Answer: The API Server stores all cluster state data in etcd.
Important Details:
What’s Stored:
- All Kubernetes objects (Pods, Services, ConfigMaps, Secrets)
- Node registration data
- Resource quotas
- RBAC policies
- Custom resources
API Server’s Role:
- Acts as a gateway to etcd
- Validates data before writing
- Handles optimistic concurrency (resource versions)
- Never caches writes (always writes to etcd)
Key Point: The API Server is the ONLY component that talks to etcd directly.
Verification:
# Check etcd data (from control plane)
ETCDCTL_API=3 etcdctl get /registry/pods/default/my-pod \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
5️⃣ Does the API Server run on worker nodes?
Answer: ❌ No. It ONLY runs on control-plane (master) nodes.
Why?
- Control plane components need isolation from workloads
- Security separation (control vs. data plane)
- Resource protection
Architecture:
Control Plane Nodes:
✅ API Server
✅ etcd
✅ Controller Manager
✅ Scheduler
Worker Nodes:
✅ kubelet
✅ kube-proxy
✅ Container runtime
❌ API Server (NOT here!)
How Workers Communicate:
# Kubelet on worker node talks TO API Server
kubelet → API Server (reports node/pod status)
← (receives pod specifications)
Verification:
# Check where API Server runs
kubectl get pods -n kube-system | grep apiserver
# You'll see it only on control-plane nodes
# Check node labels
kubectl get nodes --show-labels | grep control-plane
6️⃣ What are some core functions of the API Server?
Answer:
The 5 Core Functions:
1. Authentication (AuthN) 🔐
- Verifies “Who are you?”
- Methods: certificates, tokens, OIDC, webhooks
2. Authorization (AuthZ) 🚦
- Verifies “What can you do?”
- Methods: RBAC, ABAC, Node authorization
3. Admission Control ✅
- Validates and mutates requests
- Examples: ResourceQuota, PodSecurityPolicy, webhooks
4. Validation 📝
- Checks if resource definition is valid
- Schema validation, required fields
5. Persistence 💾
- Stores validated objects in etcd
- Ensures consistency
Request Pipeline:
Request → AuthN → AuthZ → Admission → Validation → etcd
↓ ↓ ↓ ↓ ↓ ↓
Who? Allowed? Mutate? Valid? Store? Done
6. RESTful API Serving 🌐
- Exposes resources via HTTP/HTTPS
- Supports watch streams for real-time updates
Example Request Flow:
kubectl apply -f deployment.yaml
1. AuthN: Certificate validated ✓
2. AuthZ: RBAC check "can create deployments?" ✓
3. Admission: ResourceQuota check ✓
4. Validation: Schema check ✓
5. Persistence: Write to etcd ✓
6. Response: deployment.apps/nginx created
7️⃣ How does the API Server interact with etcd?
Answer: The API Server is the ONLY client that talks to etcd directly.
Key Characteristics:
1. Exclusive Access 🔒
- No other component accesses etcd directly
- All data flows through API Server
- This ensures validation and consistency
2. Read Operations 📖
# When you run:
kubectl get pods
# API Server:
1. Queries etcd for Pod objects
2. Filters based on namespace/labels
3. Returns formatted response
3. Write Operations ✍️
# When you run:
kubectl create -f pod.yaml
# API Server:
1. Validates the Pod spec
2. Writes to etcd at key: /registry/pods/namespace/name
3. Waits for etcd confirmation
4. Returns success/failure
4. Watch Mechanism 👀
# Controllers watching for changes:
Controller → API Server → etcd (watch key prefix)
← (event stream)
etcd Data Structure:
/registry/
├── pods/
│ ├── default/
│ │ ├── nginx-pod
│ │ └── app-pod
│ └── kube-system/
│ └── coredns-pod
├── services/
├── deployments/
└── configmaps/
Why This Design?
- ✅ Centralized validation
- ✅ Consistent access control
- ✅ Audit logging
- ✅ Version control
8️⃣ What happens if the API Server is down?
Answer:
Impact Analysis:
❌ What BREAKS:
- ❌ No new Pods can be created
- ❌ kubectl commands fail
- ❌ Scaling operations frozen
- ❌ Controllers can’t reconcile
- ❌ Admission webhooks unavailable
- ❌ No updates to cluster state
✅ What CONTINUES Working:
- ✅ Existing Pods keep running (kubelet manages them)
- ✅ Networking still works (kube-proxy maintains rules)
- ✅ Services continue routing traffic
- ✅ Container restarts happen (kubelet handles)
- ✅ Node-local operations continue
Real-World Scenario:
# Before API Server goes down:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
app-1 1/1 Running 0 5m
# API Server crashes
$ kubectl get pods
Error from server: the server is currently unable to handle the request
# But check the actual pod:
$ docker ps # (on the worker node)
CONTAINER ID IMAGE STATUS
abc123 my-app Up 5 minutes # Still running!
Recovery:
# In HA setup (multiple API Servers)
- Load balancer redirects to healthy instance
- Zero downtime for cluster operations
# Single API Server
$ systemctl restart kube-apiserver
# Or
$ kubectl -n kube-system delete pod kube-apiserver-xxx
Production Best Practice: Always run multiple API Server replicas (typically 3 or 5) behind a load balancer for HA.
9️⃣ Is the API Server stateless or stateful?
Answer: The API Server is STATELESS.
What This Means:
Stateless Design:
- ✅ No local data stored
- ✅ All state lives in etcd
- ✅ Each request is independent
- ✅ No session affinity required
Benefits:
1. Horizontal Scaling 📈
# Run multiple replicas
apiVersion: v1
kind: Pod
metadata:
name: kube-apiserver-1
spec:
containers:
- name: kube-apiserver
image: k8s.gcr.io/kube-apiserver:v1.28.0
---
apiVersion: v1
kind: Pod
metadata:
name: kube-apiserver-2 # Another instance
2. Load Balancing ⚖️
kubectl → Load Balancer → API Server 1
→ API Server 2
→ API Server 3
↓
etcd
3. Easy Recovery 🔄
- If one crashes, others handle requests
- No data loss (state in etcd)
- Quick failover
Example HA Setup:
# Check multiple API Servers
$ kubectl get pods -n kube-system | grep apiserver
kube-apiserver-master-1 1/1 Running
kube-apiserver-master-2 1/1 Running
kube-apiserver-master-3 1/1 Running
Load Balancer Config:
apiVersion: v1
kind: Endpoints
metadata:
name: kubernetes
namespace: default
subsets:
- addresses:
- ip: 10.0.1.10 # API Server 1
- ip: 10.0.1.11 # API Server 2
- ip: 10.0.1.12 # API Server 3
ports:
- port: 6443
Interview Trap: “Does API Server cache data?” → It has read caches for performance, but NEVER caches writes. All writes go to etcd immediately.
🔟 What are some sub-components/features inside the API Server?
Answer:
Major Sub-Components:
1. Request Handling Pipeline 🔄
HTTP Request → Decoder → Authentication → Authorization →
Admission Controllers → Validation → etcd Write → Response
2. Aggregated API Layer 🔌
- Extends Kubernetes API with CRDs
- Supports API extensions (metrics-server, custom APIs)
- Example:
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta1.metrics.k8s.io
spec:
service:
name: metrics-server
namespace: kube-system
3. API Groups & Versions 📚
- Core:
v1(Pods, Services) - Apps:
apps/v1(Deployments, StatefulSets) - Batch:
batch/v1(Jobs, CronJobs) - Custom:
mycompany.io/v1(CRDs)
Check Available APIs:
kubectl api-resources
kubectl api-versions
4. OpenAPI Spec (Swagger) 📖
# Access OpenAPI spec
kubectl proxy &
curl http://localhost:8001/openapi/v2
5. Watch Mechanism 👁️
# Efficient resource watching
kubectl get pods --watch
# Uses HTTP/2 streaming for real-time updates
6. Admission Controllers 🛡️ Built-in:
NamespaceLifecycleLimitRangerResourceQuotaPodSecurityPolicy
Custom:
- Mutating webhooks
- Validating webhooks
7. Authentication Plugins 🔐
- Client certificates
- Bearer tokens
- Bootstrap tokens
- OpenID Connect (OIDC)
- Webhook authentication
8. Authorization Modes 🚦
- RBAC (Role-Based Access Control)
- ABAC (Attribute-Based)
- Node authorization
- Webhook authorization
🚀 Advanced Production Scenarios (Questions 11-15)
1️⃣1️⃣ Explain the complete API Server request flow with an example
Answer:
The Complete Journey of a kubectl Request
Scenario: kubectl create -f nginx-pod.yaml
Step-by-Step Flow:
Phase 1: Client → API Server 🔗
1. kubectl reads nginx-pod.yaml
2. Parses YAML to JSON
3. Makes HTTP POST to: https://api-server:6443/api/v1/namespaces/default/pods
4. Includes authentication credentials (certificate/token)
Phase 2: Authentication 🔐
API Server checks:
- Is the certificate valid?
- Is the token not expired?
- Is the user known to the cluster?
Options:
✓ X.509 Client Certificates
✓ Service Account Tokens
✓ OIDC Tokens
✓ Webhook Authentication
Result: User identified as "john@example.com"
Phase 3: Authorization 🚦
API Server checks RBAC:
- Does user have "create" permission on "pods"?
- In namespace "default"?
Query:
Subject: john@example.com
Verb: create
Resource: pods
Namespace: default
Check against RoleBindings/ClusterRoleBindings
Result: ✓ Authorized (user has "edit" role in default namespace)
Phase 4: Admission Control ✅
4a. Mutating Admission (can modify request)
# Example: Pod Security Policy adds security context
Original:
spec:
containers:
- name: nginx
image: nginx
After mutation:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
containers:
- name: nginx
image: nginx
Mutating Admission Webhooks in order:
MutatingAdmissionWebhook(custom webhooks)PodSecurityPolicyLimitRanger(adds default resource limits)ServiceAccount(injects service account token)
4b. Validating Admission (accept/reject, no modification)
# Example: ResourceQuota check
Check:
Namespace "default" has quota:
requests.cpu: 10 cores
Used: 8 cores
Requested by this pod: 1 core
Result: ✓ Within quota (8 + 1 = 9 < 10)
Validating Admission Webhooks:
ResourceQuotaPodSecurityPolicy(validation)ValidatingAdmissionWebhook(custom webhooks)
Phase 5: Schema Validation 📝
API Server checks:
- Required fields present? (name, containers)
- Valid values? (restartPolicy: Always/OnFailure/Never)
- Correct types? (port: integer, not string)
- API version supported? (v1 is valid)
Validation against OpenAPI schema:
✓ All required fields present
✓ No unknown fields
✓ Correct data types
Phase 6: Persistence 💾
1. API Server writes to etcd:
Key: /registry/pods/default/nginx-pod
Value: <serialized pod object>
2. etcd confirms write with revision number
3. API Server gets confirmation from etcd
Phase 7: Response 📤
HTTP 201 Created
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"name": "nginx-pod",
"namespace": "default",
"uid": "abc-123",
"resourceVersion": "12345",
"creationTimestamp": "2025-01-01T10:00:00Z"
},
"spec": { ... },
"status": {
"phase": "Pending"
}
}
Phase 8: Controller Watches 👀
1. Scheduler watches new Pods in "Pending" state
2. Scheduler picks a node and updates Pod.spec.nodeName
3. This triggers another API Server flow (update request)
4. Kubelet on assigned node watches for Pods
5. Kubelet sees the Pod and creates containers
Complete Timeline:
T+0ms: kubectl → API Server (POST request)
T+5ms: Authentication ✓
T+10ms: Authorization ✓
T+50ms: Admission controllers (mutating + validating) ✓
T+60ms: Schema validation ✓
T+70ms: Write to etcd
T+80ms: etcd confirms
T+85ms: Response to kubectl ✓
T+100ms: Scheduler watches the event
T+200ms: Scheduler assigns node
T+300ms: Kubelet creates container
Error Example – Authorization Failure:
$ kubectl create -f pod.yaml
Error from server (Forbidden): pods is forbidden:
User "john@example.com" cannot create resource "pods"
in API group "" in the namespace "default"
# Flow stopped at Phase 3 (Authorization)
Diagram:
kubectl (client)
↓
[TLS/HTTPS]
↓
API Server (6443)
↓
1. Decode Request (YAML→JSON)
↓
2. Authentication (Who?)
├─ X.509 Cert ✓
├─ Token ✓
└─ OIDC ✓
↓
3. Authorization (Can they?)
└─ RBAC Check ✓
↓
4. Admission Control
├─ Mutating Webhooks
└─ Validating Webhooks
↓
5. Schema Validation
↓
6. Serialize & Write to etcd
↓
7. Return Response
↓
kubectl (receives confirmation)
↓
Controller watches event
↓
Reconciliation begins
Pro Tip: Enable audit logging to see this entire flow for every request!
# API Server flag
--audit-log-path=/var/log/kubernetes/audit.log
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
1️⃣2️⃣ How do you troubleshoot “The connection to the server was refused” error?
Answer:
Error Message:
$ kubectl get pods
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Step-by-Step Troubleshooting:
Step 1: Check Cluster Context 🔍
# Verify current context
kubectl config current-context
# Check cluster endpoint
kubectl config view --minify | grep server
# Should show something like:
# server: https://10.0.1.10:6443
Step 2: Verify API Server is Running 🏃
# On control plane node
sudo systemctl status kube-apiserver
# Or for static pod:
kubectl get pods -n kube-system | grep apiserver
# Or check Docker/containerd:
sudo docker ps | grep apiserver
sudo crictl ps | grep apiserver
Step 3: Check API Server Logs 📋
# If running as systemd service
sudo journalctl -u kube-apiserver -f
# If running as static pod
kubectl logs -n kube-system kube-apiserver-master-1
# Or directly from Docker
sudo docker logs <api-server-container-id>
Step 4: Verify Network Connectivity 🌐
# Can you reach the API Server endpoint?
curl -k https://10.0.1.10:6443/version
# Check if port is listening
sudo netstat -tlnp | grep 6443
# Or
sudo ss -tlnp | grep 6443
# Test from worker node
telnet 10.0.1.10 6443
Step 5: Check Firewall Rules 🔥
# Ensure port 6443 is open
sudo iptables -L -n | grep 6443
# For firewalld
sudo firewall-cmd --list-ports
# For cloud providers, check security groups
aws ec2 describe-security-groups --group-ids sg-xxx
Step 6: Verify etcd is Healthy 💾
# API Server depends on etcd
ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health
# Should show:
# 127.0.0.1:2379 is healthy
Step 7: Check Certificates 🔐
# Verify API Server certificate not expired
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
# Check certificate validity
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -A 2 Validity
Step 8: Check kubeconfig 📝
# Verify kubeconfig is properly configured
cat ~/.kube/config
# Check if certificate is valid
kubectl config view --raw
# Test with explicit kubeconfig
kubectl --kubeconfig=/path/to/config get pods
Common Causes & Solutions:
Cause 1: API Server Crashed
# Check crash logs
sudo journalctl -u kube-apiserver --no-pager | tail -50
# Common issues:
# - etcd unreachable
# - Certificate expired
# - Out of memory
# - Configuration error
# Restart API Server
sudo systemctl restart kube-apiserver
Cause 2: Wrong Endpoint in kubeconfig
# Fix the server endpoint
kubectl config set-cluster kubernetes \
--server=https://correct-api-server-ip:6443
# Or edit directly
kubectl config view
Cause 3: Certificate Issues
# Regenerate certificates (kubeadm)
sudo kubeadm init phase certs apiserver
# Or manually renew
sudo kubeadm certs renew apiserver
Cause 4: etcd is Down
# Check etcd health
sudo systemctl status etcd
# Restart etcd
sudo systemctl restart etcd
# Then restart API Server
sudo systemctl restart kube-apiserver
Cause 5: Resource Exhaustion
# Check if node has resources
free -h
df -h
# Check API Server resource usage
kubectl top pod -n kube-system kube-apiserver-xxx
Diagnostic Script:
#!/bin/bash
echo "=== Kubernetes API Server Diagnostics ==="
echo "1. Current Context:"
kubectl config current-context
echo "2. API Server Endpoint:"
kubectl config view --minify | grep server
echo "3. API Server Pod Status:"
kubectl get pods -n kube-system | grep apiserver
echo "4. Test Connectivity:"
API_SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
curl -k $API_SERVER/version
echo "5. Check Port 6443:"
sudo netstat -tlnp | grep 6443
echo "6. etcd Health:"
ETCDCTL_API=3 etcdctl endpoint health \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
echo "7. Certificate Expiry:"
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
echo "=== Diagnostics Complete ==="
Quick Recovery Commands:
# For kubeadm clusters
sudo kubeadm reset
sudo kubeadm init
# Restore from backup
sudo cp /backup/admin.conf /etc/kubernetes/admin.conf
export KUBECONFIG=/etc/kubernetes/admin.conf
# Force restart all control plane components
sudo systemctl restart kubelet
1️⃣3️⃣ How do you secure the API Server in production?
Answer:
Multi-Layered Security Approach:
Layer 1: Network Security 🌐
1. Private API Server
# Don't expose API Server publicly
# Use bastion host or VPN
API Server: 10.0.1.10:6443 (private IP)
Public Access: Via bastion/VPN only
2. Firewall Rules
# Only allow specific IPs
sudo iptables -A INPUT -p tcp --dport 6443 -s 10.0.0.0/16 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 6443 -j DROP
# For cloud providers (AWS example)
aws ec2 authorize-security-group-ingress \
--group-id sg-xxx \
--protocol tcp \
--port 6443 \
--source-group sg-yyy # Only from specific security group
3. TLS Everywhere
# API Server flags
--tls-cert-file=/etc/kubernetes/pki/apiserver.crt
--tls-private-key-file=/etc/kubernetes/pki/apiserver.key
--client-ca-file=/etc/kubernetes/pki/ca.crt
--tls-min-version=VersionTLS13 # Use TLS 1.3
Layer 2: Authentication 🔐
1. Disable Anonymous Access
# API Server flag
--anonymous-auth=false # No anonymous requests allowed
2. Use Strong Authentication Methods
# Prefer OIDC for users
--oidc-issuer-url=https://accounts.google.com
--oidc-client-id=kubernetes
--oidc-username-claim=email
--oidc-groups-claim=groups
# Use X.509 certificates for service accounts
--service-account-key-file=/etc/kubernetes/pki/sa.pub
3. Rotate Tokens Regularly
# Service account tokens should expire
--service-account-issuer=https://kubernetes.default.svc
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key
--service-account-max-token-expiration=24h
4. Enable Bootstrap Tokens with Limited Lifetime
apiVersion: v1
kind: Secret
metadata:
name: bootstrap-token-abc123
namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
token-id: abc123
token-secret: xyz789
expiration: 2025-12-31T23:59:59Z # Token expires
usage-bootstrap-authentication: "true"
usage-bootstrap-signing: "true"
Layer 3: Authorization 🚦
1. Enable RBAC (Required!)
# API Server flag
--authorization-mode=Node,RBAC
# Node authorization for kubelet
# RBAC for everything else
2. Principle of Least Privilege
# Good: Specific permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
# Bad: Too permissive
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"] # ❌ Don't do this!
3. Regular RBAC Audits
# Check who has cluster-admin
kubectl get clusterrolebindings -o json | \
jq -r '.items[] | select(.roleRef.name=="cluster-admin") | .metadata.name'
# Audit user permissions
kubectl auth can-i --list --as=user@example.com
Layer 4: Admission Control ✅
1. Enable Essential Admission Controllers
--enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ResourceQuota,LimitRanger,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
2. Pod Security Standards
# Enforce pod security at namespace level
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
3. Admission Webhooks for Custom Policies
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: security-policy
webhooks:
- name: validate.security.company.com
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
clientConfig:
service:
name: security-webhook
namespace: security-system
failurePolicy: Fail # Block on webhook failure
Layer 5: Audit Logging 📝
1. Enable Comprehensive Audit Logging
# API Server flags
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
2. Audit Policy Example
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log all requests at RequestResponse level
- level: RequestResponse
omitStages:
- RequestReceived
# Don't log read-only requests
- level: None
verbs: ["get", "list", "watch"]
# Always log authentication failures
- level: RequestResponse
omitStages: []
users: ["*"]
verbs: ["*"]
resources:
- group: ""
resources: ["secrets", "configmaps"]
3. Forward Logs to SIEM
# Use fluentd/logstash to forward to Elasticsearch/Splunk
# Monitor for suspicious activity:
# - Multiple failed authentication attempts
# - Privilege escalation attempts
# - Access to secrets
# - Exec into pods
Layer 6: API Server Configuration ⚙️
1. Rate Limiting
# Prevent API abuse
--max-requests-inflight=400
--max-mutating-requests-inflight=200
# Per-user rate limiting
--enable-priority-and-fairness=true
2. Disable Insecure Port
# Never use insecure port in production!
--insecure-port=0 # Disable completely
3. Restrict API Server Binding
# Don't bind to 0.0.0.0
--bind-address=10.0.1.10 # Specific internal IP
--advertise-address=10.0.1.10
4. Enable Encryption at Rest
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml
encryption-config.yaml:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-key>
- identity: {} # Fallback to unencrypted
Layer 7: Monitoring & Alerting 🚨
1. Monitor API Server Metrics
# Prometheus metrics
kubectl port-forward -n kube-system kube-apiserver-xxx 8080:8080
curl http://localhost:8080/metrics
# Key metrics:
# - apiserver_request_duration_seconds
# - apiserver_request_total
# - apiserver_current_inflight_requests
# - authentication_attempts
2. Alert on Suspicious Activity
# Prometheus AlertManager rule
- alert: HighFailedAuthentication
expr: rate(apiserver_authentication_attempts{result="failure"}[5m]) > 10
for: 5m
annotations:
summary: "High rate of failed authentication attempts"
- alert: UnauthorizedAPIAccess
expr: rate(apiserver_request_total{code=~"403|401"}[5m]) > 5
for: 5m
annotations:
summary: "Multiple unauthorized API access attempts"
Layer 8: Regular Security Practices 🔄
1. Certificate Rotation
# Auto-rotate certificates
--client-ca-file=/etc/kubernetes/pki/ca.crt
--tls-cert-file=/etc/kubernetes/pki/apiserver.crt
--tls-private-key-file=/etc/kubernetes/pki/apiserver.key
# Check expiry
kubeadm certs check-expiration
# Renew before expiry
kubeadm certs renew all
2. Regular Security Scans
# Scan for vulnerabilities
kube-bench run --targets master
# Check RBAC permissions
kubectl auth can-i --list --as=system:serviceaccount:default:default
# Audit cluster security
kube-hunter --remote <api-server-ip>
3. Keep Kubernetes Updated
# Stay within supported versions (n-2)
# Current: 1.28, 1.27, 1.26 are supported
# Upgrade regularly
kubeadm upgrade plan
kubeadm upgrade apply v1.28.0
Complete Security Checklist:
✅ Network Security
☑ API Server on private network
☑ Firewall rules restricting access
☑ TLS 1.3 enabled
☑ Certificate validation enforced
✅ Authentication
☑ Anonymous auth disabled
☑ OIDC configured for users
☑ Service account tokens expire
☑ Certificate rotation automated
✅ Authorization
☑ RBAC enabled
☑ Least privilege principle applied
☑ Regular RBAC audits
☑ No wildcard permissions
✅ Admission Control
☑ Essential admission controllers enabled
☑ Pod Security Standards enforced
☑ Custom admission webhooks deployed
✅ Audit & Monitoring
☑ Audit logging enabled
☑ Logs forwarded to SIEM
☑ Metrics monitored
☑ Alerts configured
✅ Configuration
☑ Encryption at rest enabled
☑ Insecure port disabled
☑ Rate limiting configured
☑ API server hardened
✅ Operations
☑ Regular security scans
☑ Kubernetes version updated
☑ Security patches applied
☑ Incident response plan ready
Pro Tip: Use a policy-as-code tool like OPA (Open Policy Agent) for centralized security policy management!
1️⃣4️⃣ How does the API Server handle high availability and scaling?
Answer:
High Availability (HA) Architecture:
Component Architecture:
┌────────────────────────────────────────┐
│ Load Balancer (VIP) │
│ api.k8s.company.com:6443 │
└────────────────────────────────────────┘
↓ ↓ ↓
┌───────────┐ ┌───────────┐ ┌───────────┐
│API Server │ │API Server │ │API Server │
│ Master-1 │ │ Master-2 │ │ Master-3 │
└───────────┘ └───────────┘ └───────────┘
↓ ↓ ↓
┌─────────────────────────────────────┐
│ etcd Cluster (3 nodes) │
│ (also distributed for HA) │
└─────────────────────────────────────┘
1. Multiple API Server Instances 🔄
Why Multiple Instances:
- ✅ Eliminate single point of failure
- ✅ Distribute load across instances
- ✅ Zero-downtime updates
- ✅ Increased throughput
Setup with kubeadm:
# Initialize first control plane
kubeadm init --control-plane-endpoint "api.k8s.company.com:6443" --upload-certs
# Join additional control planes
kubeadm join api.k8s.company.com:6443 \
--token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--control-plane \
--certificate-key <cert-key>
Manual Setup (systemd):
# On each master node
sudo systemctl enable kube-apiserver
sudo systemctl start kube-apiserver
# Each API Server is independent
# They don't communicate directly with each other
2. Load Balancer Configuration ⚖️
Options:
A) HAProxy (Common)
# /etc/haproxy/haproxy.cfg
frontend k8s-api
bind *:6443
mode tcp
option tcplog
default_backend k8s-api-backend
backend k8s-api-backend
mode tcp
balance roundrobin
option tcp-check
# Health check
tcp-check connect port 6443
# API Server instances
server master-1 10.0.1.10:6443 check fall 3 rise 2
server master-2 10.0.1.11:6443 check fall 3 rise 2
server master-3 10.0.1.12:6443 check fall 3 rise 2
B) Cloud Load Balancer (AWS NLB)
# Create network load balancer
aws elbv2 create-load-balancer \
--name k8s-api-lb \
--type network \
--subnets subnet-xxx subnet-yyy \
--scheme internal
# Target group with health checks
aws elbv2 create-target-group \
--name k8s-api-targets \
--protocol TCP \
--port 6443 \
--vpc-id vpc-xxx \
--health-check-protocol TCP \
--health-check-port 6443
C) keepalived (Virtual IP)
# /etc/keepalived/keepalived.conf
vrrp_instance K8S_API {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass secret123
}
virtual_ipaddress {
10.0.1.100/24 # Virtual IP
}
}
3. Stateless Design Enables Scaling 📈
Why Stateless = Easy Scaling:
# No session affinity needed
kubectl request → Load Balancer → Any API Server
# Each request is independent
# No sticky sessions required
# Can add/remove API Servers anytime
Horizontal Scaling:
# Start with 3 API Servers
Master-1, Master-2, Master-3
# Need more capacity? Add more!
kubeadm join ... --control-plane
# Now have 5 API Servers
Master-1, Master-2, Master-3, Master-4, Master-5
# Load automatically distributed
4. Leader Election (For Coordinated Tasks) 👑
Some Components Need a Leader:
# Controller Manager and Scheduler use leader election
# Only one instance actively reconciles at a time
# API Server itself does NOT use leader election
# (all API Servers are active simultaneously)
# Check current leader
kubectl get lease -n kube-system
NAME HOLDER AGE
kube-controller-manager master-1_abcd1234 5d
kube-scheduler master-2_efgh5678 5d
# Note: No API Server lease (all are active)
Why API Server Doesn’t Need Leader Election:
- Stateless operations
- etcd handles consistency
- Each instance can serve requests independently
5. etcd High Availability 💾
API Server Performance Depends on etcd:
# Run 3 or 5 etcd instances (odd numbers for quorum)
etcd-1, etcd-2, etcd-3
# Each API Server connects to all etcd instances
--etcd-servers=https://10.0.1.10:2379,https://10.0.1.11:2379,https://10.0.1.12:2379
etcd Quorum:
3 nodes: Can tolerate 1 failure (need 2 for quorum)
5 nodes: Can tolerate 2 failures (need 3 for quorum)
Formula: (N/2) + 1 nodes needed for quorum
6. Performance Tuning for Scale ⚡
API Server Flags:
# Handle more concurrent requests
--max-requests-inflight=400 # Default: 400
--max-mutating-requests-inflight=200 # Default: 200
# Increase watch cache size
--watch-cache-sizes=pods#1000,services#100
# Enable request priority and fairness
--enable-priority-and-fairness=true
# Increase worker threads
--apiserver-count=3 # Number of API Server instances
# Tune etcd connection pool
--etcd-servers-overrides=/events#https://etcd-events:2379 # Separate etcd for events
Connection Pooling:
# API Server maintains connection pools to:
# - etcd
# - Webhooks
# - Aggregated API servers
--etcd-compaction-interval=5m # Compact etcd periodically
7. Monitoring & Metrics 📊
Key Metrics to Monitor:
# Request rate
apiserver_request_total
apiserver_request_duration_seconds
# Inflight requests
apiserver_current_inflight_requests
# etcd latency (critical!)
etcd_request_duration_seconds
apiserver_storage_db_total_size_in_bytes
# Watch cache
apiserver_watch_cache_capacity
apiserver_watch_cache_capacity_increase_total
# Error rates
apiserver_request_total{code=~"5.."} # 500 errors
Example Grafana Query:
# Request rate by verb
sum(rate(apiserver_request_total[5m])) by (verb)
# P95 latency
histogram_quantile(0.95, sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (le))
# Failed requests
sum(rate(apiserver_request_total{code=~"5.."}[5m]))
8. Rolling Updates 🔄
Zero-Downtime Updates:
# With 3 API Servers behind load balancer
# Update one at a time
# Step 1: Drain master-1
kubectl drain master-1 --ignore-daemonsets
# Step 2: Update API Server on master-1
ssh master-1
sudo apt-get update && sudo apt-get upgrade -y kubelet kubeadm kubectl
sudo kubeadm upgrade node
# Step 3: Uncordon master-1
kubectl uncordon master-1
# Step 4: Repeat for master-2, master-3
# Users experience no downtime!
9. Disaster Recovery 🚨
API Server Failure Scenarios:
Scenario A: Single API Server Fails
Before:
LB → [API-1 ✓, API-2 ✓, API-3 ✗]
Load balancer detects failure via health check
Removes API-3 from pool
Requests go to API-1 and API-2
Impact: None (transparent failover)
Scenario B: Multiple API Servers Fail
Before:
LB → [API-1 ✗, API-2 ✗, API-3 ✓]
Only API-3 is handling ALL traffic
May have performance degradation
But cluster still operational
Action: Restore failed instances ASAP
Scenario C: All API Servers Fail
Cluster operations frozen
kubectl commands fail
Controllers can't reconcile
But existing Pods keep running! (kubelet manages them)
Recovery:
1. Restore API Servers from backup
2. Ensure etcd is healthy
3. Restart API Server processes
10. Capacity Planning 📈
How Many API Servers Do You Need?
Small Cluster (< 50 nodes):
API Servers: 2-3
etcd nodes: 3
Medium Cluster (50-250 nodes):
API Servers: 3-5
etcd nodes: 3-5
Consider dedicated etcd for events
Large Cluster (250-1000 nodes):
API Servers: 5-7
etcd nodes: 5
Separate etcd for events
Tune API Server flags
Very Large Cluster (1000+ nodes):
API Servers: 7+
etcd nodes: 5-7
Multiple etcd clusters (main + events + custom)
Advanced tuning required
Consider federation
Complete HA Checklist:
✅ Multiple API Server Instances
☑ At least 3 instances
☑ Across different availability zones
☑ Independent failure domains
✅ Load Balancer
☑ TCP load balancer configured
☑ Health checks enabled
☑ SSL termination at API Server (not LB)
✅ etcd High Availability
☑ 3 or 5 etcd instances
☑ Quorum maintained
☑ Regular backups
✅ Monitoring
☑ Metrics collection (Prometheus)
☑ Dashboards (Grafana)
☑ Alerts configured
✅ Disaster Recovery
☑ Backup strategy defined
☑ Recovery procedures documented
☑ Regular DR drills
✅ Performance
☑ API Server tuned for load
☑ Connection pools optimized
☑ Capacity planning done
Pro Tip: Always test failover scenarios in staging before production deployment!
1️⃣5️⃣ What is API Server aggregation layer and when would you use it?
Answer:
API Aggregation Layer extends the Kubernetes API with additional API servers without modifying the core API Server.
What Is It? 🤔
The aggregation layer allows you to:
- ✅ Add custom APIs alongside core Kubernetes APIs
- ✅ Run these APIs as separate services
- ✅ Make them appear as part of the main Kubernetes API
- ✅ Use kubectl with these custom resources
Architecture:
kubectl
↓
Main API Server (kube-apiserver)
↓
Aggregation Layer
├─ Core APIs (pods, services, deployments)
├─ metrics-server (metrics.k8s.io/v1beta1)
├─ custom-api-server-1 (mycompany.io/v1)
└─ custom-api-server-2 (ml.company.io/v1alpha1)
How It Works ⚙️
1. Register an APIService
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta1.metrics.k8s.io
spec:
service:
name: metrics-server
namespace: kube-system
port: 443
group: metrics.k8s.io
version: v1beta1
insecureSkipTLSVerify: false
groupPriorityMinimum: 100
versionPriority: 100
2. Create the Backend Service
apiVersion: v1
kind: Service
metadata:
name: metrics-server
namespace: kube-system
spec:
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: metrics-server
3. Deploy Custom API Server
apiVersion: apps/v1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server:v0.6.0
args:
- --cert-dir=/tmp
- --secure-port=8443
4. Use the Custom API
# Now you can use kubectl with this API
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes
kubectl top nodes # Uses metrics-server API
Request Flow 🔄
User: kubectl top nodes
↓
Main API Server receives request for metrics.k8s.io/v1beta1
↓
Aggregation layer sees this isn't a core API
↓
Looks up APIService registration
↓
Proxies request to metrics-server service
↓
metrics-server processes request
↓
Returns data
↓
Main API Server returns to user
When to Use It 🎯
Use Case 1: Metrics Server 📊
# Provides node and pod resource metrics
kubectl top nodes
kubectl top pods
# Backend: metrics-server aggregated API
Use Case 2: Custom Resource Monitoring 🔍
# Example: Database metrics API
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1.database-metrics.company.io
spec:
service:
name: db-metrics-server
namespace: monitoring
group: database-metrics.company.io
version: v1
Use Case 3: External Service Integration 🔌
# Example: Cloud provider API integration
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1.cloud-resources.aws.company.io
spec:
service:
name: aws-api-bridge
namespace: cloud-integration
group: cloud-resources.aws.company.io
version: v1
# Now can do:
kubectl get ec2instances.cloud-resources.aws.company.io
Use Case 4: ML Model Serving 🤖
# Custom API for ML model management
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1alpha1.ml.company.io
spec:
service:
name: ml-api-server
namespace: ml-platform
group: ml.company.io
version: v1alpha1
# Usage:
kubectl get models.ml.company.io
kubectl describe model my-tensorflow-model
Use Case 5: Multi-Cluster Federation 🌐
# Federate resources across multiple clusters
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta1.federation.k8s.io
spec:
service:
name: federation-apiserver
namespace: federation-system
group: federation.k8s.io
version: v1beta1
Advantages ✅
1. Clean API Extension
- Doesn’t modify core API Server
- Custom APIs feel native to Kubernetes
- Works with kubectl, client-go, etc.
2. Service Separation
- Custom API Server can crash without affecting core APIs
- Independent scaling
- Different performance characteristics
3. Version Management
- Different versions (v1alpha1, v1beta1, v1)
- Gradual API evolution
- Backward compatibility
4. Domain-Specific Logic
- Complex business logic in custom API Server
- Not limited by core API Server constraints
Comparison: Aggregation vs CRDs 🆚
| Feature | Aggregation Layer | CRDs |
|---|---|---|
| Complexity | High | Low |
| Setup | Requires custom server | Built-in to K8s |
| Performance | Can be optimized | Uses main API Server |
| Validation | Custom code | OpenAPI schema |
| Storage | Your choice (not etcd) | etcd only |
| Use Case | Complex APIs, external data | Simple custom resources |
When to use Aggregation:
- Need custom storage backend
- Complex business logic
- High-performance requirements
- External system integration
When to use CRDs:
- Simple custom resources
- Data fits in etcd
- Standard CRUD operations
- Declarative management
Example: Building a Simple Aggregated API Server 💻
1. Create APIService Definition
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1.hello.company.io
spec:
insecureSkipTLSVerify: true # Use proper certs in production!
group: hello.company.io
groupPriorityMinimum: 1000
versionPriority: 15
service:
name: hello-api
namespace: default
port: 443
version: v1
2. Simple API Server (Go)
package main
import (
"encoding/json"
"net/http"
)
type Greeting struct {
APIVersion string `json:"apiVersion"`
Kind string `json:"kind"`
Message string `json:"message"`
}
func handler(w http.ResponseWriter, r *http.Request) {
greeting := Greeting{
APIVersion: "hello.company.io/v1",
Kind: "Greeting",
Message: "Hello from aggregated API!",
}
json.NewEncoder(w).Encode(greeting)
}
func main() {
http.HandleFunc("/apis/hello.company.io/v1/greetings", handler)
http.ListenAndServeTLS(":8443", "tls.crt", "tls.key", nil)
}
3. Deploy
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-api
spec:
replicas: 2
selector:
matchLabels:
app: hello-api
template:
metadata:
labels:
app: hello-api
spec:
containers:
- name: api
image: company/hello-api:v1
ports:
- containerPort: 8443
---
apiVersion: v1
kind: Service
metadata:
name: hello-api
spec:
ports:
- port: 443
targetPort: 8443
selector:
app: hello-api
4. Use It
kubectl apply -f apiservice.yaml
kubectl apply -f deployment.yaml
# Now use the custom API
kubectl get --raw /apis/hello.company.io/v1/greetings | jq .
# Output:
{
"apiVersion": "hello.company.io/v1",
"kind": "Greeting",
"message": "Hello from aggregated API!"
}
Real-World Example: metrics-server 📈
Check if metrics-server is using aggregation:
kubectl get apiservices | grep metrics
# Output:
v1beta1.metrics.k8s.io kube-system/metrics-server True 5d
View the APIService:
kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
Use the metrics API:
# Through aggregation layer
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq .
# User-friendly command
kubectl top nodes
Troubleshooting Aggregated APIs 🔧
Problem: API not available
# Check APIService status
kubectl get apiservices
# Look for "Available" column
# Check service exists
kubectl get svc -n <namespace> <service-name>
# Check pods are running
kubectl get pods -n <namespace> -l <selector>
Problem: Certificate issues
# Check if TLS is configured correctly
kubectl get apiservice <name> -o yaml | grep -A 5 caBundle
# Test connectivity
kubectl get --raw /apis/<group>/<version> -v=8
Problem: Aggregated API slow
# Check API Server logs
kubectl logs -n kube-system kube-apiserver-xxx | grep aggregator
# Check custom API Server logs
kubectl logs -n <namespace> <custom-api-pod>
Best Practices 💡
- Always use TLS for aggregated API servers
- Implement proper authentication (verify client certificates)
- Add health checks for APIService availability
- Version your APIs (v1alpha1 → v1beta1 → v1)
- Monitor performance separately from main API Server
- Use proper RBAC for custom resources
- Document your APIs (OpenAPI spec)
Summary 📝
Aggregation Layer = Extending K8s API with custom API servers
Use When:
✓ Need custom storage (not etcd)
✓ Complex business logic
✓ External system integration
✓ High-performance requirements
Don't Use When:
✗ Simple CRUD operations (use CRDs instead)
✗ Data fits in etcd
✗ Don't need custom logic
Examples:
• metrics-server
• Custom resource monitoring
• Cloud provider integrations
• ML platforms
🧠 Key Takeaways for Learners
Core Concepts:
- API Server = cluster gateway + single source of truth
- Located in control plane (not worker nodes)
- Talks to everyone: users, controllers, scheduler, kubelet
- Stores all state via etcd (only etcd client)
- Stateless design → can scale horizontally
- Critical for security: AuthN, AuthZ, admission control
Advanced Concepts:
- Request pipeline: AuthN → AuthZ → Admission → Validation → etcd
- High Availability: Multiple instances behind load balancer
- Performance tuning: Caching, rate limiting, connection pooling
- Security layers: Network, authentication, authorization, admission, audit
- Aggregation layer: Extends API without modifying core
Production Essentials:
- Always run multiple API Servers (HA)
- Enable audit logging (security & troubleshooting)
- Monitor metrics (request rates, latency, errors)
- Secure with RBAC + Admission Controllers
- Plan capacity based on cluster size
Interview Tips:
- Know the request flow in detail
- Understand HA architecture
- Explain security layers
- Differentiate Aggregation vs CRDs
- Can troubleshoot common issues
📚 Further Study
Practice Labs:
- Set up 3-node HA API Server cluster
- Enable audit logging and analyze logs
- Create a simple aggregated API server
- Troubleshoot “connection refused” errors
- Monitor API Server metrics with Prometheus
Next Topics:
- Part 3: Kubernetes Controllers Deep Dive
- Part 4: Kubernetes Networking
- Part 5: Kubernetes Storage
Bookmark this for your next interview! 🚀
#Kubernetes #K8s #DevOps #SRE #Interview #CloudNative #APIServer #TechInterview
