When you run kubectl apply, you are not issuing an imperative command to start software directly. Instead, you are recording a desired state in a distributed database, triggering a cascade of autonomous control loops. Understanding Kubernetes internals reveals that the system operates as an asynchronous distributed state engine built on declarative consensus, optimistic concurrency control, and continuous reconciliation. This deep dive explains the mechanical breakdown of that process, from the client-side patch calculation to the final commit in etcd, and how the cluster then works to achieve your declared intent.
The Declarative Reality of kubectl apply
Many developers perceive kubectl apply as a remote procedure call: "Kubernetes, spin up my containers right now." The reality is fundamentally different. The kubectl binary doesn't talk to worker nodes directly, nor do the servers running your workload ever receive a direct command to start the application.
Consider this: if every control plane node lost power immediately after the command executed, your containers would continue running uninterrupted. This illustrates that Kubernetes isn't a management tool that pushes changes; it's a state engine. When you apply a manifest, you're updating the cluster's source of truth. Controllers then observe this change and work continuously to make the actual state of the cluster match the declared intent.
The journey of a single kubectl apply command divides into five distinct stages:
- Client Preparation:
kubectlperforms schema discovery and computes a 3-way strategic merge patch. - API Server Gauntlet: The
kube-apiserverruns the payload through authentication, authorization, mutating webhooks, schema validation, and validating webhooks before committing to etcd. - Controller Reconciliation: The Deployment Controller detects the updated specification and creates or updates ReplicaSets.
- Scheduling: The
kube-schedulerwatches for pending Pods, filters out unsuitable nodes, scores the survivors, and writes a binding back to the API server. - Node Execution: The
kubeletdaemon on the target worker node observes the assignment and directs the Container Runtime Interface (CRI) to launch containers.
Client-Side Mechanics: The 3-Way Strategic Merge Patch
Before any network packet leaves your workstation, kubectl performs critical validation and conflict resolution. It doesn't simply upload your YAML file as-is. First, it queries the cluster's Discovery API (typically /openapi/v2 or /openapi/v3) to cache the cluster's OpenAPI schema locally. This allows the CLI to validate your manifest against the cluster's version, ensuring fields exist, indentation is correct, and types match.
The core challenge kubectl apply addresses is concurrent modification. Imagine you apply a deployment with replicas: 3. An hour later, a Horizontal Pod Autoscaler (HPA) scales it to replicas: 10. If you then edit your local YAML to change only the container image and run kubectl apply, what should happen to the replica count?
If kubectl simply overwrote the live state with your local file, it would reset the replicas to 3, undoing the HPA's work. To prevent this without requiring resource locking, Kubernetes uses a 3-Way Strategic Merge Patch. This mechanism compares three data sources to determine the intended changes:
- The Local Configuration: The YAML file currently on your disk.
- The Live Cluster State: The current object as recorded in the cluster database.
- The Last-Applied Configuration: A serialized JSON snapshot of the object's state the last time
kubectl applywas run on it. This is stored in thekubectl.kubernetes.io/last-applied-configurationannotation on the live object itself.
The patch engine calculates two key differences:
- What changed between your previous local file (represented by the last-applied annotation) and your current local file? This identifies your explicit local changes.
- What changed between your previous local file (last-applied annotation) and the live cluster state? This identifies changes made by other actors (like controllers or other users).
The logic then follows a specific set of rules to merge these changes:
- If a field was modified in the live cluster (by an HPA, for example) but not in your local file,
kubectlpreserves the live value. - If a field was modified in your local file,
kubectlupdates it in the cluster. - If a field was deleted from your local file (meaning it was present in the last-applied annotation but is now absent from your current local file),
kubectlremoves it from the cluster.
Once computed, kubectl serializes this into a JSON Patch payload and issues an HTTP PATCH request to the API server. This request specifies the application/strategic-merge-patch+json content type.
1PATCH /apis/apps/v1/namespaces/default/deployments/web-service HTTP/1.1
2Host: your-kubernetes-api-server
3Content-Type: application/strategic-merge-patch+json
4Accept: application/json
5
6{
7 "spec": {
8 "template": {
9 "spec": {
10 "containers": [
11 {
12 "name": "web-service",
13 "image": "your-new-image:v2.0"
14 }
15 ]
16 }
17 }
18 }
19}This intelligent patching ensures that your kubectl apply only changes what you explicitly intended to change, leaving other fields managed by controllers (like autoscalers) or other users intact. This is a cornerstone of Kubernetes's declarative and eventually consistent model.
The kube-apiserver Gauntlet
Once the HTTP request hits the kube-apiserver, it enters a strict processing pipeline. This is where the security, consistency, and overall integrity of the cluster are enforced. The API server acts as the sole entry point for all communication in the cluster. It doesn't run controllers or schedulers itself; its primary role is to manage the cluster's state and validate all requests against it.
The processing flow within the API server follows a specific, ordered sequence:
- Authentication: The server first verifies who is making the request. This typically involves checking X.509 client certificates, bearer tokens (like service account tokens), or OpenID Connect (OIDC) tokens. If authentication fails, the request is rejected immediately with an "Unauthorized" error.
- Authorization: After authentication, the server verifies what the authenticated identity is allowed to do. This is primarily handled by RBAC (Role-Based Access Control). It checks if your authenticated user or service account has the necessary permissions (e.g.,
patchadeploymentin thedefaultnamespace). If authorization fails, the request is rejected with a "Forbidden" error. - Mutating Admission Webhooks: These webhooks are external HTTP callbacks that can modify the request object before it is validated. For example, a webhook might automatically inject a default sidecar container, add specific labels, or set default resource limits if none are specified in your YAML. The object is mutated, and these changes are applied to the request body.
- Schema Validation: The API server then performs rigorous validation of the object against the API schema for the specific Kubernetes version it's running. It ensures that all required fields are present, data types are correct, and no immutable fields (like a deployment's
nameafter creation) are being modified. - Validating Admission Webhooks: These webhooks are also external HTTP callbacks, but unlike mutating webhooks, they cannot modify the object. Their sole purpose is to perform custom business logic checks and either allow or reject the request. Examples include ensuring a pod isn't scheduled on a node with specific taints, enforcing organizational naming conventions, or checking that resource limits adhere to policy.
If the object passes all these stages without error, the API server is ready to persist it. This multi-stage process is critical because any failure at an earlier stage results in an immediate error response to your kubectl command, preventing inconsistent or unauthorized state from ever entering the system.
Etcd Consensus and the Raft Protocol
The final step in the API server's pipeline for a successful request is persisting the object to etcd. This is where Kubernetes internals leverage a robust distributed system component. etcd is a distributed, consistent key-value store that serves as Kubernetes's primary backing store for all cluster data. It uses the Raft consensus algorithm to maintain consistency and durability across the control plane nodes.
When the API server writes to etcd, it doesn't just save a key-value pair. It performs a transactional update. The write request is sent to the leader etcd node. The leader then proposes the entry to its internal log and replicates this log entry to its follower nodes.
The Raft protocol requires a majority quorum to commit an entry. For instance, in a 3-node etcd cluster, at least 2 nodes (the leader plus one follower) must acknowledge the write before it's considered committed. If the leader fails to get a quorum within a timeout period, it steps down, and a new leader is elected. This mechanism ensures that the state in etcd is always consistent, highly available, and durable, even in the face of node failures.
For a kubectl apply operation, this means:
- The API server waits for the etcd transaction to successfully commit across the quorum.
- Only after the etcd leader confirms the commit does the API server return a
200 OKresponse tokubectl. - The response includes the updated object with its new
resourceVersion.
This resourceVersion is crucial for optimistic concurrency control. It's a monotonically increasing identifier for the state of an object. If another client tries to update the same object using an outdated resourceVersion, etcd will reject the write, forcing that client to refetch the current state and retry its operation. This prevents race conditions and ensures that clients are always working with the most up-to-date information without needing complex distributed locks.
Controller Reconciliation and Scheduling
Once the object is successfully persisted in etcd, the kubectl apply command is technically complete from the client's perspective. However, the actual work of running your containers has only just begun. This is where Kubernetes's control plane components, primarily controllers, take over.
Controllers continuously watch the API server for changes to specific resource types. They operate in reconciliation loops, constantly comparing the desired state (as recorded in etcd) with the observed state (the current reality of the cluster) and taking action to bridge any gaps.
- Deployment Controller: This controller watches for changes to Deployment objects. When it detects an updated specification (e.g., a new container image or replica count), it compares the desired state from the Deployment's
specwith the observed state (the current ReplicaSets associated with that Deployment). If they differ, it creates a new ReplicaSet or updates an existing one to reflect the desired number of Pods and their configuration. - ReplicaSet Controller: This controller watches for changes to ReplicaSet objects. Its job is to ensure that the number of running Pods matching its selector always equals the
replicasfield in its specification. If there are too few Pods, it creates new Pod objects. These newly created Pods initially have nonodeNameassigned; they are in thePendingstate, waiting to be scheduled.
The kube-scheduler is another critical control plane component that watches for these Pending Pods. Its role is to assign a suitable node to each Pod. It performs this in two main phases:
- Filtering: The scheduler retrieves a list of all available nodes in the cluster. It then filters out nodes that cannot meet the Pod's requirements. This includes checking for sufficient CPU and memory resources, matching node selectors, tolerating taints, and satisfying volume requirements.
- Scoring: From the remaining suitable nodes, the scheduler scores them based on various factors to find the "best" fit. Scoring considers aspects like node utilization, inter-pod affinity/anti-affinity rules, and preferred taints/tolerations.
Finally, the scheduler creates a Binding object (or updates the Pod's spec.nodeName field) that links the Pod to the selected node. This binding is then persisted back to the API server, updating the Pod's state from Pending to Bound (though this specific state isn't always explicitly visible).
Node Execution and the Kubelet
The final stage of the kubectl apply journey happens on the worker node itself. The kubelet is the primary agent running on each worker node. It continuously watches the API server for Pod objects that have been assigned to its specific node. When it sees a new Pod binding for its node, it takes action to bring that Pod to life.
The kubelet's actions typically involve:
- Network Configuration: The kubelet calls the configured CNI (Container Network Interface) plugin. The CNI plugin is responsible for setting up the pod's network namespace, assigning it an IP address, and configuring network routes and firewall rules, allowing the pod to communicate within the cluster and potentially externally.
- Container Runtime Communication: The kubelet communicates with the CRI (Container Runtime Interface) implementation (e.g., containerd, CRI-O). It instructs the CRI to pull the necessary container images from a registry if they are not already present locally on the node.
- Pause Container Launch: The kubelet first launches a special "pause" container (also known as an
infracontainer). This container shares its network and IPC namespaces with all the application containers within the Pod. This ensures that all containers in a Pod have a stable network interface and can communicate with each other via localhost, even if individual application containers crash and restart. - Application Containers Launch: Finally, the kubelet directs the CRI to start your actual application containers as defined in the Pod's specification.
At this point, your application containers are running within the Pod on the designated worker node. The kubelet continues to monitor the health and status of these containers, reporting their status back to the API server. If a container crashes, the kubelet restarts it according to the Pod's restart policy, ensuring the application remains available.
Common Errors and Troubleshooting
Understanding these Kubernetes internals helps you diagnose and troubleshoot common issues more effectively.
ConflictErrors: These occur when theresourceVersionin your local state (or thelast-applied-configurationannotation) is older than the live state in the API server. This typically happens if another process (like an autoscaler, a controller, or another user) modified the object between your lastgetand your currentapply. The fix is usually to re-runkubectl applyto fetch the latest state and re-calculate the patch, or in some cases, usekubectl deleteand then re-apply if you intend a full overwrite.InvalidErrors: These errors originate from the schema validation stage in thekube-apiserver. They indicate that your YAML manifest contains missing required fields, incorrect data types, or attempts to modify immutable fields. Carefully check your YAML against the Kubernetes API documentation for the specific resource version.ForbiddenErrors: These errors come from the authorization stage in thekube-apiserver. They mean your authenticated user or service account lacks the necessary RBAC permissions to perform the requested operation (e.g.,patchadeploymentin a specific namespace). Verify your roles and role bindings.- Pods Stuck in
Pending: This indicates that thekube-schedulercould not find a suitable node for the Pod. Common causes include insufficient node resources (CPU, memory), unmet node selectors, taints on nodes that the Pod doesn't tolerate, or unsatisfied volume requirements. Checkkubectl describe pod <pod-name>for scheduler events. - Pods Stuck in
ContainerCreatingorImagePullBackOff: These usually point to issues during thekubelet's execution phase.ImagePullBackOffmeans the kubelet couldn't pull the container image (e.g., incorrect image name, private registry credentials missing, network issues).ContainerCreatingcan indicate problems with the CNI plugin, volume mounts, or the container runtime itself. Checkkubectl describe pod <pod-name>andkubectl logs <pod-name> -c <container-name>for more details.
Key Takeaways
kubectl applyis a declarative operation; it updates the desired state in etcd, and controllers continuously reconcile the cluster's reality to match that state.- The 3-Way Strategic Merge Patch is crucial for
kubectl apply, preventing unintended overwrites of concurrent changes made by other controllers or users. - The
kube-apiserveracts as the cluster's gatekeeper, enforcing security and consistency through a strict pipeline of authentication, authorization, webhooks, and schema validation. - etcd, powered by the Raft consensus algorithm, ensures that all state changes are durable, consistent, and highly available across the control plane.
- The actual work of starting containers is performed by the
kubeleton worker nodes, triggered by thekube-scheduler's decision to bind a Pod to a specific node.
FAQ
Does kubectl talk to worker nodes directly?
No. kubectl only communicates with the kube-apiserver. Worker nodes communicate with the API server via watchers and the kubelet. There is no direct command channel from kubectl to the nodes.
What happens if etcd loses a majority of nodes?
If etcd loses a majority of its nodes (meaning it cannot form a quorum), it cannot accept any new write operations. The API server will hang on write operations, and kubectl apply commands will time out. While reads might still work if the leader is available, no state changes can be committed, effectively freezing the cluster's ability to respond to desired state changes.
Why does kubectl use a merge patch instead of a full replace?
A full replace would overwrite any fields that were changed by other actors (like autoscalers, other controllers, or other users) since your last application. The 3-Way Strategic Merge Patch ensures that only the fields you explicitly changed in your local YAML are updated, preserving other legitimate changes made to the live object.
Can admission webhooks modify the object?
Yes, mutating admission webhooks can modify the object before it is validated and persisted to etcd. Validating admission webhooks, however, cannot modify the object; they can only allow or reject the request based on its current state.
About the Author
Suraj Kumar is a writer, entrepreneur, and the CEO and founder of this website, sharing simple and practical insights on business, creativity, and personal growth. With experience building digital projects, they enjoy helping others learn, grow, and succeed online.




